Flavio Silva

🧹 Tidy First? By Kent Beck - My review and takeaways

Tue, 26 Dec 2023 13:00:00 +0000

“Messy code is a nuisance. “Tidying” code, to make it more readable, requires breaking it up into manageable sections. In this practical guide, author Kent Beck, creator of Extreme Programming and pioneer of software patterns, suggests when and where you might apply tidyings to improve your code while keeping the overall structure of the system in mind. (…)” O’Reilly - Tidy First

The book is clear, objective, and easy to read. Its simple explanations and examples on empirical software design carry sophisticated concepts on evaluating when, how, what, and if to tidy at all.

TL;DR: The book is an 8/10 and I recommend it to any programmer at any level of experience. It’s not a 10/10 because the first section about actual tydings with practical examples felt a bit unnecessary. I understand why it’s there, but the discussions presented in sections 2 and 3 were way more valuable for me. It might be the other way around depending on the experience though.

Sections

The author split the book into three sections: Tidyngs, Managing, and Theory.

Tydings

According to Kent Beck, tidyings are a subset of refactoring. Tydings are the cute, fuzzy little refactorings that nobody could possibly hate on. In this section, the author presents several tidyings that can be done to leave things better than before, e.g. adjusting the order based on reading or cohesion, improving comments, etc.

For people with little or no experience with refactorings, this is going to probably be the most valuable part. Personally, the “nobody could possibly hate on” piece, is the highlight of this section. It helped me to create a mental model to identify the line between a tidy and a “full-blown” refactor that would require much more work and discussion.

Managing

“Tyding is geek self-care” - Kent Beck, Tidy First?

Here is where things started to become more interesting for me. This section addresses when to start and stop tidying, and more importantly how to combine tyding, changing the structure of the code, with changing the behavior of the system.

In this section is the reason why I’d love to change the title of this book is “First, After, Later, Never”. It provides a simple and objective guideline to decide when and if to tidy something. I won’t spoil it here so you can read this part from the book and reach your conclusions.

Theory

This was my favorite section and every chapter was very interesting to read, digest, and process. Particularly the correlation between software and cash flow versus options. This analogy helped me to think more structurally about the real cost of software, its development, and its evolution.

Another highlight of this section is having a clear distinction between behavior and structural changes. Where the behavior changes are the ones related to what the software does and structural changes are the ones that support what the software needs to do. E.g. software for HR offers features like payroll, holiday planning, etc. These features are the behaviors, anything else to support these features is structural, such as the code organization, how data is stored, retrieved, etc. Understanding this distinction, their costs, and values helps to better judge when to assume debt and plan to pay/tidy.

Conclusion

This book is the first of a trilogy. The upcoming books are not yet released (as of the date of writing - 22 December 2023), and they will extend the scope of changes and impact. The first one is focused on the individual programmer (you), the second will be focused on the team, and the third will focus on all stakeholders.

🐔 What happened with my project on the Rinha de Backend challenge

Mon, 28 Aug 2023 13:00:00 +0000

I participated in the Rinha de backend (pt-BR) or “Backend Rooster Fight” in English challenge and I was super excited to see the results of my super hacky and performant solution.

The results were scheduled to be published at 21:00 BRT (2 AM CEST my local time). So I stayed up til later that night grabbed popcorn, and waited for the results.

Well, I wish I hadn’t waited…

TL;DR; My solution didn’t run on the official test server because I built the docker image to linux/arm64 instead of linux/amd64 🤦🤦🤦 . I fixed the issue and ran the same tests afterward and I reached ~50% of points of the winner, mainly because in my solution I didn’t tolerate any risk of losing writes. Checkout my repository flavio1110/rinha-de-backend with my solution.

What was the Rinha de Backend?

Before we get into the details, a brief explanation of what was the challenge.

From July 28th to August 25th, the Backend Fight was held, a tournament in which the API that supported the most load during a stress test would be the winner. Participants had to implement an API with endpoints to create, query and search for ‘people’ (a kind of CRUD without UPDATE and DELETE). In the tournament, participants still had to deal with CPU and memory restrictions – each participant had to deliver the API in docker-compose format and could only use 1.5 CPU units and 3GB of memory. More details on technical aspects can be found in the instructions) – (translated from its repository in English).

On top of the description above, some things important to highlight are:

You can write your API with any language or framework, as long as you can build a docker image out of it
The solution has to run two instances of the API in parallel.
The requests will be balanced as you wish between these instances via Nginx.
You have to use one of the following data stores: MySQL, PostgreSQL, or MongoDB.

It was a super fun, informal, practical, and a very good opportunity to exercise and learn new things.

How the winners were determined?

To make the comparison easier and fun, the single parameter used to determine the winner of the challenge was the number of people inserted into the database.

How did I do?

POV you are looking at me while I check the results.

I didn’t. I was disqualified because the container with my APIs didn’t start. Therefore I got nothing. 🤦🤦🤦

Wrong platform?

Yes, that’s true. My solution didn’t even run because it failed to start the APIs. I built the images for linux/arm64 instead of linux/amd64.

It was a mistake on my end, and it made me super disappointed. I ran the tests on the CI pipelines, and it gave me confidence the image was properly built and I didn’t double-check the requirements. Shame on me.

Well, I was disqualified but I was curious to see how it would perform on a server with the same configuration as the one used for the official tests. So, I fixed the platform of the image, spun up an EC2 with the same configuration, and finally ran the tests. The results were a bit disappointing compared to the TOP 10.

How did I approach the problem?

I had two constraints in mind while designing and building:

It has to work – each endpoint has to do what it is supposed to do.
Keep things as simple as possible.

Each of these points brought a few tradeoffs that ended up limiting the overall performance of the solution.

You can check my solution on flavio1110/rinha-de-backend. It’s built with Go and PostgreSQL.

Given many APIs were performing super well, a few days before the last day of the challenge, the load for the stress test was doubled, and it had a massive impact on the performance of my API. You can check the comparison below:

Before…

Print of part of the report generated by Gatling of my solution run on Github BEFORE the load was doubled.

After…

Print of part of the report generated by Gatling of my solution run on Github AFTER the load was doubled.

What were the bottlenecks?

The inserts were the bottleneck. Because I didn’t tolerate the risk of losing writes and wanted to keep it simple, I missed the opportunity to introduce a distributed mechanism to validate the data before trying to insert it into the database and perform the inserts in batch.

Validating the uniqueness of a field Apelido (Nickname)

On my designed solution, without a distributed cache, it was not possible to perform without the DB in 100% of the cases.

I had an in-memory cache, but if the nickname wasn’t there because it wasn’t there because the entry was inserted via the other instance, that request would reach the DB, occupy a connection, take resources, etc.

Introducing a distributed cache like Redis would enable me to add and check the entry in a single place. As a result, we could decrease the memory necessary to run the APIs and move to the cache.

Batching inserts

I inserted each entry per a valid request because I didn’t want to risk losing valid writes in case there was any error between the construction of the batch and its execution.

This was a huge problem because it used too many DB connections and resources, and slowed down the creation request.

Did I like it?

Despite my results, I loved the challenge! I learned quite some tricks with nginx config and some insteresting stuff like using Redis or even PostgreSQL as a “PubSub”. The interaction with the community via Twitter and Github was super nice! I also liked the fact of working with something challenging and closer to reality. Loved it!

What could I have done differently?

Build the image using the correct platform.
A challenge is a challenge. I should have been more flexible with the writing.
Tweak the Nginx configuration to load balance based on fewer connections and disable logs.
Use a distributed cache and drop the read table.
Batch inserts.

What’s next?

Check the official results and look into the repositories of the participants. I guarantee I’ll learn something new.

Stay tuned on @rinhadebackend for the next challenges, I can’t wait for the next ones.

Other than that, I’ll apply the lessons learned to my solution and hopefully get better results. Watch flavio1110/rinha-de-backend and see how it will evolve.

This challenge is officially over, but you can still do it. Challenge accepted?

🔥 Test DB integrations with Testcontainers

Mon, 17 Apr 2023 13:00:00 +0000

Picture this, you have a critical and reasonably complicated piece of logic in your application that is handled in the database. Despite any change on it (or around it), you must have a 100% guarantee that piece continues to work just fine. So, what do you do?

Back in the ancient days…

Ancient cave art of many hand prints. (Credit: Petr Kratochvila/Shutterstock)

Before the “age of containers”, the solution would be a variant of the following:

Have a database with the schema aligned with the application’s version.
Ensure your CI tool has access to that database, to run the tests in your CI pipeline.
Create a script to arrange the necessary data for the test.
Write your testing using the DB.
Create a script to revert any DB changes performed by the test, such as inserts, updates, etc.

This is a very nice post about the Docker: Life Before and after.

What’s the problem with testing with a shared database?

Simply put, it’s complex, expensive to maintain, and has many other reasons to fail. For example:

The DB’s schema version is not the same as the application’s version you want to test.
There is a change in the network policy that blocks access to the database.
Someone’s test setup corrupted the data that you expected to have, so you arrange script fails, or the test fails because of missing data.

Because of the cost and flakiness of such tests, in many cases, the solution was to move logic from DB to the application’s code when performance was not a problem, or just manually test it from time to time and hope it doesn’t break because of an unforeseen reason (it-never-happened-in-the-last-30-minutes).

Containers everywhere

With the popularization of containers, not only developing, runnings, and deploying became easier, but also integrating and testing your external dependencies became easier, faster, and more dependable.

We can now effortlessly run not only the regular dependencies, like RDBMSs and Message Brokers, but we can emulate many cloud provider services using tools like localstack.

What about testing? Testcontainers to rescue!

Testcontainers makes it simple to create and clean up container-based dependencies for automated integration/smoke tests. The clean, easy-to-use API enables developers to programmatically define containers that should be run as part of a test and clean up those resources when the test is done.

It supports many popular languages/frameworks like Go, Java, .NET, Rust, and others. You can check the full list on its official site](https://www.testcontainers.org/) and because it’s open source, you check out and contribute to it in its repositories.

How to use Testcontainers on integration tests?

For this example, assume that one of the features of your software is to search for users based on a few criteria. Your users are stored in a PostgreSQL database and you query them using SQL. Something like:

func searchPeople(ctx context.Context, db *sql.DB, params searchParams) ([]person, error) {
	query := `select first_name, last_name, city
		 from people
		 where ($1::text is null or first_name = $1)
		   and ($2::text is null or last_name = $2)
		   and ($3::text is null or city = $3)
		 order by first_name asc`

  var people []person

  //Execute query, check errors, populate people variable

  return people, nil
}

The search criteria will evolve, by adding new filters or adding new features like pagination. Therefore, we need to make sure that whenever we touch this query its existing behavior is not broken. Let’s write a test for it using Testcontainers.

Writing the test

To be able to run these tests against a real database, we need to:

Start the DB
Create the schema
Arrange data, Act, and Assert

Let’s see in detail how to perform each one of these steps.

1. Start the DB

Let’s create a function that will start a DB, return its connection string and a function to terminate it when we are done.

func startTestDB(ctx context.Context) (string, func(t *testing.T), error) {
	var envVars = map[string]string{
		"POSTGRES_USER":     "user",
		"POSTGRES_PASSWORD": "super-secret",
		"POSTGRES_DB":       "people",
		"PORT":              "5432/tcp",
	}

	getConnString := func(host string, port nat.Port) string {
		return fmt.Sprintf("postgres://%s:%s@%s:%s/%s?sslmode=disable",
			envVars["POSTGRES_USER"],
			envVars["POSTGRES_PASSWORD"],
			host,
			port.Port(),
			envVars["POSTGRES_DB"])
	}

	req := testcontainers.ContainerRequest{
		Image:        "postgres:14",
		ExposedPorts: []string{envVars["PORT"]},
		Env:          envVars,
		WaitingFor:   wait.ForSQL(nat.Port(envVars["PORT"]), "pgx", getConnString).
      WithStartupTimeout(time.Second * 15),
	}
	pgC, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
		ContainerRequest: req,
		Started:          true,
	})
	if err != nil {
		return "", nil, fmt.Errorf("failed to start db container :%w", err)
	}
	port, err := pgC.MappedPort(ctx, "5432/tcp")
	if err != nil {
		return "", nil, fmt.Errorf("failed to get mapped port :%w", err)
	}
	host, err := pgC.Host(ctx)
	if err != nil {
		return "", nil, fmt.Errorf("failed to get host :%w", err)
	}

	connString := fmt.Sprintf("postgres://%s:%s@%s:%d/%s?sslmode=disable",
		envVars["POSTGRES_USER"],
		envVars["POSTGRES_PASSWORD"],
		host,
		port.Int(),
		envVars["POSTGRES_DB"])

	terminate := func(t *testing.T) {
		if err := pgC.Terminate(ctx); err != nil {
			t.Fatalf("failed to terminate container: %s", err.Error())
		}
	}
	return connString, terminate, nil
}

The code above is self-explanatory if you are familiar with Go, but there are a few aspects that I’d like to highlight:

Testcontainers will map and assign a random port. Therefore, in this case, instead of connecting to 5432, it’s necessary to connect to the assigned port. This is similar to the hostname. Therefore, to get your correct connection string it’s needed to resolve port and host.
The container will automatically shut down when the application is finished. However, it is recommended to defer its termination as soon as it is started. For that, a terminate func is returned by the method above.

So far our test looks like this:

func TestSearchPeople(t *testing.T) {
	ctx := context.Background()
	connString, terminate, err := startTestDB(ctx)
	if err != nil {
		t.Error(err)
	}
	defer terminate(t)

	db, err := openDB(connString)
	if err != nil {
		t.Fatal("fail to open DB", err)
	}
	// Continue
}

2. Create the schema

With a DB up and running we can initiate it with the expected schema, a.k.a migrate the DB schema.

In the real world, we would use something more sophisticated like go-migrate, but for the sake of simplicity let’s write something simpler.

func migrateDB(ctx context.Context, db *sql.DB) error {
	_, err := db.ExecContext(ctx, "create table if not exists people (first_name text,last_name text,city text)")
	if err != nil {
		return fmt.Errorf("failed to migrate DB: %w", err)
	}
	return nil
}

3. Arrange data, Act, and Assert

The last step before the actual test is to arrange the data. We could accomplish it together with the schema migration, however keeping these two parts separate helps to keep each test isolated, avoiding side effects when the data is tweaked.

For our tests, we need to insert 4 different people with distinct names and cities. Then we can use this data to act and assert the results. This is what the final test func looks like, testing every variation of our query.

func TestSearchPeople(t *testing.T) {
	ctx := context.Background()
	// Start the DB
	connString, terminate, err := startTestDB(ctx)
	if err != nil {
		t.Error(err)
	}
	// Terminate the container when the func TestSearchPeople finishes
	defer terminate(t)

	// Open the the database
	db, err := openDB(connString)
	if err != nil {
		t.Fatal("fail to open DB", err)
	}

	// Migrate schema
	if err := migrateDB(ctx, db); err != nil {
		t.Fatal("fail to migrate DB", err)
	}

	insert := `insert into people values
			('Flavio', 'Silva', 'Olbia'),
			('Joost', 'Van Huis', 'Amsterdam'),
			('Aldben', 'Arimeritin', 'Istambul'),
			('Nando', 'Pelect', 'Perth');`

    // Arrange data
	if _, err := db.ExecContext(ctx, insert); err != nil {
		t.Fatal("fail to insert initial data", err)
	}

	// Several test cases
	t.Run("without filters", func(t *testing.T) {
		people, err := searchPeople(ctx, db, searchParams{})
		assert.NoError(t, err)
		assert.Equal(t, 4, len(people))
	})

	t.Run("filtering by first name", func(t *testing.T) {
		people, err := searchPeople(ctx, db, searchParams{firstName: strPtr("Joost")})
		assert.NoError(t, err)
		assert.Equal(t, 1, len(people))
		assert.Equal(t, "Joost", people[0].firstName)
	})
	t.Run("filtering by last name", func(t *testing.T) {
		people, err := searchPeople(ctx, db, searchParams{lastName: strPtr("Arimeritin")})
		assert.NoError(t, err)
		assert.Equal(t, 1, len(people))
		assert.Equal(t, "Arimeritin", people[0].lastName)
	})
	t.Run("filtering by city", func(t *testing.T) {
		people, err := searchPeople(ctx, db, searchParams{city: strPtr("Olbia")})
		assert.NoError(t, err)
		assert.Equal(t, 1, len(people))
		assert.Equal(t, "Olbia", people[0].city)
	})
}

With this test, we have the guarantee the covered scenarios are working. Profit!

You can check the results by running the tests.

go test ./... -v
=== RUN   TestSearchPeople
2023/05/09 15:48:55 github.com/testcontainers/testcontainers-go - Connected to docker:
  Server Version: 20.10.22
  API Version: 1.41
  Operating System: Docker Desktop
  Total Memory: 3932 MB
2023/05/09 15:48:55 Starting container id: 95dc2928dae3 image: docker.io/testcontainers/ryuk:0.3.4
2023/05/09 15:48:56 Waiting for container id 95dc2928dae3 image: docker.io/testcontainers/ryuk:0.3.4
2023/05/09 15:48:56 Container is ready id: 95dc2928dae3 image: docker.io/testcontainers/ryuk:0.3.4
2023/05/09 15:48:56 Starting container id: adedca079bc6 image: postgres:14
2023/05/09 15:48:57 Waiting for container id adedca079bc6 image: postgres:14
2023/05/09 15:48:58 Container is ready id: adedca079bc6 image: postgres:14
=== RUN   TestSearchPeople/without_filters
=== RUN   TestSearchPeople/filtering_by_first_name
=== RUN   TestSearchPeople/filtering_by_last_name
=== RUN   TestSearchPeople/filtering_by_city
--- PASS: TestSearchPeople (3.48s)
    --- PASS: TestSearchPeople/without_filters (0.00s)
    --- PASS: TestSearchPeople/filtering_by_first_name (0.00s)
    --- PASS: TestSearchPeople/filtering_by_last_name (0.00s)
    --- PASS: TestSearchPeople/filtering_by_city (0.00s)
PASS
ok  	github.com/flavio1110/go-for-csharp-devs/experiments/test-db-interactions	.724s

You can check the source code of this test on my experiments repository.

Final thoughts

Despite Testcontainers reducing the cost of integration tests and making them more reliable, it doesn’t remove the cost entirely. The tests will take more time to run (especially if where you are running the tests doesn’t have the image downloaded yet).

Nevertheless, in my opinion, the price is low compared to the benefit that it yields.

What about you? How do you write integration tests, and what are your main challenges?

🔥 Improve your Frontend skills with Frontend Mentor

Tue, 11 Apr 2023 13:00:00 +0000

When studying HTML, CSS, and Javascript it’s hard to practice with real-world examples, so oftentimes we get stuck trying to come up with a good design and then building it.

Frontend Mentor to the recue!

Frontend Mentor offers front-end coding challenges and interesting projects to practice your HTML, CSS, and JavaScript. It is as close as it gets to work on a professional real-world project.

After creating your account, you can start one of its challeges. There are many free challenges that can get you started and give you plenty of things to play with. However, if you want take one step further, have access to premium challenges, figma and sketch design files, and private solutions, you can subscribe to the pro developer.

How does it work?

Once you start a challenge, you will download a zip for that challenge. For example the challenge Four card feature section, will give you a zip file containing the instructions, necessary images, basic design details, and the most important the design images for desktop and mobile!

These files are all you need to start hacking your front end!

After the challenge is complete you can submit it and get feedback from the community! That’s excellent!

I’ve been playing with some challenges myself, (the image of the banner is from my own implementation of the Four Cards feature section).

I’m not a Frontend specialist by no means, but you can find my solutions so far on my GitHub repository.

What about you? Do you know any other tool that offers such an immersive and near-real-world experience?

🐉 got: Embracing my lazyness with my custom git CLI

Thu, 30 Mar 2023 13:00:00 +0000

got is a CLI written in Go, created on top of the git CLI, to make my life easier by shortening some commands I use daily.

Because it’s built on top of git, all git commands will also work just fine and I don’t need to keep switching between got and git!

If I want to clean up my local dead branches I can got rmb instead of git branch | grep -v "main" | xargs git branch -D (you can call me lazy. I accept that 😅).

xkcd Automation.

But… Why?

My first experience with a Distributed version control system (DVCS), was many years ago with mercurial hg, and despite taking some time to get used to the new way of working as compared to a centralized version control system, I got very used to its short commands, aliases, and simplicity. We could do something like hg pull or hg pul (that’s not a typo. This is an actual alias for hg pull), we were able to close a branch and commit a message in a single command like hg commit --close-branch -m 'closing this branch'. It was super nice and handy! Then, eventually, I started working with git and I was amazed by its differences, possibilities, and features. However, it felt more verbose for day-to-day tasks. So I ended up creating a bunch of aliases for the commands that I use more often, or commands that are longer and I always have to google it to remember. Sneak peek of my ~/.zshrc

#...
alias push='git push origin head'
alias stat='git status -s'
alias gbr="git branch | grep -v "main" | xargs git branch -D"
alias pick="git cherry-pick"
# and a few more...
#...

Speaking on zsh, if you use it with the go plugin, you will have a conflict with the alias got for go test.

The aliases work so well and are easy to maintain… But I wanted something fancier and all have the opportunity to go through the process of writing a CLI in Go.

How

Got is built using cobra, which makes the work so much easier. Cobra has its own CLI called Cobra Generator that helps bootstrap your CLI project and add commands. You can check the full documentation here, but here goes the basic usage:

Install

With go installed, open the terminal and execute the following command:

go install github.com/spf13/cobra-cli@latest

Booststraping your CLI

Navigate to a folder that you want to have your project, init a go module, then execute the cobra init. e.g

go mod init my-cli
cobra-cli init

Voilá! You have your custom CLI, ad you can run it with go run .! You will see a result like:

A longer description that spans multiple lines and likely contains
examples and usage of using your application. For example:

Cobra is a CLI library for Go that empowers applications.
This application is a tool to generate the needed files
to quickly create a Cobra application.

You can now add a command with executing cobra-cli add ping, then execute the command again. You will see a result like:

A longer description that spans multiple lines and likely contains
examples and usage of using your application. For example:

Cobra is a CLI library for Go that empowers applications.
This application is a tool to generate the needed files
to quickly create a Cobra application.

Usage:
  my-cli [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  ping        A brief description of your command

Flags:
  -h, --help     help for my-cli
  -t, --toggle   Help message for toggle

Use "my-cli [command] --help" for more information about a command.

The foundation for your CLI is in place, you only need to worry about the actual logic of the commands because cobra will take care of all the pumbling for you.

You can check several examples in the cobra documentation, and use it as a base to create your shine CLI.

Ok, what about the logic in got?

In got I have two ways of executing some logic. 1) using go-git to perform some actions like iterate in all local branches and delete all except main, and 2) executing a git cli command from my Go application directly.

go-git

go-git is a highly extensible git implementation library written in pure Go. It can be used to manipulate git repositories at low level (plumbing) or high level (porcelain), through an idiomatic Go API. - (Description from its repo).

It is such a well-documented, flexible, and powerful lib. I highly recommend looking into it if you even thought to do something with git, or if you intend to create your own lib. 10/10!

The example below stages all files (including untracked) and commit them:

path, err := os.Getwd() // get the current path
exitIfError(err)

r, err := git.PlainOpen(path) // open the repository related to the path
exitIfError(err)

w, err := r.Worktree() // get the current worktree
exitIfError(err)

_, err = w.Add(".") // Stage all files
exitIfError(err)

_, err = w.Commit("commit yay!", &git.CommitOptions{}) // commit
exitIfError(err)

I have to say that for basic operations like this one, we could be better served by executing the git cli directly like in the example below. However, go-git opens several possibilities like in the support of several type of storage, such as in-memory, file system, or anything you can think of as long as you implement the Storer interface. At this moment I’m not using such features yet, but I’m planning to use them for some commands like got squash that will squash all commits of the current branch.

cmd := exec.Command("git", "add", "-A")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stdout
_ = cmd.Run()


cmd = exec.Command("git", "commit", "-m", "commit yay!")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stdout
_ = cmd.Run()

Fallback to git

It is important because I wanted to “proxy” all unknown got commands to git. In this way, I can use got for my custom commands and the standard git commands without worrying about what is available where. The other benefit is I don’t need to implement things that are good enough or I don’t use so often.

Implementing it was fairly simple. The snippet below is part of the root.go

func Execute() {
	err := rootCmd.Execute()
	if err != nil {
		fallbackToGit()
	}
}

func fallbackToGit() {
	cmd := exec.Command("git", os.Args[1:]...)
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stdout
	_ = cmd.Run() // Skipping error here, because git already sends it to stdout, and I don't have anything else to do with it.

Conclusion and what’s next?

This small project started as an experiment for playing with writing a custom CLI, but I ended up creating something that is very useful for me. Win-win!

I still have a few commands I want to introduce in got, but my next step is to create a reasonable test suit, so I can have confidence that things work as they suppose to. This is especially important given it is responsible for my interaction with git. That’s a big deal!

Anyways, I created it to attend to my laziness, but if got looks interesting to you, feel free to use, fork, and contribute. 💪

I hope this post can help you to see how easy is to write a CLI using Go, and maybe can inspire you to identify things that you do daily and can be somehow optimized.

Till next time o/

📺 YouTube: Suggestions #1

Tue, 28 Mar 2023 13:00:00 +0000

We live in an unprecedented era in which we can learn pretty much anything we want for free (or paying very little depending on how much patience you have with ads).

YouTube is undoubtedly one of the best sources of free content. The only challenge is really to find good channels where you learn stuff and have a good time while doing it.

This is the first issue of a series of posts in which I’m going to recommend three channels with excellent videos.

TechWorld with Nana

YouTube Channel

Twitter

This is my favorite channel for DevOps-related topics. Nana’s didactics and real-world examples help a lot to get to know new topics or deep dive into a specific subject. This 4-hour long k8s full course is just one of many others. Check it out.

Matt Pocock

YouTube Channel

Twitter

Advanced TypeScript wizardry, plus on-the-day updates from the latest TypeScript releases (and other open-source loveliness). Check it out

Fireship

YouTube Channel

Twitter

The Gateway Drug for developers. This is how the channel describes itself. 😅

The super funny weekly code report and the 100 seconds pills of knowledge about many different things are very informative and entertaining. Check it out.

What about you? What other YouTube channels do you watch to get up to speed?

📂 Go: Importig a CSV to PostgreSQL

Sun, 26 Mar 2023 13:00:00 +0000

If you are using Go ad PostgreSQL, and need to performa a bulk import a CSV, it’s most likely you will find the COPY protocol is the feature that suits you better. In that direction, you will find examples using pgx CopyFrom that relies on the native protocol, and it’s fairly easy to use. However, depending how it’s used you can have an exponencial increase of memory consumption of your application making it unreliable and more expensive to run.

TL;DR;: Don’t load the file in memory and use pgx.CopyFromRows. Instead, use the io.Reader of the file and implement a custom pgx.CopyFromSource.

Checkout the repository with examples and details presented here.

Context

Most of the examples out there, are either using CopyFromRows or CopyFromSlice. However, the big problem is these two options require you to have the entire content in memory to use. This is not a big deal when dealing with small files, there won’t be concurrent usage, or you have infinite memory 😅.

How big is the problem?

Comparing the memory consumption for the two distinct approaches importing a file with ~16MB (1M rows).

Approach/metric	TotalAlloc	Sys
Stream file	61 MiB	12 Mib
Read entire file	84 MiB	58 Mib
	+37.70%	+346.15%

Yes, you read it right! using CopyFromRows obtained +346.15% of memory from the OS! 58 MiB instead of 12 MiB.

You can read more about the meaning of each metric on https://golang.org/pkg/runtime/#MemStats and find the source code and details of the comparisson on this repository..

What’s the most efficient way for using the COPY protocol with pgx?

Instead of reading the entire in memory, the idea is to stream each line of the file directly to PostgreSQL. In this way, we only need to keep the current line in-memory as opposed to the entire file.

How can we do it?

The CopyFrom method receives an implementation of the interface CopyFromSource.

Let’s implement this interface using a CSV file with the followng three columns: first_name, last_name, and city.

func newPeopleCopyFromSource(csvStream io.Reader) *peopleCopyFromSource {
   csvReader := csv.NewReader(csvStream)
   csvReader.ReuseRecord = true // reuse slice to return the record line by line
   csvReader.FieldsPerRecord = 3

   return &peopleCopyFromSource{
       reader: csvReader,
       isBOF:  true, // first line is header
       record: make([]interface{}, len(peopleColumns)),
   }
}

type peopleCopyFromSource struct {
   reader        *csv.Reader
   err           error
   currentCsvRow []string
   record        []interface{}
   isEOF         bool
   isBOF         bool
}

func (pfs *peopleCopyFromSource) Values() ([]any, error) {
   if pfs.isEOF {
       return nil, nil
   }

   if pfs.err != nil {
       return nil, pfs.err
   }

   // the order of the elements of the record array, must match with
   // the order of the columns in passed into the copy method
   pfs.record[0] = pfs.currentCsvRow[0]
   pfs.record[1] = pfs.currentCsvRow[1]
   pfs.record[2] = pfs.currentCsvRow[2]
   return pfs.record, nil
}

func (pfs *peopleCopyFromSource) Next() bool {
   pfs.currentCsvRow, pfs.err = pfs.reader.Read()
   if pfs.err != nil {

       // when get to the end of the file return false and clean the error.
       // If it's io.EOF we can't return an error
       if errors.Is(pfs.err, io.EOF) {
           pfs.isEOF = true
           pfs.err = nil
       }
       return false
   }

   if pfs.isBOF {
       pfs.isBOF = false
       return pfs.Next()
   }

   return true
}

func (pfs *peopleCopyFromSource) Err() error {
   return pfs.err
}

You can now use this implementation in the CopyFrom method. e.g.

_, err := pgxConn.CopyFrom(ctx, pgx.Identifier{"people"}, peopleColumns, newPeopleCopyFromSource(csvStream))

Conclusion

Using the CopyFrom with CopyFromRows or CopyFrom will significantly increase the memory comsultion of your application. The high memory usage can bring several problems like OOM errors, increase of costs, unavailability, etc.

By using a custom implementation of CopyFromSource will make your application much more efficient, reliable, and cheaper to ru.

You can find the entire source code of the examples above on this repository. There you will also find more deatils about the comparisson and the not-so-great implementation.

🔢 Go: Where is the decimal type?

Tue, 21 Mar 2023 13:00:00 +0000

Go doesn’t have a primitive decimal type for arbitrary-precision fixed-point decimal numbers. Yes, you read it right. Therefore, if you need to deal with fixed-point precision there are two main options:

Use an external package like decimal, which introduces the decimal type. However, the current version (1.3.1), can “only” represent numbers with a maximum of 2^31 digits after the decimal point.
Use int64 to store and deal with these numbers. For e.g. given you need 6 precision digits, therefore 79.23, 23.00, and 54.123456, become respectively 79230000, 23000000, and 54123456.

There is an open proposal to add decimal float types (IEEE 754-2008) in the std lib. However, for now, it’s just a proposal being discussed, without guarantee it will be ever added.

🧠 Go resources for beginners

Sun, 19 Mar 2023 13:00:00 +0000

Below a living list of compiled links for whoever is learning Go.

Official docs

Effective Go - https://go.dev/doc/effective_go
- A document that gives tips for writing clear, idiomatic Go code. A must-read for any new Go programmer. It augments the tour and the language specification, both of which should be read first.
Go Code Review Comments https://github.com/golang/go/wiki/CodeReviewComment
- This page collects common comments made during reviews of Go code, so that a single detailed explanation can be referred to by shorthands. This is a laundry list of common mistakes, not a comprehensive style guide. You can view this as a supplement to Effective Go.
Go Docs - https://go.dev/doc/
- Root for many useful documentations
FAQ - https://go.dev/doc/faq
- Answers to common questions about Go.

Blogs

Dave Cheney - https://dave.cheney.net/
- Specially the practical go section, it has TONS of good advice and real-world examples of how to deal with daily challenges. I highly recommend it.

Courses

The way to go - https://www.educative.io/courses/the-way-to-go
- It’s a course from educative.io, it goes from the basics concepts of the language to more advanced ones. It also brings very interesting insights into the differences between the approaches of Java/C# to Go..
How to code in go - https://www.digitalocean.com/community/tutorial_series/how-to-code-in-go
- A great collection of tutorials that cover basic Go concepts, ideal for beginers

Videos

Go in 100 seconds - https://www.youtube.com/watch?v=446E-r0rXHI&t=38s
- Short introduction about Go
Simplicity is complicated - https://www.youtube.com/watch?v=rFejpH_tAHM
- Rob Pike talks about how Go is often described as a simple language. It is not, it just seems that way. Rob explains how Go’s simplicity hides a great deal of complexity, and that both the simplicity and complexity are part of the design.
Concurrency is not parallelism - https://www.youtube.com/watch?v=qmg1CF3gZQ0&t=1582s
- Rob Pike talks about concurrency and how Go implements it
Understanding channels - https://www.youtube.com/watch?v=KBZlN0izeiY&t=1011s
- Channels provide a simple mechanism for goroutines to communicate, and a powerful construct to build sophisticated concurrency patterns. We will delve into the inner workings of channels and channel operations, including how they’re supported by the runtime scheduler and memory
Concurrency in Go - https://www.youtube.com/watch?v=\_uQgGS_VIXM&list=PLsc-VaxfZl4do3Etp_xQ0aQBoC-x5BIgJ
- Playlist with a few short videos about different components of concurrency in Go
Just for func - https://www.youtube.com/watch?v=H_4eRD8aegk&list=PL64wiCrrxh4Jisi7OcCJIUpguV_f5jGnZ
- A very complete playlist of tutorials given by Francesc Campoy, a past Developer Advocate for the Go team at Google, that cover simple to advanced topics in Go
Golang crash course - https://www.youtube.com/watch?v=SqrbIlUwR0U
- A 90 minutes video that covers most of Go features with cristal clear live coding examples, excelent for beginers to get a fast gist of Go

Podcasts

Go Time - Spotify
- Diverse discussions from around the Go and its community

👋 Welcome

Sun, 19 Mar 2023 13:00:00 +0000

Welcome, here I’ll be sharing my discoveries, studies, and experiences in the software industry. As a software engineer, I’ve had the opportunity to work on a variety of projects and technologies, and I’ve learned a lot along the way.

My goal in creating this blog is to document my journey and share what I’ve learned with others. Whether you’re a seasoned developer or just starting out, I hope that the insights and knowledge I share will be useful and valuable to you.

In this blog, you can expect to find posts about various topics related to software engineering, such as programming languages, frameworks, tools, best practices, and more. I’ll also share my thoughts on the latest trends and developments in the industry.

Above all, I hope that this blog will inspire you to continue learning and growing as a software engineer. Thanks for stopping by, and I look forward to sharing my experiences with you.