Some special mentions:

• Karan Sharma - I personally learnt and got into nomad because of Karan and also contributed once upstream. He’s an avid open source enthusiast, experiments a lot and share it across in his blog.
• Jake Howard - Most of my self-hosted infra is inspired by Jake’s infrastructure. He writes a lot on self hosting.
• Sean Goedecke - Writes more about politics in the corporate world. Sometimes developers need the harsh pills and should know about how a company thinks, behaves and rewards it’s people.
• Lalit Maganti - Similar to Sean, but also have some technical entries as well. I personally love his essays.
• Simon Willison - Writes a lot on the recent developments in the field of LLMs.

And the following blogs that cover a lot on linux, AI and computers in general:

• Anton Zhiyanov
• Michael Stapelberg
• Vishnu Haridas
• Anthony Simon
• Herman Martinus
• Cameron Westland
• Jaz

and the usual - news.ycombinator.com and lobste.rs.

In python you can use libraries like spaCy, BERT, etc to do this or you can also use LLM to extract this info as shown here.

• piHole and dnscrypt-proxy has been replaced with Adguard Home since it does both.
• photoprism - self-hosted alternative to google photos and iCloud photos. I have bought a new iPhone with iCloud plans for my family, but I’m still backing up all the photos here just in case. I am planning to explore immich in the next year when I get some time.
• loki, prometheus, grafana, blackbox-exporter and alertmanager - for observability.
• jellyfin - currently testing as an alternative to plex.
• uptime-kuma - for uptime monitoring.
• Actual Budget - helps me to keep track of my finances. Earlier, I was using MyExpenses in android which is open-source with some paid features and since I’m not carrying my android phone daily, I moved to actual and it’s been great so far.

I added two servers in late 2023 in addition to the raspberry pi and they’ve been running for 2 years without any problems:

1. blr - server from DigitalOcean in Bangalore region. Takes care of latency sensitive and critical services.
2. hydra - server from Hetzner in Germany. The arm64 shared servers are good in terms of price-to-performance ratio. I use this server for all the resource heavy services.
3. pi - a raspberry pi 4B connected to my home network and runs a DNS server alone for now.

All the services are deployed as docker containers via ansible and terraform. I talk to all these servers via tailscale SSH.

All set (we thought), but no-one visited the site. We constantly checked the stats page and there’ll be some random users from random countries who visited the homepage and will leave in a few seconds. We just don’t know what’s wrong with the site. Then we came to know about SEO (Search Engine Optimization) which made us change almost all of our content as per the Google SEO recommendations. Then this blogger site was eventually moved to Drupal, then to Wordpress and then finally to a static site built on top of jekyll.

This small side-project helped me learn a lot about managing servers, DNS, SEO, HTML, CSS, JS, PHP and ruby to some extent. After getting my first job at Cognizant, all these learning alongside very minor explorations with docker helped me land my second job.

After joining Freshworks as a Platforms Systems Engineer, we are supposed to manage the servers for our services and all my previous experiences came in handy. Somewhere in the middle around 2020, projects like Kubernetes and Nomad were interesting and I learnt them both. I tried deploying a sample rails app using these orchestrators and even contributed a small enhancement to Nomad. Luckily the organisation was moving to kubernetes and my previous experience helped me to move faster and also assist other developers who are new to it. In the meanwhile, I moved to another team managing multiple golang services and it was way easier for me to onboard because of my previous open-source stint.

These are just the major instances which I can think of where my side-quests have helped me to connect the dots. At any point of time in my career, I have at-least one side-project that I actively work on. It may not see the light of day, but it helps to learn something new. So, if you’re a developer, please have a side-project.

from typing import List

class A:
    items: List[B]

class B:
    name: str

To fix, we should reorder the classes:

from typing import List

class B:
    name: str

class A:
    items: List[B]

Official documentation - https://docs.astral.sh/uv/guides/scripts.

# create a script
uv init --script example.py --python 3.13

# add depedencies
uv add --script example.py 'requests' 'pandas'

# run the script
uv run example.py

This entry was supposed to go live on December 2023 and it was only published until a year after in 2024.

I was 10 years old when I got hooked with Road Rash on a PC at my neighbour’s house. My parents got me a Toshiba laptop when I was 17 years old and I played some casual games in it from time to time. This laptop served me great for 4 years and then one day during heavy thunderstorms, it got fried due to an electrical surge.

I bought a second laptop — this time a Lenovo with Nvidia 960M graphics inside with my side gig money. This one worked for 3 years and I extended 3 more years by changing its internal disk to SSD. By this time, I also had a 13-inch macbook pro from work and since it’s unix based, it became my goto for all development work.

I then bought two more mac devices (2018 Macbook Pro and 2023 M2 Mac Mini), but the urge to build myself a desktop PC was still there and I wanted to build a PC myself. Currently, Mac Mini works great for my development work and for some cracked sessions in Factorio, but some good games like Cities Skylines, Stronghold Crusader series, Anno 1800 and other castle simulation RTS games were only available on PC.

I had some free time in December and thought I could build a PC for the holidays. I stumbled upon GamersNexus and lurked on IndianGaming discord for research and clarified some of my questions with the community. Since I already had a 4k monitor (Dell Ultrasharp U2720Q), I wanted to build a system that can comfortably handle at least 1440p. So, after tons of research this is my parts list:

If you’re building a PC in India, you should definitely visit pcpricetracker to compare prices between multiple PC parts online retailers. Amazon was costly even with a 5% discount from Amazon Pay card. You should also check with offline retail stores in your cities. I got most of the parts from an offline retail store in Coimbatore. The cabinet and the CPU cooler that I wanted were not available in that store, so I had to order online at PC Studio.

After all the parts arrived, I unpacked all the parts. I knew it will take a lot of patience, so I prepared everything for the BIOS POST.

unpacked parts on the floor. in hindsight, doing it on the floor was a mistake.

I started setting up the motherboard, RAM, PSU and the CPU. I saw the BIOS POST and then proceeded with the usual build. After mounting the CPU cooler, NVMe drive, GPU and some cable management, it was ready. All worked well after boot and sadly when I was closing the side glass panel of the cabinet, I dropped 5cms above a hard surface and it cracked.

shattered glass panel of the cabinet

When I looked this up on the internet, I realized it’s one of the newbie mistakes the new builders make. Fortunately, Corsair support was kind enough to send a replacement without any charges even after I offered to pay them. Thanks Corsair. Till the replacement came I used a cloth to seal the cabinet. Anyway, this is the final build after spending 4 hours on it.

final image of the cabinet. sorry i didn’t take any pictures in-between.

It’s beautiful, isn’t it? Okay, there are no fancy RGBs, but it gets the job done. Now I’m playing all the games at 4k and the system handles it very well. I just wish I have more time to game :).

Update 19, Sep 2023: Go team confirmed the default behaviour will be changed from version 1.22.

At work, both my previous project and the current project I’m working on relies on SQS for failure retries with exponential backoff. The current project is a webhook service that delivers webhooks to our customers from our products and also among internal services. We use SQS for failure retries and the current queue setup looks like this:

“visibility timeout” above refers to the maximum time SQS will wait before sending the same message to another consumer API call if the message is not deleted by your consumer. This will be helpful in cases where your consumer pod/instance crashed and the messages should be re-delivered to other consumers. “maximum receive count” refers to the maximum number of consumer deliveries SQS will do for a same request before moving it to the dead-letter queue.

When a request fails on the first try due to various reasons (connection timeout, non-2xx response, etc), the service will push the request to the first failed queue (failed-0) and mark it as failed. The failed consumers should delete the message from the current failed queue (failed-0) irrespective of the result of the operation, else they’ll become visible to subsequent ReceiveMessage API calls.

We were in the middle of migrating internal test endpoints and found that one endpoint was failing a lot because of connection timeouts and our golang consumers were unable to keep up with the traffic (~20k rps). All these requests were pushed to the failed queue (failed-0) and our consumers were accepting the messages, but only a very few of these failed messages were deleted and pushed to the second failed queue (failed-1). Also, I can see our consumer processing the same message 10 times (what?).

Meanwhile the NumberOfMessagesNotVisible metric (the number of messages that are currently processed by the consumers) for the first queue spiked to the maximum limit of 120K allowed by SQS. Post this limit SQS won’t return any messages in ReceiveMessage api calls causing a delayed delivery for all the failed messages.

So, there are two issues at hand - the consumer is processing duplicate messages and the NumberOfMessagesVisible reached the 120k limit. Just to be sure that the producer is not pushing any duplicate messages to the SQS queue, I checked the logs (we’re using ELK stack internally) and can see we’re producing the message only once. I also polled for messages multiple times in the AWS SQS console and there I don’t see any duplicates (though it’s very random and we can’t be sure). So, the bug might be in the consumer code?!

I was also able to reproduce this in my local setup and started a debugging session to check whether SQS ReceiveMessage API call returned any duplicate messages. As per design, after receiving the messages via ReceiveMessage, each message will be pushed to a buffered channel which is consumed by a goroutine worker pool. The code looked like this:

for {
	resp, err := SqsClient.ReceiveMessage(ctx, &sqs.ReceiveMessageInput{
		QueueUrl:              &queueUrl,
		MessageAttributeNames: []string{"ApproximateReceiveCount"},
		MaxNumberOfMessages:   10,
		WaitTimeSeconds:       20,
	})
	if err != nil {
		if ctx.Err() != nil {
			log.Ctx(ctx).Debug().Msg("context cancelled. stopping sqs reader")
			break
		}
		log.Ctx(ctx).Error().Err(err).Msg("error while calling ReceiveMessage")
		continue
	}

	// if resp is nil, log and continue
	if resp == nil {
		log.Ctx(ctx).Info().Msg("sqs receiveMessage response is nil")
		continue
	}

	// iterate and push each message to the buffered worker channel
	for _, msg := range resp.Messages {
		workerChan <- &msg
	}
}

Do you find anything weird in the snippet above? Look carefully again inside the for loop. Still nothing? I also didn’t notice anything and started the debugger. Turns out, I’m bitten by the golang’s loop variable gotcha. Go by design handles for loop variables differently than other languages. If you pass the loop variables by reference in golang, it will always point to the last element. So when &msg was passed to the workerChan we sent the last element of resp.Messages 10 times and ignored the first 9 elements.

This explains why the 120k limit was also hit. When the worker goroutine consumes a message, it’ll process it and delete it from the queue as required by SQS. Since we’re deleting 1 in 10 messages and ignoring the other 9, after the configured visibility-timeout interval these messages will be returned again in ReceiveMessage API call. The below code fixes this issue:

for _, msg := range resp.Messages {
	m := msg // assign it to a temp variable
	workerChan <- &m
}

This fix won’t be needed in newer versions of go as per the latest discussions, but you never know when you need a rock solid observability for your entire infrastructure. In my case, having dead letter queues and alerts for NumberOfMessagesNotVisible helped narrowing down this issue faster. Always have dead-letter queues configured kids!

Though, I’m not currently self-hosting email (and I strongly recommend you not to) for good reasons, it’s a good start to consider hosting the easy things. I own a raspberry pi model 4B which I thankfully got before the pandemic.

Here’s the list of items I self-host:

• piHole - a rock solid DNS server. It blocks trackers across the network for all of our devices. Adguard Home is also a good alternative.
• dnscrypt-proxy - An upstream for piHole since it doesn’t support DNS-over-HTTPS. I’m currenty using cloudflare’s DoH as upstream.
• miniflux - simple, clean and minimalistic RSS reader. I follow and read about other bloggers and news publications via miniflux.
• syncthing - synchronises and backs up my data across all devices and servers. Mobile clients are also available.
• plex - local media server. I mostly consume new content from online streaming platforms but there are some older movies I have in mp4/mkv format.
• tranmission-daemon - a lightweight torrent client with a web UI. Since I have a higher FUP plan from ACT (~4TB per month), I seed linux distros and scientific papers.

Things I wish to self-host in future:

• headscale - open-source co-ordination server for tailscale network. I use tailscale for all peer-to-peer communications between my devices and I want to move away to headscale in future.
• maddy - all in one email server. I currently use Fastmail, but I want to go with a self-hosting solution.

If you’ve found any interesting software that I’m missing out on, please let me know.