aksh.dev

Book Review: Designing Data-Intensive Applications

Sun, 29 Dec 2024 22:00:00 +0530

⭐⭐⭐⭐ | Mandatory reading for every engineer

After seeing major recommendations from a lot of respected folks¹, I finally decided to read DDIA this year. And right off the bat, I’ll just reaffirm many of the things I heard/read — I agree that it should be mandatory reading for engineers. I also agree that it should be read multiple times, once every few years as you progress through your career. I for one feel quite unlucky not to have come across this gem of a book sooner, but at the same time really glad that I did now!

For me the greatest value the book provided was in the form of formalization & reaffirmation — as a Tech Lead with almost 6 years of professional experience working with distributed systems, albeit more on the user end of the spectrum, I’ve collected a lot of incidental knowledge that typically comes with such experience. And reading through DDIA, one of my main observations was that Kleppmann does a really great job at formalizing this knowledge. Working at the same organization for the past few years, I have left a long trail of decisions in my wake. For me DDIA provided great validation for some of those decisions — giving me more confidence in my ability in making them, often with a healthy dose of “Whew! I sure managed to dodge that invisible bullet!” In other words, DDIA helps you reflect on your knowledge and decisions through a very clear lens of formalized knowledge.

DDIA helps you reflect on your knowledge and decisions through a very clear lens of formalized knowledge

If you’re someone with relatively less experience, a lot of things might not click the way they would if you had more experience, but just trust me when I say that you’ll be much better off knowing them now than later. The fact that the concepts & knowledge I picked up from the book have already helped me multiple times in the past few months, a couple times in very critical situations, should be evident enough. And I’m fairly positive that they’ll continue to do so for years to come.

DDIA also teaches you what you should beware of — acting as a sort of a reference of things that can and will go wrong when working with modern systems. This aspect of it is invaluable in my opinion. Imagine being able to spend just a few hours of your life to gain knowledge of things that folks only get to learn over years, often through ending up in very tricky situations. This knowledge allows you to create and improve a personal framework for architectural decision making.

Imagine being able to spend just a few hours of your life to gain knowledge of things that folks only get to learn over years

Another impact it had on me was igniting an interest in the construction and internals of distributed systems, such as databases, brokers etc. I have a feeling that this is likely to have a major impact on my career trajectory in the near future. It also widened the sphere of my understanding — I recently attended P99 Conf, and was pleasantly surprised by how much of it I was able to absorb; the same goes for Dist Sys blogs I’ve been reading recently.

All this said, the book isn’t perfect in every way — at times the focus seems to deviate from things that most require it — but that might just be an artefact of my personal experiences and preconceptions. I also found it to be a demanding read in some sections, either by virtue of trying to cram too much into too little, or at times the structuring of contents requiring tracing too many references in order to grasp some concepts. Chapter 9 comes to mind as the main offender in this regard². But note that this is only relative to the rest of the book — DDIA is by far the easiest and smoothest to read technical book I’ve read. Kleppmann (along with Chris Riccomini) is working on the second edition³ currently, and in addition to being excited by all the new concepts that’ll be covered, I’ll also look forward to some of these being addressed to make for an even better book.

… inviting you to break through the traditional approaches of designing distributed systems and give a fair consideration to the alternatives, lest they be missed in the noise

Another major goal that Kleppmann seems to accomplish is expanding your worldview on how to view and design distributed systems, inviting you to break through the traditional approaches and give a fair consideration to the alternatives, lest they be missed in the noise. I suspect his ideas played an important role in the development of things such as AtProto⁴, indicating the continuing achievement of this goal. The discussion about ethical considerations in the end was also a welcome surprise: Kleppmann didn’t have to, but he still did, which I respect and admire. It will push me to consider the consequences of the things I build or help build in a more holistic manner, and also give some bandwidth to reflecting, both internally and externally, on such consequences in my body of (future) work.

As I hinted at the start — DDIA is definitely the kind of book that gives you more every time you read it, the most deserving of this claim, at least of all the books I’ve read. I’m certainly planning to revisit it in a couple years or so. And I think you should too.

Thanks for reading! Feel free to share this with others on the butterfly site, the elephant site, the ~~bird~~ unicode character site, or maybe even the orange site?

A very special thanks to Phil Eaton — he played a major role in reigniting my passion for reading via his book club, while helping me discover some iconic books. Make sure to check it out and join future runs! ↩︎
Murat describes this well in his review of chapter 9: “The chapter tries to do too much. Almost an entire semester of distributed systems content is force-jammed into this chapter … For a more manageable load, much of this chapter should be moved to other chapters.” ↩︎
Chris Riccomini on LinkedIn: Designing Data-Intensive Applications, 2nd Edition ↩︎
Kleppmann posted about this on Bluesky, and has also co-authored the paper Bluesky and the AT Protocol: Usable Decentralized Social Media. As of writing he is serving as an “Advisor to the Bluesky team” (taken from his Bluesky profile description). ↩︎

About me!

Mon, 02 Sep 2024 22:00:00 +0530

Hey there! Who am I? I’ve somewhat figured it out, so I’ll give it a go …

I’m a 27 year old male; software engineer by passion and profession. When I’m not watching a talk from a conference or ~~wasting time~~ keeping in touch with the latest in tech on ~~twitter~~ X, I’m chipping away on my rust learning path. And I’m usually listening to music. Professionally, for the past few years I’ve mainly been working on developing backend frameworks for the JVM while maintaining some microservices with cross-cutting concerns. Currently I’m leading the backend platform team at 99 Acres. Oh, and I got married to the love of my life in 2023!

In the past I’ve worked on Android Development, both in java & kotlin and liked using python (read: writing pythonic code) or kotlin (read: writing idiomatic (partially?) functional code) for some other stuff. I love customizing linux (I use arch, btw) and android, although I get to spend lesser and lesser time doing it. As my career has progressed, I’ve found myself venturing more & more into the PKM (Personal Knowledge Management) rabbit hole, and currently I’m in the process of switching over from Notion to Obsidian.

My drive to develop is to create software that makes something easier for someone, somewhere. I pride myself as being an active member of the developer community and like giving back whenever and however I can. I value best practices and believe that a programming language is more than just syntax: it’s a philosophy, a way of thinking.

Interested? Discover me @ GitHub and Catch me @ ~~twitter~~ X, bsky or linkedin!

The Joy of Hackable Tools

Sun, 21 Jan 2024 18:48:43 +0530

Discovering how the joy of hacking weighs in favour of tools that allow it

I just got my first neovim related MR merged!

#FeelsGoodMan

This got me thinking about the joy that is hackable tools: tools that allow you not just to customize, but to hack around and create stuff that you want. Customizability is becoming more and more an expectation these days, even if only at a very elementary level: it’s uncommon, for example, to find a website that doesn’t have a dark mode toggle. And I have really enjoyed customization with a lot of things, such as my android phone¹, my OS (arch, btw), my terminal², my browser³, my editors, my shell, the list goes on.

However, customization can only take you so far. Here I was customizing lualine for my neovim setup, excited about adding Code::Stats xp⁴ to it for a light dose of self gratification. But alas it wasn’t to be. After banging my head against some plugin configurations, it became clear that it was more than just a configuration issue. I was up against a bug.

Remembering lazy’s dev directory option from ThePrimeagen’s neovim revamp stream⁵, I decided to take a look under the hood of the codestats plugin⁶ itself. After tripping myself up by adding both the short name and the directory where lazy had cloned it by default (yep, it loops), I finally got a print statement in the plugin code to show up in neovim. Awesome! A few print statements later, I had the bug cornered, and a couple minutes later I had submitted a PR⁷.

The barrier to entry: A single line change

This little incident brought joy to me. The barrier to entry was actually that low: just a single line change and you’re off to the races. All of my lua knowledge was just the Learn X in Y minutes page⁸ that I had gone through quickly before starting my setup. And getting my hands dirty with the plugin code made neovim plugin creation seem so much more approachable. If I wanted x feature I could just write my own plugin, et voilà, the feature is ready!

Getting pretty good at lua

The debate on whether switching to vim et al is going to improve your productivity, or just lead to time being wasted in setting it up, is timeless. Even though I myself had been tending to the former side of this debate already, the thing that finally tipped me over was an unexpected one: the joy that comes from being able to get into the code that makes your tool work and change how a part of it functions. This joy is highly underrated in my opinion, and makes the debate entirely one-sided, at least for me.

******

I had been hearing a lot about Obsidian. More and more people I tend to agree with or look to for inspiration had been talking about it. I can’t place the tipping point exactly, but it was probably a mix of No Boilerplate’s pitch⁹, jonhoo’s usage of it¹⁰ and kepano’s philosophy¹¹. But there was the thing of Notion hosting all my current knowledge-base, work journal and task management. So when the official migration plugin¹² was released, I decided to take the plunge.

Obligatory Obsidian Graph Screenshot

I exported my workspace, selected it in the migration tool, and gleefully looked at all my notes getting translated and The Graph™ populating. However, once I went to see a couple notes, all was not well. My ToC were gone, and so were the intricately chosen emojis for each page. But most importantly, all my code blocks were weird and unhighlighted. I was taken aback for a moment. It was not to be after all. I remembered seeing a github link for the migration plugin, and went to see if there was an issue for this. Sure enough, there was one¹³, but it looked like it was already resolved. After double-checking the versions etc., I submitted another one¹⁴. Now to wait.

And then it clicked. It’s an open source plugin! I don’t have to wait, I can just fix it myself! Hmm, it’s typescript. I have coded all of 0 lines of javascript in my life. But maybe I can still give it a go? I just have to find and fix a bug. And the plugin being well-written, with all steps being nicely structured as separate functions, inspired some confidence. But now I’ll have to set it up :/ I don’t want a bunch of node modules on my laptop, eww. But it’s 2023, surely I can just open it up in some cloud IDE? Maybe codespaces won’t bug out this time and actually successfully open a project for once?

It did. A few minutes of tinkering around (it wasn’t a couple hours at all), and I had figured out the commands and the built package file and where to place it to make it work in my obsidian. A couple iterations of change + build + copy-paste later, I had the code blocks formatting fixed.

That joy again! The possibilities were endless: even though not that simple this time due to my complete lack of knowledge of the JS ecosystem, I could hack around and fix anything I wanted, or even add features to make things work the way I wanted: which is exactly what I did over the next few days¹⁵. Even though my MR was rejected¹⁶ in favour of the actual Notion migration contributor’s much more elegant and sensible fix¹⁷, I went on to add support for random things that bothered me about the imported notes¹⁵.

Official Version vs My Hacked Version

Again, there is the debate of whether you should get into Obsidian, with the added overhead of setting it up with plugins that might go unmaintained any day and overpriced or over-complicated syncing, or just stay with Notion and focus on doing your thing instead. Here it’s easier to tend to the former side for me. But when I went through the experience of hacking away and adding a bunch of features that resulted in an importing experience that was arguably an upgrade from Notion even visually, there was no doubt left.

Imported note before hacking | Original Notion page | Imported note after hacking

And even though right now I’m writing this blog in Notion (sorry for not giving you a perfect ending¹⁸), I’m going to switch away this year to Obsidian without a doubt. I’ve already enlisted my wife as a test subject (albeit with partial success) by forcing her to switch over from Evernote (eww) and Notion to Obsidian and setting up hacky syncing via dropbox. It’s even part of my resolution for 2024, so of course it’s bound to happen no matter what.

Can't let those 24 people down

But for now I’m going to proceed with my neovim setup, with part of me hoping to encounter a bug that I can just go ahead and fix and be on my way with a bunch of joy.

Thanks for reading, and don’t forget to spread the joy! Share this on the ~~bird~~ unicode character site, the elephant site or maybe even the orange site?

A_K_S_H_ | XDA Forums ↩︎
Cycling tabs in recently used order · kovidgoyal/kitty · Discussion #3720 · GitHub ↩︎
userchrome, anyone? ↩︎
Code::Stats ↩︎
Neovim Config - Part 1 - Lazy Vim - YouTube ↩︎
GitHub - liljaylj/codestats.nvim: Code::Stats plugin for Neovim ↩︎
Fixed get_xp & get_level by aksh1618 · Pull Request #3 · liljaylj/codestats.nvim · GitHub ↩︎
Learn Lua in Y Minutes ↩︎
Hack your brain with Obsidian.md - YouTube ↩︎
Hardware and software [2024 edition] - YouTube ↩︎
File over app — Steph Ango, see also: Obsidian’s manifesto ↩︎
Free your notes - Obsidian ↩︎
Notion code blocks are imported without line breaks · Issue #82 · obsidianmd/obsidian-importer · GitHub ↩︎
Notion code blocks still being imported without line breaks · Issue #107 · obsidianmd/obsidian-importer · GitHub ↩︎
I got so excited that I went completely overboard and actually created a release! Release 1.3.1-aksh-0.1.0 · aksh1618/obsidian-importer · GitHub ↩︎ ↩︎
Notion: Fix codeblocks being imported without line breaks by aksh1618 · Pull Request #108 · obsidianmd/obsidian-importer · GitHub ↩︎
fix newlines in notion codeblocks by joshuatazrein · Pull Request #118 · obsidianmd/obsidian-importer · GitHub ↩︎
I did end up editing it in Obsidian before publishing, so I guess there was a happy ending after all! I wanted to go for footnotes instead of links for this one to enhance readability, and a couple of Obsidian plugins came in really handy ↩︎

Developing an Android App in 2019

Sun, 17 Mar 2019 13:48:43 +0530

Embracing the 'new' Android (or how the Bitotsav '19 app became a reality)

Background: Pantheon ‘17 ⏪

Almost two years ago, in September 2017, a friend, Ashank Anshuman convinced me to work on an app for the technical fest of our institute. We worked for about two weeks, day and night, getting it ready for release just in time for the fest. Although we were exhausted, it was an amazing feeling, getting something ‘out there’, in production, that people actually used! It served its purpose perfectly well, helping the organizers easily convey everything to the participants.

Pantheon ‘17 - Apps on Google Play

In its endeavor to provide a national platform for the youth to showcase their technical skills; displaying…

play.google.com

It was rated 4.9 with around 120 reviews, which Google’s bots removed for some reason, but that’s a story for another time. We received some requests to share the source of the app, but we declined, due to many reasons - but mostly because we weren’t that satisfied with the code, especially the parts rushed through near the end. We simply didn’t have enough time and experience to write code good enough for people to learn from and/or use.

Here We Go Again! 🙃

Fast forward to November 2018: Ankit Agrawal (he’s that ‘fest guy’) asks me to join the team for Bitotsav, our annual socio-cultural fest, to which I agree, as I was looking for some excuse to revisit Android. This time I convinced Ashank (took a lot of convincing!) to work on the app.

We didn’t do much in December, but I did start reading on things like Architecture Components, AndroidX, Jetpack etc. I had also been getting familiar with Kotlin over the past few months, with a couple courses being instrumental: A two-part course by the one and only Hadi Hariri and another more recent one by Svetlana Isakova and Andrey Breslav (which he introduced in KotlinConf 2018). Kotlin was thus the obvious choice for the app.

Introduction to Kotlin Programming

Kotlin 1.0 was released in February 2016, and since that time it’s been embraced by developers around the world…

shop.oreilly.co

Kotlin for Java Developers | Coursera

Kotlin for Java Developers from JetBrains. The Kotlin programming language is a modern language that gives you more…

www.coursera.org

Decisions 🦉

The first half of January also went by with not much code being written, as I was preoccupied and couldn’t reach the college till 16th Jan. We did, however, make some major decisions:

Use Kotlin exclusively
Use Feature-based packaging
Use Jetpack Architecture Components with AndroidX
Use API 21 as Min API (22 might’ve been a better choice)
Use Android Studio Canary
Use Git Flow & SemVer
Write code good enough to make it public after the fest 🙈

Jetpack with AndroidX

So basically, a hard reset from all the experience of developing an app in 2017 to the bleeding edge in 2019. It was really exciting, but also a big challenge.

Code Code Code! 💻

We decided that Ashank would take care of the Backend of the app (DB & Networking, Notifications with FCM, Background Processing) and I’d take care of the Frontend and Integration, just like we did for Pantheon ’17. Many resources came in handy while getting started and as we worked, but the best by far were these awesome codelabs offered by Google:

Room with a View - Kotlin (ViewModel, LiveData & Room with Coroutines)
Using Kotlin Coroutines in your Android App
Data Binding Codelab
Navigation Codelab: (Navigation Architecture Component)
Background Work with WorkManager

Google Codelabs

Google Developers Codelabs provide a guided, tutorial, hands-on coding experience. Most codelabs will step you through…

codelabs.developers.google.com

Also, the Sunflower & Google IO 18 apps by Google were ideal codebases for the purpose of reference. Android Dev Summit app would also have been a good source for reference, had I known about it before!

googlesamples/android-sunflower

A gardening app illustrating Android development best practices with Android Jetpack. - googlesamples/android-sunflower

github.com

With these in our arsenal, we began coding. I decided to use the new Navigation architecture component to implement a single-Activity app architecture. Ashank started with Room and FCM. I had also been thinking about using Koin for IoC, but wasn’t so sure.

Incidentally, Joe Birch launched a Koin course right around that time on caster.io (Features small, to the point courses by professionals, each free for a week on launch!), and decided to go with it. No regrets there! Seriously, the android support is amazing and the documentation is fabulous ❤️

Koin

In this course, we’ll be learning about a dependency injection framework known as Koin by building a fully functional…

caster.io

With Navigation and Koin setup, I began with the UI, deciding was to use material design components exclusively for UI for which the guidelines and docs came in handy. Also, I had to use Data Binding because I love it! Meanwhile Ashank implemented WorkManager, which we decided to use instead of Firebase Job Dispatcher, going full Jetpack!

The first thing I developed was the Schedule UI, which helped me get comfortable with Architecture Components. With that done, I moved on to the Registration-flow UI, probably the most complex part of the app, which featured advanced usage of LiveData and Navigation for implementing three steps with live-validated forms (Worth a blog post of its own, coming soon?!). This made me a lot more confident in these components, and it was a smooth ride from there on. We completed the planned features, discovered some gotchas, fixed some bugs.

Launch 🚀

With major features implemented, we did some UI brush-ups, completed some final TODOs, and were ready for launch! As a finishing touch, I added something I had been planning for all along:

A different color theme on every configuration change! This was done to compliment the theme of the fest: “Colours of Asia”

The app was live on the Play Store on Feb 11, 2019! 🎉🎉

Bitotsav ‘19 - Apps on Google Play

Bitotsav ‘19 the 29th edition of Birla Institute of Technology, Mesra’s annual socio-cultural fest is all set to take…

play.google.com

Bitotsav ’19 App

Fixes & Updates

We faced (the only!) two bugs within hours, which we fixed right away. The first one was related to DAO methods being marked suspend, but I’m still not a 100% sure why exactly it was happening 😅. The second one was caused by obfuscation causing a serialization failure, and was easily fixed with a Keep annotation.

Then I started working on the next update, in which I added the leaderboard in the feed and the night events for the fest in the schedule, along with some other changes. A third update followed adding some more minor features.

The fest went well, and the app was used by over 1000 participants!

We did face a minor issue due to an incorrect DB entry in the servers by our friend Sushant Gupta, who later proceeded to write a rather dramatic blog post on the same.

DDoS Attack on Bitotsav ‘19 Website

This is not a technical write up. This is a story that I want to share which might be a lesson for several Web & App…

cs.sonudoo.com

After the fest, we released a final update, storing the event details, feed etc. as JSONs as part of the app and redacting the contact numbers of the organizers for privacy reasons.

Going Public!

It was time to open source the code! This time around, we had taken care to write understandable code, and it was ready for the world. I prepared a slick README, and to remove the contact numbers from the history of the repository, we used the amazing BFG Repo Cleaner tool.

The code for the Bitotsav ’19 app is now public, for anyone to review, refer to, learn from or use! Check it out and don’t forget to leave a 🌟 😄

aksh1618/Bitotsav-19

Official app for Bitotsav ‘19. Contribute to aksh1618/Bitotsav-19 development by creating an account on GitHub.

github.com

Challenges 🧱

We did face some challenges during the development:

Time Limitations: The main challenge we faced was of having a very limited time to learn very new concepts and use them to create an app to be used by hundreds. This time constraint led to a lot of continuous hours being put in, leading to stress and fatigue, but we were able to power through and deliver!
WorkManager with Coroutines: On the technical side of things, we faced some minor challenges with WorkManager and Coroutines, but were able to overcome them. Hoping for a better support for coroutines throughout the Android SDK as development continues 🤞.
API 21: We chose min API 21 to avoid having to adapt everything to work on older versions, as most android devices are on API 21 and above anyway. But surprisingly, some things refused to work on API 21, especially view backgrounds. It was really frustrating, making me wish we had set min API as 22, even more so when we found out that the app had been installed only on two API 21 devices: the ones we had tested on 😿.
Lack of Devices: Another challenge we faced was not having enough devices to test on. During Pantheon ’17, we had a hostel with around 200 people, and thus testing was easy. This time around most people were gone for internships, so we were left with relying on having confidence in the code!
No code reviewers: A lot of what we used was new to us, and we did our best to ensure we were doing everything right. But having a reviewer would have been very helpful. Even now, if you feel like you could do a quick review of the app’s code, we’d be very thankful!
The Apocalypse: We did also face that ‘ DDoS ’ issue, read Sushant’s blog post linked above, you won’t regret it!

Takeaways ✅

Kotlin + Jetpack = ❤️ : The main takeaway was that Android Development has come a long way and with Kotlin and Jetpack, it’s definitely a lot more fun and purely joyful! Seriously, there were multiple orgasmic moments during the coding process!
Nothing is Impossible: Kinda cliché but true: if you have the will to work hard, you can do anything, no matter how difficult. Sure, there will be stressful phases, but keep powering through. Just believe in yourself!

Regrets 😅

No Instant App / App Bundles: We just straight up missed this one. Didn’t even think about it. Ah well, maybe next time.
No tests: I know, this is a big one! Having proper tests in place could have helped us a lot, but due to the time constraints, we decided not to write tests until a ‘later’ time, which hasn’t yet come 🙈.

TL;DR. 📗

Starting with an app with 2019?

Use Kotlin & Coroutines

Learn Kotlin and use it exclusively!
Learn to use coroutines with Android.

Use Jetpack Components with AndroidX

Learn to use Navigation Component for Single-Activity Architecture.
Learn to use Lifecycle Components for UI & Room for persistence.
Do yourself a favor and use Data Binding!
Learn to use WorkManager for background processing.

Use Material Components

Write Tests!

Well we couldn’t but you definitely should! Do NOT skip tests.

Refer to Source Code

.. of apps that do these things: Sunflower App, IO App, Dev Summit App or, of course, the Bitotsav ‘19 App! (also, leave a star 😛)

Stay up to date

Subscribe to blogs and newsletters to stay up to date! Here are some to start with: Android Weekly, ProAndroidDev, AndroidPub, Kotlin Weekly. Overwhelmed? Check out this awesome talk by Huyen Tue Dao:

Be Like Water: Keeping up with Android

If you enjoy talks from 360 AnDev, please support the conference via Patreon! The one constant of working in mobile is…

academy.realm.io

Well, there we go. This is a great time for Android Development, so go get started with your new app, and don’t forget to have fun while doing it!

Thanks a lot for reading, tweet at me 🐦 if you learned something, constructive criticism welcome 😄