Learn Building Modern Go applications

The Distributed Monolith Trap (And How to Escape It)

Wed, 28 Jan 2026 16:00:00 +0000

Quick takeaways

Start with a monolith - don’t create microservices from day one; the overhead isn’t worth it until you have real pain points
Microservices solve human problems, not just technical ones - they help teams work independently, not just scale systems
Tight coupling over HTTP is still tight coupling - separating services doesn’t automatically give you isolation
Watch for signals - lots of calls between services or frequent changes across multiple services suggest wrong boundaries
Sometimes joining services is the answer - merging tightly coupled microservices back together can be the right move

Introduction

We talk about splitting software: when to do it, how to avoid common pitfalls, and why microservices aren’t always the answer. We share stories from past projects where we jumped on the microservices hype train without thinking through the real problems we were trying to solve. The key insight is that splitting software is more about organizing teams and maintaining good boundaries than about technology.

Notes

The Domain Engineer training - includes a complex modular monolith example
Modular monolith approach - starting with a well-structured monolith that can be split later if needed
Event Storming - a technique for discovering system boundaries by mapping domain events
Conway’s Law - the idea that systems mirror organizational structure
Team Topologies book by Matthew Skelton and Manuel Pais - recommended for understanding how team structure affects software design
Reverse Conway Maneuver - designing organizational structure first, then letting software follow
Service mesh tooling (like Istio) - for observability between microservices

Quotes

This aspect of microservices I think can be also misleading. Because it seems like you have these separate things. You see the deployments, you see the different repositories. For an outsider, you can think they are separate beings that can be changed independently of each other. It can be a trap.

Miłosz

You can say that microservices are maybe solving a bit of technical problems, but it’s mostly solving the human problem.

Robert

So if you consider splitting your project, I would start with this question: if you even need to do it. Because you often see people struggling or hear them asking, I’m starting a new project, kicking off some application, and what services do I need to create? So probably at this stage, you don’t need to consider this at all.

Miłosz

I think it’s also nicely visible if you take your service mesh tooling, and probably will see that between some microservices will have a lot of calls going back and forth. And it’s usually the sign that maybe those two services shouldn’t be separate.

Robert

The more developers you add to the project, the more complicated it gets. Because this is also important factor. If you are a single person working on software like this, it will be fine probably because you can keep the context in your head. You will refactor to whatever you want. But if you have like 20 people working on it, it starts getting more complicated.

Miłosz

But there’s one good thing about microservices is that it’s forcing the boundaries really hard. So it’s hard to go over network if it’s not possible.

Robert

Timestamps

Transcript

Miłosz [00:00:00]: So, if you consider splitting your project, I would start with this question: if you even need to do it, because you often see people struggling or hear them say, ‘How do I start?’ I’m starting a new project, kicking off some application. What services do I need to create? This aspect of microservices I think can be also misleading. Because it seems like you have these separate things. You see the deployments, you see the different repositories. For an outsider, you can think they are separate. Entities separate beings that can be changed independently of each other. It can be a trap. You have to think: what problem do you want to solve? This way, rather than just ‘Oh yeah industrial best practice.

Robert [00:00:41]: Probably, can say that microservices are maybe solving a bit of technical problems, but it’s mostly solving the human problem. Probably, you’re mentioning the monolith first approach that we are big advocates. We’ve tried it multiple times for products that become bigger later. If done properly, it’s working. It’s accelerating you a lot at the beginning and allows you to change it later.

Miłosz [00:01:05]: And what’s surprising to me, it’s not as popular.

Miłosz [00:01:11]: So, welcome after break. Let’s talk about splitting software, different ways of splitting software, and why it’s important in the first place.

Robert [00:01:20]: Yeah, I think it’s a good topic because, probably, it’s the topic that everybody writing software in this or another way needs to know how to do that. And if it’s not done properly, it may create a lot of issues.

Miłosz [00:01:37]: Probably even if we end up doing coding with AI in the next couple of years, it will be an important topic because it’s like one of the key architectural decisions you can make in your project.

Robert [00:01:50]: Especially, from our experience, even if people were doing that, usually they were missing that. And I would not expect that some cloud code or something like that.

Miłosz [00:02:00]: Handle it better than probably the opposite, actually, because there are so many anti-patterns on the internet. You know, like people tend to make mistakes when splitting software or do it in a very naive way. So yeah, I would expect coding agents to probably make similar mistakes.

Robert [00:02:20]: I think it’s you know not only about some inexperienced people—I think also sometimes very experienced people, without experience, may miss some signals from the systems. So when you’re building some system, it’s sometimes hard to miss some signal when something is going in the wrong direction. And it happens even for those experienced people.

Miłosz [00:02:42]: So it’s often the issue with big codebases, right? When we work with a big project, over time tends to be more difficult to work with, especially with bigger teams or multiple teams. And then the topic of splitting comes up. And I was thinking about, you know, our past projects on different companies, how important the problem is. And one thing I realized is that when we were working on our own projects before our full-time jobs, the stopping wasn’t really that important. Important so it tells me something right, that’s even if those applications were sometimes fairly complex, wasn’t. If it’s just you and maybe another person working on some project, it might still be fine sticking to one repository or codebase.

Robert [00:03:45]: Yeah, so probably can explain why, for example, people like Peter levels can have one PHP file that has 10,000 lines of code and he can work.

Robert [00:03:55]: But if you are two, also probably. But if you have people of five, ten, twenty, fifty people, it no longer works.

Robert [00:04:05]: When some application is done as proof of concept, later it evolves to some product that is making money. Sometimes it’s missing this. Transition phase from the application that was built as proof of concept by a team of five— even if okay, if you have team of five, it’s still not a big issue to handle, even very bad code, and be able to extend that. But when the team is growing, the product is growing, the number of features is growing, well, we start to have a problem. And the question is, what you do about that.

Miłosz [00:04:39]: Yeah, so if you consider splitting your project, I would start with this question: if you even need to do it. Because you often see people struggling or hear them asking, ‘How do I start I’m starting a new project, kicking off some application, and what services do I need to create? So we are probably at this stage, you don’t need to consider this at all. Right, because it’s not not the pain point, and the overhead may be big enough that it will not pay you back.

Robert [00:05:14]: So you probably are mentioning the monolith first approach that we are big advocates of. We tried it multiple times for products that become bigger later, and if done properly, it’s accelerating you a lot at the beginning and allows you to change it later.

Miłosz [00:05:32]: And what’s surprising to me it’s not that popular, or maybe now it’s probably a bit more popular than it was. I i see it mentioned here and there, but back back when we first tried it, I think no one talked about it that much.

Robert [00:05:45]: Yeah, but I think it’s probably still some small percent. And it’s maybe also a bit because of our bubbles. Yeah, maybe. But it’s still not.

Miłosz [00:05:55]: It’s not like a crazy different idea from creating microservices.

Robert [00:06:00]: But if you ask a random developer, what do you hear about? Kubernetes or modular monolith? I’m pretty sure I know the statistics.

Miłosz [00:06:08]: Yeah, the hype is a funny factor here. So I remember in one project—when I joined— we had one huge codebase.

Miłosz [00:06:20]: Typically, that’s the software no one wants to work with, and everyone would like to see refactored. And it was also around the time of the microservices hype era. So somehow we made a decision together as a team that we will split the monolith, as probably everyone back then did.

Robert [00:06:40]: Yeah, I remember how everybody was optimistic because it was like, ‘whoa,’ microservices. Finally, a solution for all of our problems.

Miłosz [00:06:49]: And when I think about it now, it’s funny how we had all these issues with this project and we assumed, ‘Okay, microservices will solve this for us I mean, the project was better in the end for sure, and I was responsible for the design of the framework and all the networking and stuff. So, as a programmer, it was very fun. It was great. Probably we followed the

Robert [00:07:16]: ‘fun’ principle too often.

Miłosz [00:07:18]: Yeah. But looking back at it, did it need all this complexity and distributed systems stuff? Probably not. Probably some parts could be simplified.

Miłosz [00:07:31]: But I think we just jumped on the hype train back then.

Robert [00:07:37]: But I think a good lesson from that was that you, as a team, saw some signal that ‘okay, there is something wrong with this project,’ probably it was like, ‘okay, we cannot maintain it, we cannot add new features easily.’

Robert [00:07:51]: So the problem was visible there.

Miłosz [00:07:54]: Right. So the question is whether splitting software is a good approach to fix these kind of issues.

Robert [00:08:02]: But do you think that maybe what would work better there was maybe a bit more balanced approach— so maybe a bit less microservices or staying as a monolith— was better.

Miłosz [00:08:16]: Maybe less microservices or modular monoliths for the start. Because the issue for sure was that the boundaries were wrong in the initial software, so there was, you know, some function calls all over the place from one place to another with no control. The more developers you add to the project, the more complicated it gets. Because this is also important factor. If you are a single person working on software like this, it will be fine probably because you can keep the context in your head. You will refactor to whatever you want. But if you have like 20 people working on it, it starts getting more complicated.

Robert [00:08:57]: But still, I think it’s still manageable with modular monoliths.

Miłosz [00:09:01]: Yeah, for sure. Yeah, but the issue is if you don’t have these good boundaries, you don’t know how to progress. Everyone tries to do something and it ends up as this big ball of mud.

Robert [00:09:14]: So in other words, you are modifying something in one place and everything around it goes up and you have no idea why. It should have zero dependency, but it seems that it does.

Miłosz [00:09:27]: Yeah, it’s risky and you don’t like making changes there.

Robert [00:09:30]: So the big bow of math or spaghetti code by the right, right.

Miłosz [00:09:34]: So if you have these good boundaries, as microservices, promised everything will be fine because you make these changes in one small Please.

Miłosz [00:09:45]: There’s low impact, you can deploy it independently from the other services, everyone’s happy, which makes sense, but you can’t do the same thing without microservices.

Robert [00:09:56]: But there’s one good thing about microservices is that it’s forcing the boundaries really hard. So it’s hard to go over network if it’s not possible. I mean, when you have a big ball of mud, you can call everything if you don’t have proper encapsulation. Okay, you can do proper encapsulation, but it requires some kind of regime of hardware. Or tooling, at least. Yes, yes. And with microservices, as long as you will not get exposed public endpoint, it’s not. So it’s probably showing why sometimes microservices may help. And have the same outcome with more cost, obviously. But again, if you go too far, it’s much harder to undo that because, for example, if you were wrong with modules in modular monolith, you can copy from one catalog to another, join everything. We have one module with microservices. You need to change your Kubernetes configs. You need to change all dependencies in other team services.

Robert [00:11:02]: It’s a mess, right?

Miłosz [00:11:03]: So maybe it’s easier to see the microservices as well because you can show someone, ‘Yeah, this is our Kubernetes deployments. They can see, okay, yeah.’ So this is your, those are your modules or microservices. The monolith is more complex because you need some kind of structure. It’s still possible, and we did that, but maybe not as easy to see.

Robert [00:11:28]: And I think there is also one thing that microservices may make easier out of the box. So this is all sort of observability, because you can put some Istio or whatever between those services and you have it kind of out of the box again, not for free. Yeah, with much less flexibility, because Again, it’s just put in the place by some cloud ops, DevOps team, whatever. But probably the flexibility is a lot limited. Right. From other side. You can achieve the same thing with your module monolith, but it requires some effort because you need to write it in code and you need to maintain that. other side, for example, I like this approach because it’s giving you a lot of flexibility. And it’s also removing some external dependency on the team that, in my opinion, should be more about running some transparent infrastructure rather than adding something like that. But from other side, it also depends because, for example, for some big enterprises, super big products, it kind of makes sense because you can have it out of the box and have it unified everywhere.

Robert [00:12:36]: So again, it depends on what kind of company we’re talking about.

Miłosz [00:12:40]: Right. So microservices are kind of more tangible. You have better observability. You can see them. And modules are more abstract. You have to more understand the project to see them. Yeah.

Robert [00:12:54]: But yeah, trade-offs. We are mentioning that to show that there is not only one way, because I think now the way with doing it with Kubernetes, with some service mesh, it can be a way. Definitely for some bigger companies. But again, if you’re working on a startup, maybe not— maybe you can do it by hand and it will be much more flexible, much more expensive.

Miłosz [00:13:16]: But this aspect of microservices, I think, can be also misleading because it seems like you have these separate things, right? You see the deployments, you see the— you know.

Miłosz [00:13:30]: different repositories. From an outsider, you can think they are separate entities, separate beings that can be changed independently of each other, but it can be a trap. I remember one project where, when we joined, they were described as independent and isolated from each other. So you can change one thing, it doesn’t impact the others. But it turned out the kind of isolation layer was HTTP API between them. So, on the outside, it looks like nice, clean separation. But in reality, they are still very tightly coupled. They just communicate over the network instead of function calls. I remember this confusion we had and trying to explain it to the management as well. Why this nicely separated system is suddenly a problem?

Miłosz [00:14:34]: Because from the outside, it looks like you have those tangible, separate microservices. The reality is much different.

Miłosz [00:14:43]: That’s also one of the hard parts: how to know. If the isolation is good enough.

Robert [00:14:51]: Yeah, and I think it’s often outcome of, let’s say, simplistic approach to splitting those services. So I would probably see two cases. One. We’re getting database tables and we have separate service for each database table. Hooray, microservices.

Miłosz [00:15:07]: I often see this when someone asks us, on Discord or whatever, for some feedback on a system they did. And very often it’s something like, ‘So I have this application. And I have users microservice and orders microservice. And at this point, I know this won’t be easy.’

Miłosz [00:15:31]: For most e-commerce systems, you can just separate users out of the orders, or it will be, you know, there will be some tight coupling probably between them anyway, as long as you’re not building some extremely big yeah, exactly. So it depends, but you know, if you take a fairly simple system, and you have those three main entities, and you think each entity should have their own microservice. Then you will probably end up with many problems because just separating those services won’t give you isolation. It will just give you this illusion of isolation.

Robert [00:16:13]: Yeah, and I think it’s also nicely visible if you take your service mesh tooling, and probably will see that between some microservices will have a lot of calls going back and forth.

Robert [00:16:26]: And it’s usually the sign that maybe those two services shouldn’t be separate. For example, those users and orders, those orders. So you can see that.

Robert [00:16:40]: To do an update to order or to do some shipment stuff or whatever, you need to call users many, many, many times.

Robert [00:16:50]: Can see that if, for example, you would need to add some field to, probably it can be customers, maybe not users in this case. So let’s say orders and customers. Commerce. And let’s imagine that orders is handing the shipment or something like that. And you can see the Skype link nicely when you have some new. For example, you need to add to user something that needs to be added to shipment. And in worst case, if it’s handled by two teams, a project that could be done in one day with modular monolith is starting to take one month because you need to go back and forth with two teams and agree on contracts and on everything.

Miłosz [00:17:30]: You have to wait for the other team to pick it up for the sprint or whatever they work with. So it takes twice as long at least.

Robert [00:17:38]: Yeah, yeah. And, you know, It’s often killing these small features that could be developed within one day or even if I code it now. Sometimes those features are that simple that it can be easily vibecoded. But with microservices, you no longer can because it needs to be orchestrated on a higher level. Maybe, you know, era of vibecoding will be era of modular monoliths. Hopefully modular. For sure, easier to deploy. But. And I think it’s also one important technique to mention when we are talking about these patterns, when we see that. So I would say that dark two signals when you should consider if something is wrong with us. One signal is when there’s a lot of calls between those two services to do some scenario. And the second thing is when you are modifying some functionality.

Robert [00:18:31]: It requires changes in two or many services multiple times. So you see this pattern over and over. And it may be a sign that it may be wrong separation of services.

Robert [00:18:41]: To solve that, there’s one technique that I think it’s very rarely mentioned because maybe it’s, I don’t know, some heresy for many people, but it’s the microservice station. So basically you’re getting those two services and joining them.

Robert [00:18:56]: And I know that for many people it sounds like a heresy. I hope it will change over the time.

Miłosz [00:19:02]: And yeah, it’s not never not easy, depending on how they are built.

Robert [00:19:06]: But Yes, but if you are getting into problem of changing those two services over and over and two teams that needs to do that, or even within one team, you need to always change two services and kind of synchronize how you’re deploying them. It may be worth considering if it’s worth investing in joining them. Because yeah, probably to introduce some costs, they need to do some kind of cost and income.

Miłosz [00:19:32]: Yeah, it’s a good thing you mentioned Teams because it’s also something I think you can’t skip when designing.

Miłosz [00:19:40]: Creating software, especially if you have teams working on the same project, maybe not the same code base, but microservices that work with each other.

Miłosz [00:19:53]: And if you have many of the services. I think once we worked for a company that had probably more services than developers or it was something like that, which sounds funny now, but I think it’s the reality for many companies actually. Because when it’s easy to spin up a new service, maybe you have great tooling and some platform team that handles it for you.

Robert [00:20:16]: It’s interesting risk, actually, of having great tooling for easily creating microservices. I mean, if it wasn’t easy, you would think twice before creating a new microservice.

Miłosz [00:20:28]: If creating that takes one minute, you end up with teams who own 10 or more. Microservices for very simple tasks sometimes I don’t know— maybe it’s fine. Sometimes, for things like keeping avatars, for example, so it’s like—overhead— but I think the most problematic story I remember from from this was where there were some services that had no clear owner, and we both our teams and other teams’ responsibilities kind of crossed in one area. That’s not really a technical issue—it’s funny, that’s like a team organizational issue. Mm-hmm.

Miłosz [00:21:19]: And it’s very difficult to figure out if you didn’t start with a good design of the system first, but you just, you know, added the microservices for this entity here. This entity here— this entity here— maybe this database table sounds like a separate service, and then you have to kind of— you know— create some teams and you assign them the services. Artificially. Yeah. So that’s not a good idea.

Robert [00:21:47]: And two years later, if somebody asks why those services are like that, it’s: ‘It was always like that.’

Miłosz [00:21:54]: Why you ask? Yeah. So we finally figured it out somehow, but it was very painful.

Miłosz [00:22:02]: Not only because of the technical challenges, but also all the discussions between teams, company politics, stuff like that.

Robert [00:22:13]: What I remember was useful there. So in this scenario, we used event storming for that to discover that. But you don’t need to use event storming. You can use any technique to see how the flow of the system is going. And I remember that it was quite funny to see that entire critical flow of the application was. So, for example, the beginning of the flow was owned by one team, later it was some small element owned by another team, it was going over again to the previous team. And it was a lot of some small pieces. Crossing the boundary. Yes, yes. And those were usually the biggest problems because if something didn’t work, it was sometimes not clear who is responsible. And it was also harder to change this law because if team didn’t have entire ownership of this law, they were just not able to do it without some cross-team coordination.

Robert [00:23:09]: Well, what we did there, the solution was quite simple. So it was looking on high level on this and trying to understand how we can change ownership.

Robert [00:23:19]: Historical ownership, and it helped a lot. And we added also to that a bit of the microservice station, and it helped even more.

Miłosz [00:23:26]: I thought about even storming when you mentioned service mesh visualization.

Miłosz [00:23:31]: So maybe instead of using this visualization after the fact, after you have the microservices in place, you could just have this high-level plan with even storming or not.

Robert [00:23:45]: Seeing how system works before it works? How?

Robert [00:23:49]: How is it possible?

Miłosz [00:23:51]: What kind of magic is that?

Miłosz [00:23:54]: Try to design it first and then see. But you know, it’s also not that easy, right? Because it’s not just a UML. Diagram where you have users, orders, and you see—okay, this is separate, but more like ‘yeah,’ maybe that’s why I would still recommend event storming because it focuses on behavior, not on the entities. So, if you map the events that happen, then the behaviors you can see there are some— those clusters of things that happen together— and it probably means they are a good place for some kind of boundary. And if you have, for example, two clusters that are joined by two events or something like that, you can clearly see, ‘okay,’ those things can be mostly separate and they won’t interfere with each other. They won’t be tied up.

Robert [00:24:53]: And I think what’s also pretty good there, compared to some UML diagrams, is the context of time. And because, when you are looking at UML diagrams, you cannot see the order of flow. Even storming is more so for people that don’t know how to it looks like. So TLDR is that it’s basically some brainstorming session when you’re putting sticky notes for domain events or whatever events that are happening in the system. And you don’t need to do event-driven architecture for that later. So it’s more for the high-level modeling and how you implement that later, whatever. And what’s nice, it’s putting the in time order.

Robert [00:25:32]: So basically, when you have flow, you can see when the flow is starting, when it’s ending, and it’s kind of in a magical way helping you to see how to do the split there. Anyway, I think it’s a big advantage compared to those UML diagrams or in general some diagrams.

Robert [00:25:52]: Flow diagrams probably are a bit better because they also go from left to right, but I think they are not readable for very complex systems. You cannot make a flow diagram for entire system. Don’t try. But with event storming, basically you can do super big systems on higher level, but you can see the big picture of it.

Miłosz [00:26:16]: So it’s a great technique if you’re just looking for the boundaries, or even if you have an existing system and you want to see how it could be done in a different way.

Robert [00:26:25]: If you would like to hear a bit more about event storming, we’ve been discussing it one episode earlier, two episodes earlier, something like that. That so check it out but okay let’s come back to splitting.

Miłosz [00:26:38]: So the nice benefit is of having everything mapped out on some whiteboard or you know some software. Is that you can look at it from different perspectives. For example, one thing you can do is try to differentiate core domains from the generic domains. So for example, you can see some cluster where there is more of your domain language of what your software is very specific. About, different from other software. This is probably your core part and it might be one cluster. And maybe there are some parts that are more supporting, more generic and they don’t need to be involved as much. So they don’t need to know all the details of your domain. Very often this helps you to simplify this part. For example, it can be some e-commerce software or some account.

Miłosz [00:27:41]: Mounting software you can just buy, or you know, wipe code, or whatever— you don’t need to build it yourself from scratch if you don’t want to.

Robert [00:27:51]: So, in other words, not all software in your product is equal. So some parts of software probably require a bit more of love.

Miłosz [00:28:01]: And the less they know of each other, the cleaner your boundaries and the simpler the software as well.

Miłosz [00:28:08]: So even if you build yourself this generic part, I would do it in a way that it could be reused in a different company.

Miłosz [00:28:18]: It doesn’t mean you need to do it in the end, but it often helps to keep these boundaries clean and just keep some translation between them. Hmm-hmm. Of course, it can also be overdone if you try to make it super generic or use cases. So you have to be careful about that. But in general, I would try to not let the domain terms spill out of the core parts.

Robert [00:28:48]: Yeah, I think it’s also some other direction that sometimes you can also watch out for with buying things because, it sometimes works that there are some services that are generic enough that can buy it, like sending emails, sometimes accounting software, but it depends. So sometimes when you are handling very specific things like marketplaces, you might not find accounting software that can handle it for you. I mean, for example, you are building some food delivery application. I guess you won’t find accounting software that will handle it for you. So it’s kind of interesting, because if you’re building this kind of application, it’s not your core domain, probably, but for other sites, it’s a very important domain. Unit also yeah— probably building a house, but parts of it will be generic, right?

Miłosz [00:29:42]: Like printing invoices, that’s probably something someone solved some time ago and you don’t need to write it from scratch. Or if you do, we can keep it very generic.

Robert [00:29:52]: Or maybe, for example, trying to understand what’s art. tax rate for each country is a good type. I mean, you can probably hire an entire team that is spending their entire time researching what the rates for every type of food or whatever that you can have in any countries. But probably you can just buy it from external company. And other techniques for event storming. So maybe not forever event storming, but I have also in mind one thing that you mentioned: sometimes you have parts of the system, some maybe services, maybe some libraries that multiple teams are kind of owning. And sometimes it’s on the pattern, but I think sometimes it’s also not, because sometimes it can happen. For example, from domain-driven design, we have concept of shared kernel, and it’s there for some reason. So sometimes you have some parts of code that two teams need to share, need to own. M. You can use it, but I would say that it’s rather an exception than something that you need to use by default, because if you try to use it by default, probably you’ll overuse it.

Robert [00:31:02]: But I think it’s also important to keep it in mind that sometimes you have some things like that and it’s also fine.

Miłosz [00:31:09]: Yeah, it’s a very important decision when you have two teams working with some similar entity or the same entity in different contexts. So first, there’s also the bounded context concept from DDD. What I think about it the most is exactly this situation when, for example, you have this user entity which can be also a customer entity in another context and, in other contexts, be something else. And you have multiple teams working with it. One naive approach would be to deduplicate and create one main user that decides about everything and has all the fields and behaviors and so on. But it that means all teams share this part and you know they need to agree on how to develop it or, from other side, if you’re one team, don’t that because it will just create too much overhead.

Robert [00:32:05]: And this was something that we see often: there was one team and went too far with all those patterns.

SPEAKER_2 [00:32:12]: And later, it’s like, oh, no, Domain Driven Design is making our software so complicated.

Miłosz [00:32:19]: Yeah, but if you have multiple teams, you can use the bounded context concept. So you try to figure out how this user entity works. Different contexts may be separate; you know, maybe both teams can have their own user entity and they are completely separated and that’s fine. And sometimes you have this shared kernel, as you mentioned. So some parts that are so common and probably specialized.

Miłosz [00:32:49]: Like you mentioned, maybe the tax rate calculation, something like that. But it doesn’t really make sense to separate that. You just agree, okay, on this small common part. We will work together and you know both teams— for example— need to agree on reviews and stuff like that on the design of it instead of the entire big system. So when I think about what’s difficult about this problem of splitting software well, I think it’s a combination of organizational and technical complexity. It’s not just one thing.

Robert [00:33:23]: So sometimes I remember that there are some companies where the teams and services split and organization is somehow going from the top. Maybe it’s not from the top, but level or two levels higher than you are.

Miłosz [00:33:39]: Or another way is that it became kind of the default to create microservices, and very often in a naive way.

Miłosz [00:33:50]: So, for example, when I joined some project and there was some kind of migration happening from an old system to a new one.

Miłosz [00:33:59]: A few people in the team assumed, let’s use microservices from the start. It’s kind of like a default approach for some people, but it can be a trap. Because you have to think, what problem do you want to solve this way, rather than just, ‘Oh yeah, it’s the industry best practice.’

Robert [00:34:19]: Worse if you have problem that you’re trying to solve, because sometimes people may think, ‘Yeah, we have problem with I don’t know scalability or ‘We had overcomplicated POC earlier and microservices will solve our problem.’ Yes.

Miłosz [00:34:36]: So it’s helpful to think of microservices. Deployment patterns, something we discussed some time ago, I think.

Miłosz [00:34:45]: It helps when you want independent deployments, for example, for different teams. That’s a good excuse to split software.

Robert [00:34:56]: Or for example, you have that many deployments of your module or monolith that the queue is longer than you. I mean, queue is long enough that not every person is able to merge within one day. Probably it’s already designed that you can split it.

Miłosz [00:35:13]: So it’s connected to teams and how you organize developers.

Robert [00:35:20]: So in other words, probably we can say that microservices are maybe solving a bit of technical problems, but it’s mostly solving the human problem.

Miłosz [00:35:30]: Humans working together problem.

Miłosz [00:35:33]: There’s Tim Topology’s book. I’ve read some time ago. It’s great. I can recommend it. It’s specifically about this problem with the reverse Conway maneuver, I think it’s called. So basically, instead of creating a software first and then trying to assign people who work on what, you do it the opposite way. So you first design, as we mentioned, event storming, a bit similar idea. You first design the organizational structure you want to have in place, and then the software system follows that.

Robert [00:36:08]: I love when such old patterns, like always low, are still working those days.

Robert [00:36:14]: Even if people don’t realize, but, but I think. It’s good to know about some that something like that exists, because you can be a bit maybe more mindful when you’re doing such decisions. And it’s not like, oh, just everybody’s doing that or we’ve been doing it like that for ages, why we should change it. Yeah.

Miłosz [00:36:36]: It’s also great that we are past hype. Era now and many people talk mainly about how complex microservices can be. So now it’s probably we are more balanced.

Robert [00:36:47]: Probably won’t you won’t be put on the stake after saying that microservices is not the only way.

Robert [00:36:55]: Maybe in 10 ways we’ll be again in the place when everybody is using monoliths. Probably not modular, something tells me. Probably it will be still a problem.

Miłosz [00:37:06]: I think, as an ending note, we can recommend starting with monoliths if you are just getting started, especially if you are just a single developer or a small team. Yeah, I hope we should. And do it the modular monolith way.

Robert [00:37:20]: Yeah, I hope that we’ve shown that there are alternatives and it’s really not that hard to do things in those alternative ways.

Robert [00:37:30]: Definitely will not finish with this episode about modularization because it’s a topic that requires a lot of attention, in my opinion, and it can give a lot back— i mean, when you are doing it properly, it can really accelerate your work a lot, yeah, especially with modular monolith.

Miłosz [00:37:49]: Because I think what we’ve figured out works pretty well.

Robert [00:37:54]: And we’ve seen it work with teams. And we also see that it’s not that popular approach. So I think it may be also some unique skill to have.

Miłosz [00:38:04]: Yeah. And if you’d like to see a real example of this, we have a domain engineer training coming up and we will have an example of a pretty complex application with different modules built as a modular monolith. And we mentioned parts of it in this episode as well. Yeah, I think it will be cool. So we’ll leave a link in the description if you want to learn more. So we shared our stories with Spitting Software. Let us know about yours so we can also learn something and maybe help you figure out something.

Robert [00:38:40]: Yeah. So we’re here to help you. So this podcast is for you. And yeah, if you will be happy to answer in comments and maybe include something in the next episode as well.

Miłosz [00:38:53]: Yeah, if you have any topics you would like us to cover, let us know.

Miłosz [00:38:58]: See you in the next episode. See you next time. Bye. Bye-bye.

How to Know If your Software Is Overcomplicated or Oversimplified?

Fri, 19 Dec 2025 08:00:00 +0000

Quick takeaways

Complexity comes from two extremes - projects fail both when they’re overcomplicated with unnecessary patterns and when they’re oversimplified for a complex domain
Essential vs accidental complexity - essential complexity comes from the domain itself and can’t be removed, accidental complexity is created by poor implementation choices
“Keep it simple” is lazy advice - achieving simplicity takes effort; closing your eyes to complexity just pushes it elsewhere
Match patterns to the problem - using the same approach everywhere is a red flag; mix simple solutions for simple parts and sophisticated patterns for complex domains
Ship fast and iterate - if you can’t deploy daily and fear making changes, something is wrong regardless of whether it’s over or under-engineered

Introduction

We discuss where complexity in software projects comes from and how to deal with it. There are two common extremes: projects that are overcomplicated because someone applied patterns they saw at a conference without understanding them deeply, and projects that started simple but became a tangled mess as they grew.

Both can be equally difficult to work with. The key insight is understanding the difference between essential complexity (inherent to your domain) and accidental complexity (created by your implementation choices). We share diagnostic signals to identify unhealthy projects and practical advice on matching your tools to the actual problem you’re solving.

Notes

No Silver Bullet paper by Fred Brooks - discusses essential vs accidental complexity
Domain-Driven Design (DDD) - a pattern for tackling complex domains
Clean Architecture - another approach for managing complexity
YAGNI (You Aren’t Gonna Need It) - principle against premature generalization
Defensive programming - validating inputs early to prevent errors propagating
Canary releases and rollbacks - deployment strategies that can add overhead
Three Dots Labs blog - articles on Go patterns and architecture
Go with the Domain ebook - free ebook about DDD in Go with 60,000+ downloads
Wild Workouts example DDD project - complex Go project demonstrating real-world patterns
Microservices architecture - discussed as sometimes adding unnecessary complexity
CRUD applications - simple approach that works for some domains but not others
The Domain Engineer training - upcoming training mentioned for early next year

Quotes

On one hand, you have someone who says we need to support 10 million users a day—this is one extreme— but on the other hand, you also have people who say, ‘just keep it simple and it will be fine’ and it sounds good. But to me, it’s one of the lazy advice. It’s easy to say to tell someone, ‘keep it simple but it’s not that easy in the end.

Miłosz

The complexity doesn’t go anywhere, no matter how you implement it.

Miłosz

If your team is unhappy working with the project. If the team brings up plan for migration or rewrite every other day, that’s a solid sign something went very wrong. You’re probably on one extreme already and it’ll be hard to get out of it.

Miłosz

If just keeping projects simple would be that easy, all projects would be great. But unfortunately, it’s the opposite.

Robert

So you cannot close your eyes and say that this complexity doesn’t exist. It will pop up somewhere else and it will be much more expensive to handle at the end.

Robert

Don’t be dogmatic about using any tools because it’s never ending up properly.

Robert

Timestamps

Transcript

Miłosz [00:00:00]: On one hand, you have someone who says we need to support 10 million users a day—this is one extreme— but on the other hand, you also have people who say, ‘just keep it simple and it will be fine’ and it sounds good. But to me, it’s one of the lazy advice. It’s easy to say to tell someone, ‘keep it simple but it’s not that easy in the end.

Robert [00:00:18]: If just keeping projects simple would be that easy, all projects would be great. But unfortunately, it’s the opposite.

Miłosz [00:00:25]: It takes a lot of effort to have simplicity in the project.

Robert [00:00:28]: Your eyes and say that this complexity doesn’t exist, it will pop up somewhere else and it will be much more expensive to handle at the end. It’s a very important concept to understand the problem. Accidental complexity and essential complexity. Accidental complexity is created by poor implementation. If you approach the implementation of your application in an overly complicated way, so we’ll put just some patterns that are unnecessary, you will have accidental complexity. On the other hand, if your project will be, for example, this complex financial product and you will try to build as a CRUD, you’ll end up with a lot of accidental complexity there.

Miłosz [00:01:05]: Yeah, but you can reduce it if you use some patterns that play well with the domain. So you can reflect it in the code in a more natural way. Then it’s easier to work with. So the accidental complexity goes down.

Robert [00:01:21]: I’ m Robert.

Robert [00:01:21]: And I’m Miłosz.

Robert [00:01:22]: And today we’ll talk about complexity in projects and from where this complexity is coming. And from my experience, there are probably two types or two sources from where it can come. So the first one. Probably. Many of you can relate to that. It’s the situation when you are joining some project and there is some team leader or some very senior engineer. And this engineer is dictating, let’s say, some good practices and some patterns in this project.

Robert [00:01:55]: You can clearly see that those patterns are not adding a lot to this project. Actually, it’s adding a lot of unnecessary complexity. Often, this person is getting those techniques from some conference, maybe, or from some video, or maybe this person read some article. Later, this person is forcing everybody to use those patterns.

Miłosz [00:02:15]: So it’s this ivory tower architect who dictates from the top how to do things and the solutions are already invented.

Robert [00:02:26]: Yeah, so it’s often the problem. It doesn’t need to be architect because it’s sometimes maybe, again, some more senior person. But from my perspective, very often the problem in this situation is that this person doesn’t really understand those patterns very deeply.

Robert [00:02:45]: As I said earlier. This person has seen those patterns in one video or maybe one talk and knows how it looks like. So you are going to conference, there is some idea, it looks great on the paper. In practice, Well, there are a lot of complexities and it doesn’t work as simple as it was thought on the conference.

Miłosz [00:03:04]: And being an engineer on this team, maybe peer of this person, you will probably feel like this project is overengineered, right? There is too much patterns and it seems overcomplicated. It’s very difficult to get something done.

Miłosz [00:03:22]: Because there’s so much work involved.

Robert [00:03:24]: Definitely. And I think for many people, it will be the main source of complexity in projects, but maybe you have some other idea what can be another issue that can happen in projects.

Miłosz [00:03:35]: So on the other hand, I can remember other types of projects where everything seems very simple, but at some point it’s very difficult to continue development.

Miłosz [00:03:50]: Project lifetime, it’s sufficient to have some very basic patterns in place. Maybe no patterns at all.

Robert [00:03:58]: If you have small enough application, maybe you can just do only one file and sometimes it’s starting as some proof of concept or maybe side projects and because of some lack reason this is becoming pretty successful project at the end. And it’s becoming also the problem because it started to be it started as a proof of concept. And at the end, it it became a successful product.

Miłosz [00:04:22]: These are fun. You can just add new endpoints, add new database tables.

Miłosz [00:04:30]: Works fine, but then that team grows and at some point it starts to be very difficult to extend this project because you know everything is coupled together when you change something you don’t know what else is going to change and usually at this time you you might be long for some patterns actually to be used in place and I started researching some architecture, how to approach this thing. It has two extremes, right? One is overcomplicated, the other is no patterns at all and both can be equally annoying to work with, difficult to develop.

Miłosz [00:05:14]: Thank you.

Robert [00:05:14]: In this episode, we’ll cover a bit more. How to understand if your project is in the bucket of overcomplicated by using too many patterns or maybe oversimplified. So the problem that you are solving or the product that you are working with is already pretty complex and building is a crud with just frameworks. It may not work anymore. And I think it’s a very important skill that many people is missing. So I think many people are stopping on the side that, yeah, this project is very overcomplicated because we use too many patterns but it’s not only the case yeah so it’s a similar issue but Okay.

Miłosz [00:06:01]: two extremes, right? But the root cause is kind of the same. It’s using wrong tools to the wrong problem. Or you don’t know what the problem is yet and you use only what you know.

Robert [00:06:14]: Bye. It’s not only about the complexity of the project but sometimes it’s also the scalability. So we’ll also cover a bit about this today, because also you probably heard many stories like: ‘This application will have millions of users.’ At the end, we have 10 users. And it’s hard to add anything more because you prepare your application for big traffic.

Miłosz [00:06:35]: And because of that, it’s very, very complex. On one hand, you have someone who says, ‘we need to support 10 million users a day,’ which is one extreme. other hand, you also have people who say, ‘keep it simple and it will be fine and it sounds good but to me it’s one of the lazy advice.

Miłosz [00:06:55]: It’s easy to say, to tell someone, ‘keep it simple.’ But it’s not that easy in the end.

Robert [00:07:03]: If just keeping projects simple would be that easy, all projects would be great. But unfortunately, it’s the opposite. It takes a lot of effort to achieve simplicity in the project. If you’re joining a new project and you notice that this project is complex. So in this case, do you wonder if it’s complex because of some patterns that make it overly complicated? Or maybe this project has some kind of baseline complexity that made this project complex.

Miłosz [00:07:33]: I think it’s very different. Tell the difference when you just joined the project— actually. You might know that it is difficult to work with, but you might not yet know what it solves, what’s the business domain.

Miłosz [00:07:47]: So it might be not that easy to tell if it’s over-engineered or it’s just inherent domain complexity that’s not going anywhere.

Robert [00:07:58]: And I think what may be pretty useful here is distinction between complex and complicated, because I think those words are nicely encapsulating the different characteristics of this. So when we have a project that is complex, so it’s a project that has, for example, some complex domain like accounting, complex e-commerce, or is pretty old product supporting many use cases. So in this case, it’s complex, but it doesn’t need to be complicated. So, if somebody will not approach it properly, it may be complicated, but I think what’s also pretty interesting is that some projects are not complex, but they are just complicated. And in this case, it’s the scenario when, for example, this team leader went from a conference and just read some blog article, started to copy some patterns, and just unnecessarily complicated that. For the sake of it, instead of solving something.

Miłosz [00:08:53]: Exactly. Yeah, but it might be hard to tell the difference if you are not very deep into the project. I remember working with some MVPs that looked complicated on the surface. And then you start rebuilding them or trying to understand what’s going on. And you see that, okay, this is not actually technical complexity. This is how the business works. So that’s why someone did this this way.

Robert [00:09:20]: So some examples of projects that are complex, but maybe not complicated, are for some financial-related applications. So for example, some accounting systems that are supporting multiple countries and even hundreds of countries, different tax rules that can be different for multiple countries and some domain. May have some special requirements, like for example, taxi or something like that. And this kind of projects are very big. Baseline complexity. And by the way, we’re currently working actually on a training that we’ll be releasing at the beginning of the next year when we’ll be showing how to build such very similar domain actually. So you can see the description. Yeah, so check the domain engineer in Google. So you will find it. But what I think it’s pretty interesting in this kind of projects is that if you, at the first site, look on all these accounting rules and support of multiple countries, you might have a feeling that it’s super complicated, but actually depends how you approach implementation of that.

Robert [00:10:37]: So no matter what you do, in the implementation part, it will be complex because it’s just a complicated domain.

Miłosz [00:10:46]: The complexity doesn’t go anywhere, no matter how you implement it.

Robert [00:10:49]: And often this is named as essential complexity. This is the complexity that is kind of created by the domain. And the opposite of that is. Accidental complexity. So accidental complexity is complexity that is created by engineers, basically. So when you are, for example, preparing your application for global scalability, millions of users. So this is where you are creating the accidental complexity.

Miłosz [00:11:16]: Yeah. I think it comes from the No Silver Bullet paper from Fred Brooks, which shares the name of our podcast. It’s actually a good read.

Robert [00:11:24]: What a cool incident.

Miłosz [00:11:26]: Check it out. It goes into detail on essential versus accidental. But maybe we can just quickly go over it. Hmm, hmm.

Miłosz [00:11:37]: Hmm.

Robert [00:11:37]: Also, because I think it’s very important to understand that concept to understand the entire problem of accidental complexity and essential complexity. So basically, the accidental complexity is created by poor implementation. Because if you approach the implementation of your application in an overly complicated way, so you’ll put just some patterns that are unnecessary, you will have accidental complexity. But on the other hand, if your project will be, for example, this complex financial product and you will try to build as a CRUD, you’ll also end up with a lot of accidental complexity there.

Miłosz [00:12:16]: Yeah, but you can reduce it if you use some patterns that play well with the domain. So you can reflect it in the code in a more natural way. Then it’s easier to work with. So the accidental complexity goes down.

Robert [00:12:31]: Unfortunately, if you do the opposite— so if you have a complex problem to solve and you try to approach it in an oversimplified way— it will add. So we will have the base complexity, so the essential complexity, and you will have also accidental complexity. So this product probably will be a great mess that nobody would like to touch.

Miłosz [00:12:51]: And suddenly, we developers often bring this on ourselves, focusing on the technical details, trying to build a framework that will capture cases in the future, stuff like that, which all can add up to extensive complexity before you need all the features.

Robert [00:13:11]: It’s starting with a solution that is generic from day one, and it’s basically creating a solution that is generic. Always much more complicated than creating a solution for one use case. I have some examples when we’ve been adding some functionality and some teammates were proposing like, ‘Oh, let’s maybe create a library to do that.’ You will not need to do it in the future. But after trying to design that, we found that actually making it in a generic way is a couple times more complicated than just doing it in one way for one scenario. And sometimes it’s even easier to just copy some logic and have a bit different implementations if it doesn’t change that often in two places and independently.

Miłosz [00:14:00]: Because again, having this generic solution be just very expensive, yeah, so you need to be very conscious of where to use the generic solution, where to when to abstract away the details, and for for what system you need, what will approach.

Robert [00:14:18]: Let’s go back again to the problem of complex projects, because I think it’s also interesting that if you are working with these more complex projects, sometimes some people from other teams or some new joiners may look at it and think, ‘Oh, guys, you’re overcomplicated that so much. You’re using so many patterns and it’s hard to go over this code and understand how it’s working. It may look like this at the beginning when you are working with some more complicated domains. But people may sometimes miss the context here. And sometimes you may be in a rescue project, basically, that earlier failed. Because it was created as a CRUD, you are trying to save this project and implement it in a way that you can tackle this.

Robert [00:15:10]: Essential complexity. And some people looking at that later may be missing this context here.

Miłosz [00:15:22]: Overengineering. Yes. Yeah, but on the other hand, it’s also the opposite issue, right? If you oversimplify things from the beginning, it may be too late to change the architecture.

Robert [00:15:38]: And I think it’s also sometimes interesting when, for example, the team was changed and most people left this team.

Robert [00:15:47]: If it was a complex project, not overcomplicated, but the person that was trying to change it later. May have in mind that, oh, okay, it’s so overcomplicated, let’s try to rewrite it. So this person or this thing is spending like their half year or year to rewrite that to just realize at the end that basically, all the complicated or the complex, not complicated stuff is again there. Because when you start to support more and more use cases and edge cases, you’ll see that actually all those checks and all those complex parts were necessary there so yeah and they don’t go away when you change the approach.

Miłosz [00:16:31]: Even if you change the technical approach, the underlying domain complexity doesn’t disappear. All right. And. If you’re like a developer focused on simple code, if you like simple code, you might be annoyed that there’s new requirements coming in.

Miłosz [00:16:49]: I know you need to change the factor to capture it, but it’s just the reality. You need to support the business this way. And however you model it, the complexity is there. And if you try to avoid patterns at all costs and just do it simple, you may end up down the road with some weird solutions, like for example, instead of having a good transaction boundary where you need it, you start running cron . checks for consistency. In your system and report some errors, and for some teams this is like the business as usual approach. Let’s just fix it afterwards, but that’s so much— you know— with operating efforts. You are bombarded with alerts all the time.

Miłosz [00:17:52]: You need to react and fix the system.

Miłosz [00:17:57]: After the fact, instead of just designing in the first place in a way that it’s resilient and I think it doesn’t need to be.

Robert [00:18:04]: We don’t need to reuse any advanced patterns to avoid that. So sometimes it’s basically just some defensive programming. So let’s take an example when we’re building this accounting system, for example, and you are not validating if, for example, the amounts are positive or non-zero. And you can accept any value and issue thousands of invoices that have a total value of zero. And. Okay, you can later clean it up. You maybe have some chrome that will void them or do whatever with that. But basically, it will be much cheaper to do this defensive programming approach earlier and ensure that it will not propagate further. It’s not only basically about uh operating overhead from side of your team, but it may be also some financial team seeing later that, oh, no, we have thousands of invoices that we need to do something about that. And those developers again.

Miłosz [00:19:09]: Right. So you only push around the complexity this way instead of solving it.

Robert [00:19:15]: So you cannot close your eyes and say that this complexity doesn’t exist. It will pop up somewhere else and it will be much more expensive to handle at the end.

Miłosz [00:19:24]: Some analogy that comes to my mind is like if you decided you won’t have files bigger than 100 lines in your project. And you will have simple files, right? Small files. Sure, it’s great, but then you end up with thousands of them. And you push the complexity to the number of files instead of the length of the file.

Miłosz [00:19:47]: Back in the day, there was some big question: how big should be a microservice, and some people suggested metrics like this. This comes to my mind.

Miłosz [00:20:00]: Let’s build our huge application into really tiny services. So you solve this complexity of big code base, but now you deal with thousands of code bases.

Robert [00:20:15]: So basically, divide and conquer doesn’t always work with software. It can work, but you need to know basically how to split them. I think it may be also an interesting topic for some other episode because, Well, not a surprise. There are some patterns and ways how to detect that. And spoiler alert, it’s not as small service as you can because, at the end, you can just end up with big mess and some sagas that you need to orchestrate some really small part of the flow. But I think it’s enough for today.

Miłosz [00:20:51]: The problem here is you think the size of the code base is the issue, right? It’s similar to what we discussed before. The complexity is in the domain, so you can’t just change this. If you have a complex business and it needs a big application to work, this complexity will be there, however you model it. So you just need to make effort to model it in a way that’s best to reflect how it works and for teams to work with it.

Robert [00:21:20]: And I think it may be misleading sometimes, because you may feel that splitting this big problem into smaller microservices can actually make this problem simpler, easier to tackle. On the other side, sometimes you may wonder. How?

Robert [00:21:38]: Some application can have, I don’t know, 5000 of developers for running and developing such simple application. I know that it’s sometimes hard to judge an application from outside because you may not know all the complexity inside. But come on, thousands of developers— it’s a ton of people. Splitting it into hundreds of microservices can create an illusion that you just cut this problem into smaller and it’s easier. But there are cases when, basically, this application could be one service with 50 times less developers developing it.

Miłosz [00:22:17]: Maybe you won’t need a whole platform team then to manage all this complexity and communication between them.

Robert [00:22:25]: And if you add the extreme scalability requirements that are, I think, also often out of nowhere, because really, you can probably handle Twitter-like scale on a single VM. Just a reminder that you can have VMs that have one terabyte of RAM, 96 cores. On GCP, it’s even 150 cores per one machine. So come on, you can really handle Twitter scale on this one VM.

Miłosz [00:22:54]: RAM is expensive now, so I don’t want to do that.

Miłosz [00:22:58]: So coming back to the essential versus accidental complexity, what is the solution here?

Robert [00:23:05]: So I think for the case with cron. Again, some simple solutions like defensive programming and very very very very very very defensive programming. So you have any input that this not as expected, don’t accept that. Don’t allow to propagate it further. And the other thing, when you are splitting problems into smaller, also watch out to not split it too granularly. Other thing that is coming into my mind is trying to tackle this essential complexity instead of always splitting that. Because there are some patterns that are helping to tackle that. In, for example, this one service, it can be clean architecture, it can be domain-driven design. Thank you. Very important disclaimer. I’m not saying about some dogmatic clean architecture or domain-driven design because it’s very easy to go into caricature of those techniques instead of some pragmatic approach. So don’t be this person that you see in one video or one presentation or one article.

Robert [00:24:08]: Dig a bit deeper, try to understand what problems those techniques are solving, what problems they are not solving, when to not use them, and apply them in places where you have the essential complexity to not add more accidental complexity.

Robert [00:24:27]: Sounds like a plan.

Miłosz [00:24:28]: Yeah, that’s something I would recommend to consider where in the project you use what approach. If there’s one approach you use in your entire project everywhere, that’s probably an amber flag. Probably something is weird because The most projects don’t have only you know, only complex or only simple domains. There will be a mix of some very generic parts that, you know, we can use something simpler for it. And the core domain where you need your most sophisticated patterns.

Robert [00:25:06]: Definitely. And also when you are using some patterns that you read about, you’re trying to apply that. Also, if you feel that it doesn’t fit, don’t try to push it by force because probably it may not be fitting there or maybe you don’t understand something about this pattern.

Miłosz [00:25:21]: Sometimes you can look back after implementation and I remember we did it many times. Uh-huh. Uh, which created some domain layer and try to model something in it. And then we look at it and, hey, it’s like mostly boilerplate. Maybe we can just simplify it because there is nothing it gives us, no benefit.

Robert [00:25:44]: It’s basically a crud. So we just can have one struct getting it from API, put to the database and it’s fine.

Miłosz [00:25:51]: We have a mix of these parts of complex domains, complex patterns and some very simple ones. I think it’s a healthy sign of a codebase in general.

Robert [00:26:01]: So watch out for being dogmatic. So it’s.

Robert [00:26:05]: It’s coding, it’s not religion.

Robert [00:26:09]: Many people will probably disagree, but it’s reality.

Robert [00:26:13]: And I think it would be useful to maybe give you some advice from our experience.

Robert [00:26:20]: What are some, let’s name it, diagnostic signals to understand if your project may be over complicated or oversimplified? Because I think we can probably find a couple of signals that may suggest that there is something wrong with this project. Well, I think it’s pretty interesting is the fact that those signals may be the same for projects that are oversimplified and overcomplicated. So you may remember that at the beginning we said that it’s the same problem, but from another side, basically.

Robert [00:26:58]: It’s about half of the codebase in general. So what would be, from maybe your perspective, some red flags that the project is unhealthy? So it may be overcomplicated or oversimplified.

Miłosz [00:27:11]: Probably the king of all would be if you can ship the project fast and do it over time, iterate often, and maintain it— that’s a good sign. If you can’t do it, something is wrong. So it’s like a big number one rule.

Robert [00:27:33]: Some anti-examples here, maybe some old enterprises that are deploying. Every half year, for example.

Miłosz [00:27:40]: Yeah, like the other extreme. Yes. For example, you need to hand over your deployment to someone who does it during the night because it’s, I don’t know, risky.

Robert [00:27:56]: And you can also do it daily because it’s the scenario when you have an essential with accidental complexity. So it’s a very old project that has already a lot of edge cases to handle. But it’s also overcomplicated because maybe in some places it’s not planned strategically properly.

Miłosz [00:28:15]: Yeah.

Miłosz [00:28:17]: So if you can deploy daily, that’s probably one red flag. If you fear merging changes or just changing some areas of the project, because you don’t know what can happen. That’s another red flag, I would say. Whatever the reason, you should be comfortable deploying and managing stuff.

Robert [00:28:40]: So in other words, if your project is crude and you are able to merge it daily or a couple of times a day, and you are not afraid to introduce new changes, it’s totally fine. It’s probably enough. But for example, when you have some complex domain and you are not able to do that. Well, you may have some excuse that, oh, it’s so complicated. We are working with money. We cannot merge every day. But yeah, there it is. The reality is that if you tackle this accidental complexity, if you handle this essential complexity properly, you can even deploy multiple times a day application. Pretty complex and is doing some risky operations.

Miłosz [00:29:21]: Yeah, if you are working with money, I hope you just don’t just you know cross your fingers and deploy. But you are confident this will work. If you, if you’re not, then probably something is wrong. Maybe the project is oversimplified.

Robert [00:29:37]: Other red flag that I have in my mind is when you have some new joiner in your team and it’s taking weeks to onboard this person. So it’s kind of connected. Previous points because, if you aren’t able to deploy daily, you have fear of touching anything in your project. The same will also apply to new joiners and even more, because you’re a new joiner. You see that this project is complex, and/ or people warn you or don’t touch this part.

Miłosz [00:30:09]: You don’t want to see this.

Miłosz [00:30:14]: I’ve seen projects like this.

Robert [00:30:16]: Exactly, exactly. And in a healthy project, I would say that it should be quite safe to fail, basically. And it’s connected to defensive programming. Basically, and also having some good infrastructure in place that is maybe doing some Canary releases and rollbacks, but it can be also the opposite. For example, if you don’t have application with complex domain, and making mistake is not that big problem, but you have very complicated infrastructure for rollbacks, for Canary releases. It can add a lot of overhead for your deploying pipeline and, in this case, it even cannot deploy the simple application daily because the process of deployment is too heavy compared to what’s required for this application.

Miłosz [00:31:05]: Also, if your team is unhappy working with the project. If the team brings up plan for migration or rewrite every other day, that’s a solid sign something went very wrong. You’re probably on one extreme already and it’ll be hard to get out of it.

Robert [00:31:28]: I think many people may say, like, ‘projects like that don’t exist.’ I think it’s a nice excuse, but it’s not the case. Such projects exist that you can really deploy daily, even if it’s not very simple projects.

Robert [00:31:46]: So, to maybe recap.

Robert [00:31:48]: Those indicators a bit. So I think there are some questions that you may ask yourself. So the first one is asking yourself, Can you remove some stuff without fear that you’ll break everything, basically?

Robert [00:32:05]: And when you’re thinking about those red flags, remember, it’s not just because some patterns that some people use sometimes, it may be because premature optimizations, like preparing for scale that will never happen.

Miłosz [00:32:21]: Yeah, that’s a lot of accidental complexity there.

Miłosz [00:32:25]: Or focusing on performance before you need it. So instead of choosing something more explicit and easier to grasp, you choose something very optimized, but not really needed at the time.

Robert [00:32:41]: And sometimes it may be also preparation for the future—like, maybe at some point we’ll need X. Let’s do it now, so it will be easier. Unfortunately, usually it’s a trap. Usually it will never happen, or if this functionality will be needed, if we will need to add this functionality, we’ll find out that actually what we prepared earlier no longer works in this way. And we spent a lot of time on writing that and maintaining it. So, jagni.

Miłosz [00:33:10]: Yes, it’s easier to get things that are easy to remove or replace instead of looking for all the possible edge cases.

Robert [00:33:19]: Always have in mind that most of the techniques, tools, or whatever you are using, you’re using it to make your development lifecycle as efficient as you can. So, if you see that but something is making your development lifecycle worse, slower, and it’s slowing you down, you should maybe rethink if it’s needed or you’re applying it in the right way, or maybe it’s not needed at all. And at the end, you should aim to making some small incremental changes. And I know that it sounds easier than it is, it’s easier to say than do. But it’s a matter of experience and learning, learning and learning, and trying to be as close to this go to deploy multiple times a day. And also, watch out to not go into the trap that we already invested couple of months into this feature to make it super scalable. It doesn’t matter that we need to spend now weeks on maintaining that.

Robert [00:34:21]: Watch out for this trap. Don’t be afraid to remove some stuff that is slowing you down.

Robert [00:34:29]: Yes. So it’s important to be aware of that. So if you already invested a lot of time into something, it’s harder to work some framework. Some generic library trying to do everything. Some stocks. But it’s showing it similarly. Like you invested in something. Company and you see that it went down, and you are waiting years to recover. And probably just saying that and buying some better stock can help more. By the way, not investing advice.

Robert [00:35:03]: I was about to say that, yeah.

Robert [00:35:07]: And at the end, I think it’s important to emphasize one thing.

Robert [00:35:12]: Don’t be dogmatic about using any tools because it’s never ending up properly.

Robert [00:35:22]: Have an analogy of some worker that is, I don’t know, big fan of pneumatic hammer, and this person is just using pneumatic hammer everywhere. And it may work if this person is demolishing buildings, but if you need to nail nails, well, probably using nomadic armor for that. At best, you’ll hurt yourself.

Robert [00:35:44]: But in this case, it would be just more practical to use the normal hammer. But from the opposite side, maybe there is some fun in using a normal hammer. Saying that a normal hammer is great and it’s simple— it doesn’t need any other energy source. You can just use it everywhere. But it’s true. However, if you need to demolish a wall, it will take ages.

Miłosz [00:36:07]: For example, something that we mentioned in a previous episode about having a toolbox. You can choose your tools basically for the job.

Robert [00:36:16]: And even if it sounds, maybe, very obvious. From other side, we’ve met a lot of people that were like these people with simple hammer or pneumatic hammer used for everything, and even if it sounds that obvious, they were doing it in this way.

Miłosz [00:36:38]: So it helps if you ask yourself, why are you doing this? Why use this tool in the first place? If you want to solve something specific, for example, if you have a complex domain and that you want to model better, that’s probably a good reason to use something. Or if you kick off a project that you know is supposed to be a big one, you probably want to have some idea of the architecture of the design beforehand. But on the other hand, if you just kick off an MVP, you don’t need all this stuff. You can just start simple and grow.

Miłosz [00:37:23]: and yeah watch out for extremes, what we mentioned before. So if you are far on the over complicated or over simplified spectrum, the project probably will be difficult to work with.

Robert [00:37:40]: And the thing that we can probably recommend most, and we are saying it over and over, but play with technics sometimes in work, sometimes after work, also. So if you would like to be good at what you are doing, it’s it’s also good to practice this outside of work, because you’re also sometimes limited by the projects or maybe sometimes you’re limited by some dogmatic people that, for example, like oversimplified code and you cannot do much about that, but you can always experiment a bit in in the home. So one thing that we can probably recommend is checking our blog tree . tech and where we have a lot of articles that are totally free. We also created one pretty big Go project that you can play with and try to apply. Look how we applied some patterns there. And you can also try to play with this project and add some of your features there. And you can gain this muscle memory on how to use those patterns and later, when you will encounter some problem, you’ll have some tools in your toolbox and you’ll be able to experiment with that.

Miłosz [00:38:51]: So we tried to create more complex examples, so you can see the ‘why’ behind the patterns, not just some toy projects where they don’t make sense.

Robert [00:39:01]: Yeah, because if you have sometimes those dogmatic people about using some patterns, you can notice that very often they have very shallow knowledge because they’ve been reading some article. That again, are very shallow. So it’s like five minutes read. And they are thinking that they know everything about some technique after doing that, or doing it differently. So our articles are not simplifying things, but there are—pretty long, but they are long enough to prepare you for applying some patterns in the wild. There are some good articles out there. It’s not like only our articles are good, but very often there are some articles that are very, very shallow. And when you look on them, you can just see that applying what’s suggested there will introduce more troubles and this accidental complexity without solving any problem actually. So we would have your base complexity with accidental complexity, and at the end it will be how to work with.

Miłosz [00:40:01]: So we also have the Go with the Domain ebook about working with domain-driven design in Go. And it shows refactoring of a bigger codebase.

Miłosz [00:40:14]: This is not a toy project, but more like a real-world scenario.

Robert [00:40:19]: And it’s, I think, already pretty popular. So last time when I was checking it, it had over 60,000 downloads. And we also received a lot of emails from people that it helped them a lot in the project. So for some people, it’s quite useful.

Robert [00:40:34]: Check it out. It’s also free by the way, so just google go with the domain and I’m sure you will find that Cool. So I think that will be all for today. So thank you Miłosz. Thank you Robert. See you next time. That’s it.

DDD: A Toolbox, Not a Religion

Wed, 10 Dec 2025 12:00:00 +0000

Quick takeaways

Domain complexity matters more than technical complexity - Most projects fail not because of technical challenges, but because they don’t handle the business domain well.
DDD is a toolbox, not a religion - You don’t need to use every pattern from Domain-Driven Design. Pick what solves your actual problems.
Start with the domain model - Understanding how the business works is more important than designing the perfect schema.
Avoid solving imaginary problems - Spending months on frameworks or platforms before building actual features often leads to wasted effort.
Strategic patterns are essential for everyone - Even if you don’t use tactical DDD patterns, thinking about core domains and module boundaries matters in every project.

Introduction

In this episode, we discuss why software projects become legacy code that nobody wants to touch.

We talk about how Domain-Driven Design can help, but also why it’s often misunderstood or overused.

Instead of treating DDD as an all-or-nothing approach, we suggest to use the ideas pragmatically - picking the patterns that solve real problems in your project.

We share stories to show how focusing on domain complexity rather than technical complexity leads to better software.

Notes

Go With the Domain ebook - our guide to applying DDD ideas in Go
The Domain Engineer training - launching early 2026
Domain-Driven Design (The Blue Book) by Eric Evans - the original book that introduced DDD almost 25 years ago
Design Patterns book by Gang of Four - classic software design patterns book
Event Storming - collaborative workshop technique for understanding domains (covered in the previous episode

Quotes

The bold assumption for today’s episode is that you can go deep into many of the ideas and they shine in complex projects, but very often you can use some more basic version of the key ideas and apply it to most projects actually.

Miłosz

If you are hiring specialists in many fields, the sentence ‘if you have hammer in your hand, you will see nails everywhere’ may happen. They will spend a ton of time on solving problems that may be not the problem in this company yet.

Robert

If you consider most of applications, like SaaS apps, they are usually very similar. If you change jobs, you can end up at another completely different company and use the same tech stack. It’s not much different to use the same programming language. But on the other hand, what the company does can be completely different, and this is the differentiating factor between them.

Miłosz

The worst thing about that is sunk cost fallacy. So you’re spending more and more time on maintaining that, losing a lot of resources, time, and nerves on maintaining that. But it’s no longer needed.

Robert

We spent many hours creating this messaging framework using generics. And in the end, we had this good interface that you can very easily publish an event, very easily subscribe to it. And the result was that it was probably the only game jam we never finished.

Miłosz

You should use proper technique for a proper problem. That’s it.

Robert

If you boil DDD down to one primary theme, I would say something like it’s about understanding the domain you work with and then modeling it well in the code. Whether you call it DDD or not, it’s like a mindset.

Miłosz

It’s kind of forcing you to think, ‘Okay, is this code really important to my business, or is it less important?’ And it can kind of affect how we are thinking about this code.

Robert

Timestamps

Transcript

Miłosz [00:00:37]: Hello, I am Miłosz.

Robert [00:00:38]: And I’m Robert. And this is No Silver Bullet Podcast, where we discuss mindful backend engineering. In this show, we share takes on software engineering that will help you to grow into principal engineer level.

Miłosz [00:00:51]: Let’s talk about the usual arc of a software project.

Miłosz [00:00:55]: So first we have the kickoff, and now we have high hopes for good design and this product will be great and everything is fine. This time we’ll do it. Right. Yeah. And then we work on it for some time. Then we deploy it to production. Everything is great. And suddenly it becomes a legacy project no one wants to touch. Did you ever see this happen?

Robert [00:01:20]: Every time. Okay, not every time, but yeah, it’s very often the case.

Miłosz [00:01:27]: So it’s interesting, right? Why does it happen? Why every project ends up as this big ball of mud no one wants to work with.

Miłosz [00:01:40]: And the first version works quite well. We come up with something, it’s pretty much what we wanted, that’s fine.

Miłosz [00:01:48]: And then new features come in and very often they are quite different than what the software did in the first place. Right?

Miłosz [00:01:59]: So we need to change the design. But very often there is not enough time or developers are lazy or they don’t feel like it, whatever. But they don’t remodel everything to fit the new design. Instead, we add some workarounds or hacks and it works pretty well.

Miłosz [00:02:20]: For a while, except that maybe some bugs, maybe some unusual behaviors. And we repeat it over and over, over months or years. And then we arrive at the state that we are afraid to touch the codebase. Is there a way to fix it?

Robert [00:02:39]: Oh. It. Wouldn’t be surprising to say that it’s a problem that software always had. So probably there are some magic techniques that are trying to solve this problem. One of the most popular, let’s say, tools or techniques that are trying to approach that is domain-driven design. What?

Robert [00:03:00]: If you heard about Domain Driven Design, you probably also noticed that for some reason it’s not the default way of many teams to work. From one side we have some solution, from other side probably didn’t solve all the issues that we have.

Miłosz [00:03:16]: And it’s also sometimes confusing; sometimes it seems like something you would use only for enterprise projects, but actually it worked for us before. So we like talking about it and sharing the ideas.

Robert [00:03:29]: And look how people are surprised sometimes to hear that it worked for us because many people tried to use domain-driven design and it introduced more mass for them. Or maybe they worked in a project when somebody tried to use domain-driven design and it’s the worst experience that they have. Interesting.

Miłosz [00:03:45]: Mm.

Miłosz [00:03:47]: So the bold assumptions for today’s episode are that you can go deep into many of the ideas and they shine in complex projects, but very often you can use some more basic version of the key ideas of it and apply it to most projects actually.

Robert [00:04:06]: So— In other words, in this episode, we will cover How to not use domain-driven design, but some part of domain-driven design and how to make them useful in your projects.

Miłosz [00:04:18]: First, let’s think about what is difficult about building software.

Miłosz [00:04:24]: So let’s consider someone wants to start. and they want to learn some programming language right, what would you recommend them to learn?

Robert [00:04:34]: Maybe not, why, what I would recommend, but what people are using.

Miłosz [00:04:37]: Maybe it was a default, like us were here.

Robert [00:04:40]: So. They probably need some programming language, right? Yeah. Kubernetes, OOP design patterns.

Miłosz [00:04:48]: There are some frameworks, some good practices, some algorithms, maybe data structures, stuff like that.

Robert [00:04:55]: Useful stuff that you are using on interviews, in other words. Exactly. And it’s actually, I think, pretty interesting that a lot of learnings nowadays is focused around things that are useful for interviews, but on the other side they are not that relevant when you are working in a job. And I would say that it’s even worse because you are solving problems that normally you should not solve in the problems that you are working like.

Robert [00:05:23]: You should probably not implement your own databases and you implement binary trees because most graphics have that out of the box.

Miłosz [00:05:31]: But at the same time, if this person wants to learn, wants to join a company, they probably need to learn some computer science theory, like what is big O notation. How to write optimized algorithms for sorting, how to create data structures, stuff like that.

Robert [00:05:52]: It’s useful.

Miłosz [00:05:54]: Yeah, it is. And it’s often what you need to know to join a company. But my question is, is it enough to ship good software?

Robert [00:06:03]: Well, I would say that it’s not enough, and probably it’s even not needed, because well, we are people who are self-taught, so we’ve been learning programming ourselves, and we’ve been shipping multiple projects without that, basically, and many people were using that. They were maybe not perfect, but for the other side, they worked.

Miłosz [00:06:27]: Yeah, so I would say it’s like half of the skills you need.

Miłosz [00:06:32]: And it’s useful, it’s needed. It’s often what we focus on. But there’s also the other half of stuff you need— to ship good software. So what we discussed so far is the technical complexity, right? So making software fast and scalable, and optimizing memory usage.

Miłosz [00:06:54]: Stuff like this.

Robert [00:06:55]: Probably we can put it into the bucket of software engineering. So this is this hard part. Maybe not hard, but this. Yeah, this low-level part maybe. Yeah.

Miłosz [00:07:06]: The coding part.

Miłosz [00:07:09]: And the other half, how would you call it?

Robert [00:07:14]: So if you are working with product, probably product engineering.

Miłosz [00:07:18]: Yeah. So the product or domain challenges. So basically figuring out how the thing should work and what should it do.

Robert [00:07:29]: So this product engineering or those domain challenges, it’s less about those technical parts, but rather thinking.

Robert [00:07:42]: How to tackle complexity, how this feature should work, maybe even thinking if it will be useful. Because if you know if something is not useful, you can remove some functionality and decrease complexity with that.

Robert [00:07:58]: It’s also this: less priced part, let’s say, and that it’s connected to communicating to people.

Robert [00:08:06]: Yeah, figuring out together what you want to build.

Miłosz [00:08:11]: Maybe also talking to your users?

Robert [00:08:13]: I think it’s also partly related to some long-term maintenance. So it’s, I would say, partially part of the software engineering, but I think it’s often lost somewhere because, when you have software that is not that maintainable from other side, it’s also challenging. And I would say that a lot of people that have this enduring mindset, they like challenges. So if software is complicated, it’s great because we have a lot of challenges there to solve.

Miłosz [00:08:45]: Yeah, the technical parts are very interesting to solve.

Miłosz [00:08:49]: We like puzzles. At least I know I like them too. And the technical parts often have a very specific answer to them. Mm-hmm.

Miłosz [00:09:01]: For example, we can have an SQL query, you can benchmark it, and then you can come up with a faster query. And it’s very easy to compare if it takes 2 seconds or 10 milliseconds and it feels great to have this measurable progress. But for many businesses, it’s not really critical at all. If your website handles a request in 50 or 70 milliseconds— or whatever it is— usually not a huge difference, except for the first page load, which is critical. But for many web applications, saving some data takes one second longer or shorter; it’s not a huge deal and probably not worth spending days on.

Miłosz [00:09:52]: coding on.

Robert [00:09:53]: As long as cost is not the important part, but again, it’s also some subset of products. For many products, it’s not that important at the end.

Miłosz [00:10:03]: Exactly.

Miłosz [00:10:05]: But for most businesses, the hard part is taking the problem domain they need to solve and representing this in code.

Miłosz [00:10:17]: Especially if the problem domain itself is complex.

Miłosz [00:10:23]: And usually the real world is messy in itself. Right. So we have no easy way to compare if something works or not, like with the technical parts. We often need to experiment, see if something sticks, see how users use it.

Robert [00:10:47]: Basically, the feedback loop is much, much longer and it’s not that obvious. Like you said with SQL query, it’s easy to measure if SQL query is faster or not. But obviously you can measure some customer satisfaction or some feature, but it’s more indirect and it’s usually taking a bit longer to get back to you. And it’s also a bit harder to understand sometimes how it’s happening because you can change some functionality. And notice that, okay, some behavior changed, but sometimes you may have that many changes around it—so it’s hard to judge if it’s because of it, actually.

Miłosz [00:11:28]: Yeah, so it’s much more vague. And if you consider most of applications, like SaaS apps, they are usually very similar. If you change jobs, let’s say you used to work on web application using Go and TypeScript.

Miłosz [00:11:50]: You can end up at another completely different company and use the same tech stack. It’s not much different to use the same programming language.

Miłosz [00:12:00]: Same framework, basically, protocols, databases are very similar. They don’t change much, but on the other hand, what the company does can be completely different, and this is the No.

Miłosz [00:12:13]: the differentiating factor between them. It’s sometimes funny how companies try to make their technical choices something that makes them distinct. So I don’t know if you remember— About 10 years ago, there was this hype on MongoDB, for example. Oh yeah.

Miłosz [00:12:32]: And some companies were very open about it.

Robert [00:12:39]: Proud. So when you’ve been sending CVs and it was the job posting, it was ‘Yes, we are using MongoDB. Yes, we are web-scale, but of course, it did not really matter in the end. It’s so absurd. But it’s also interesting because during that time, some people were saying that it’s absurd, but from other sites, many people were following that and saying, ‘Yeah, it’s so great.’ You’re using MongoDB, your web scales.

Robert [00:13:08]: Let’s see what we’ll see 10 years.

Miłosz [00:13:12]: You can just as well use JSON columns in Postgres and achieve pretty much the same thing, except for some differences in very specific cases.

Robert [00:13:22]: Or losing changes because of inconsistencies between nodes.

Miłosz [00:13:28]: Yeah, but basically, if you take two companies that do the same thing and one uses one database and the other MongoDB, whatever.

Miłosz [00:13:39]: It probably won’t impact that much the end result. How well they do in business, how many users they have, how good the product is.

Miłosz [00:13:49]: There’s a lot more factors in it.

Robert [00:13:52]: Okay. In other words, we can say that many companies, many people maybe solve on the wrong thing because when starting your project, they’re thinking, ‘Okay, this time we’ll use this framework, this database.’

Miłosz [00:14:06]: Yeah, very different this time.

Robert [00:14:08]: Yeah, yeah.

Robert [00:14:10]: This time everything will be great. And again, we are in the cycle of starting in the greenfield and ending up with legacy, like: It was done five years ago, 10 years ago. So probably.

Robert [00:14:28]: The technology that we’re using— the framework, database, maybe not the most important indicator if project is successful or not, and by ‘successful’ I mean multiple things. So, if it’s maintainable, if it has users, if it makes money— probably from perspective of developers, the maintenance is the most important part. But again, it’s not the only thing.

Miłosz [00:14:54]: Yeah, but we treated that as something that’s a core thing we need to solve. Sometimes. Right.

Miłosz [00:15:03]: I remember some companies with the idea that we will dedicate top talent to this core platform framework— or something— and once we solve it, then adding new features and apps to it will be trivial. Just hire anyone who’s able to code in this language, and they will just use this great framework to implement a new feature in one day.

Miłosz [00:15:34]: I don’t know where this idea comes from, but I think it’s quite common.

Robert [00:15:40]: And I think one part of the problem may be that if you are hiring specialists in many fields, the sentence ‘if you have hammer in your hand, you will see nails everywhere’ may happen. I mean, if you find hiring specialists from multiple areas, they will spend a ton of time on solving solving problems that may be not the problem in this company yet. So, for example, this startup and it’s starting. They don’t have scale yet. And probably it could be run on a single VM. And later, you have, for example, some Kubernetes specialists. So what they can do, they can create multiple Kubernetes clusters for multiple environments and later serve. Requests per second and even 10 requests per second. It’s like nothing. You can do it with a single VM, but it’s not for free. Obviously, you are using your top talents, but for solving problems that are not important.

Miłosz [00:16:43]: And I remember myself falling into this trap, probably multiple times. But one story that I remember very well is how I used to run some game jams with friends, just making game in two days, right? And one time we wanted to create a story-driven game with a lot of dialogues, so you can choose dialogue options and it leads you to a different part of the game. Somehow we came up with the idea that we will do it event-driven. So each option will be an event that sends and triggers something. We had this idea that, now, this would be a lot of events, which will be hard to manage. So we will need this even during the framework. So, if only we get this framework right, all that’s left is to implement the dialogues.

Miłosz [00:17:30]: That would be it. So we spent many hours creating this messaging framework using generics.

Miłosz [00:17:40]: C# and I didn’t work very well. So it was super fun to come up with it. And in the end, we had this good interface that, you know, we just, you can very easily publish an event. Very easily subscribe to it. And the result was that it was probably the only game jam we never finished. We had this technical solution, but we didn’t progress with the game at all.

Robert [00:18:05]: So in other words, you are not the best specialist to be hired to build event-driven games. Exactly. Exactly.

Robert [00:18:12]: I can create a framework for you.

Miłosz [00:18:14]: I think it shows. It’s like a very condensed, you know. Scenario that I think happens on bigger scale as well, and I don’t even know where it comes from, but somehow we tend to think, if only we can create this framework first, and everything will be easy.

Robert [00:18:36]: And sometimes, if you succeed in creating this framework or whatever it is, this prerequisite, and you’re starting to rework on the things that matter.

Robert [00:18:47]: You’re starting to notice that, okay, actually, it’s totally not needed.

Robert [00:18:52]: We just spent two months on that. And you know what’s the worst thing? The worst thing about that is sunk cost fallacy. So you’re spending more and more time on maintaining that, losing a lot of resources, time, and nerves on maintaining that. But it’s no longer needed.

Miłosz [00:19:12]: Yeah, and you don’t want to get rid of it because you invested so much time. Yes, I think we’ve seen many examples like that, right? I think it’s very common when you have a platform team in a company that is supposed to make all teams work together. Very similar ways this is exactly the idea of having one great system that you just add tiny pieces to, and of course it makes sense for a bigger company. But sometimes you can get go too far. I remember one time we had the platform team create service mesh for developers. To use it, you have service discovery so you can communicate between services using just service names. Sounds great, and they spent half a year working on it. But first of all, no one asked for it.

Miłosz [00:20:07]: And we really didn’t have any use for it later. It just looked great on paper.

Robert [00:20:13]: I think we even mentioned that. Yeah, but maybe we don’t need that because we already are doing it in Middleware.

Miłosz [00:20:21]: Maybe if you have five services, maybe you can just call it with the URL of Kubernetes and wow, it’s enough.

Robert [00:20:29]: Yeah, and all the cross-cutting concerns could just handle it with common library and middlewares. That’s it. It’s probably again, it depends on context. If you have a super complex project with multiple programming languages, maybe. But in this case, we just have one technology there and it solved the problem already.

Miłosz [00:20:52]: So this is all the first part, the technical part you mentioned. Sometimes it’s easier than the domain part, the product part.

Robert [00:21:00]: And it’s very often. You can move it between projects as you said earlier. So it can be very similar across and you don’t need to reinvent the wheel basically. So we already know that this technical part is very often similar between the projects and the thing that is different is the domain.

Robert [00:21:24]: part of the project. And we already mentioned once domain-driven design, that it’s supposed to be helping us with this.

Robert [00:21:33]: Think that is unique to our company. and many of you, maybe heard about this earlier, but it’s not.

Robert [00:21:43]: wide load up. do you have some ideas why it’s like that? and actually, what the point of indesign at the first point?

Miłosz [00:21:51]: maybe it’s even difficult to define what the approach really is in a very short description. and i remember before i knew about the dd. i think my first contact was when my manager wanted to have the entire team join the training for off-domain design.

Miłosz [00:22:17]: and they sent me over like a pdf description of it with lots of details what the training is about and i remember reading through the document.

Miłosz [00:22:29]: and at the end i was like, what is this? i have no idea. How is this supposed to help me with my day-to-day work? I just completely had no idea what.

Robert [00:22:41]: What’s the idea? So it was more covering some non-technical things and at this point you were thinking, ‘Okay, it doesn’t matter’ or ‘it was not sold properly,’ let’s say.

Miłosz [00:22:51]: Yeah, I think it was missing the ‘why.’ And I think also DDD has a lot of unique names or terminology that you didn’t hear before. So if someone tells you, you will learn about bounded contexts. You’re like, ‘What?’

Robert [00:23:10]: I know about design patterns, but what is this? Or maybe aggregate and, oh, okay, so I can aggregate some data.

Robert [00:23:20]: Okay, another reason why it’s not that widely adopted, and maybe now it’s very often missing interpret. It’s also the reason that point-driven design is out there already for a pretty long time. So the blue book, so the book that was introduced it. Introducing Domain Driven Design was released almost 25 years ago and the programming world 25 years ago was totally, totally different.

Miłosz [00:23:54]: And it was based on years of experience from Eric Evans, so I guess you can say it was about software written in the 90s. So it sounds like a different era today, kind of.

Miłosz [00:24:09]: But on the other hand, you can still hear about the DD a lot. Maybe not every company practices it, but there are conferences and books dedicated to it.

Miłosz [00:24:23]: So, probably suggest to you that there are many good ideas. That’s how evolution works. Good ideas tend to survive for longer.

Robert [00:24:32]: Unfortunately, there’s also a side that many good ideas are misunderstood with time. So some good ideas are useful at the beginning. More people are trying to adopt them. But unfortunately, if those good ideas are not straightforward, but require bigger investment to understand them. Some people tend to get shortcuts and not use them as intended or use them everywhere and it’s quick ending up with some kind of caricature of some techniques, tools. So in other words, many techniques or tools at the beginning were good ideas and were solving some problems, but with time, when people were adopting that, they were starting to adopt some caricature of this technique and it’s often started to do more harm than good. It was visible in discussions that we have in many teams that some people have seen domain-driven design in some projects, but it wasn’t really domain-driven design what they see.

Robert [00:25:38]: They’ve seen some overengineering, maybe, just doing it for the sake of doing it. Yeah, so using it in places when it wasn’t needed. So it was just too simple application maybe to use that. And to make it worse, it wasn’t used in there. Right way, because the initial investment was just a big end. People tend to not spend enough time to learn something and, Well, it’s easy to imagine that after 25 years, many, many of projects like that was created and it created a lot of bad PR.

Miłosz [00:26:17]: And you can hear people saying, ‘I don’t want to work like this ever again,’ because it was a disaster. It was a super complex project that was made overcomplicated.

Robert [00:26:28]: The similar situation is also, for example, with clean architecture, when we sometimes hear that, ‘ah, no, clean architecture doesn’t work,’ and later we hear about seven layers and.

Miłosz [00:26:39]: Yes, but on the other hand, if you read the introduction to the Global. I think the core ideas are still very relevant. The introduction describes the state of most software projects. How they get stuck, what are the issues. And now it’s almost 25 years later, everything pretty much in the same spot.

Miłosz [00:27:08]: Sense. We got better with frameworks and some technical stuff, but the core idea of working with people on the product.

Miłosz [00:27:21]: Not that much different.

Robert [00:27:22]: Are you suggesting that we are reinventing the wheel in software every 10 years? No, it’s not possible. And it’s 25 years during. since releasing the book. I can imagine that we reinvented the wheel at least twice since that time.

Miłosz [00:27:41]: microservices and then ditched them. Twice. It was a full cycle. For example, in the Eric Evans book, he says that XML would be a promising technology to communicate between services. Yeah, it’s something fresh to send XML over RPC. Yeah, so since then we probably reinvented this idea a few times. We’ve.

Miłosz [00:28:10]: microservices and protocol buffers over gRPC. It’s still the same. Coming back to the question why it’s difficult to understand what DDD is, it’s very broad. It’s like an entire book of many, many ideas. and from design patterns to communication between teams. So it’s very difficult to say what doing GDD means. It’s just very broad method.

Robert [00:28:43]: We can also say that it may be philosophical in some way. Hmm, hmm.

Miłosz [00:28:48]: Yeah, and you can interpret the patterns in many ways.

Robert [00:28:54]: The domain part of projects can differ between projects. So it’s hard to give one solution that fits all. So it’s probably because of that, but it’s also creating a lot of space to interpret things. Hmm. It remains one thing. If you are with us for longer, you maybe already know that some time ago we decided to leave our daily jobs. And now we are doing our own business. But it’s not the first time when we’ve been trying to do that. And we’ve been trying to do it multiple times earlier. But I think. One thing that probably didn’t allow us to succeed earlier was that we didn’t spend enough time to understand actually how to run business.

Robert [00:29:43]: How it actually worked that you’re running a successful business. So maybe you had some idea that if you would like to have a successful business, you need to sell things. Uh, do some marketing, do some sales, and earn more than you are spending.

Miłosz [00:29:58]: Easy, but just— you just need to read one business book, yeah, which is called ‘Business’ and explains how to do business.

Robert [00:30:07]: Yeah, but in practice, it’s not that simple and it’s more sophisticated, let’s say. And if you will not spend enough time to understand how it works deeper.

Robert [00:30:20]: The chance that your business will fail is much, much higher. I think the statistics are like in five years, 50% of businesses are failing, and in terms of startups, it’s probably 80% in Twitter or something like that. So it’s a crazy number actually.

Robert [00:30:38]: And on the other side, it’s something that you can learn, but you need to spend a bit more time to learn those principles and to not do. Caricature of your business.

Miłosz [00:30:50]: It’s similar a bit to DDD. It’s a very broad term and a general approach to solving and you need to learn many patterns.

Robert [00:31:00]: And it’s hard to give one advice that will fit all cases. But it’s doable, but you need to spend some time and keep in mind that many, many, many people are not doing this exercise and are later surprised that it doesn’t work.

Robert [00:31:18]: Like with DDD, you can also see that many people are saying that running business is super hard.

Robert [00:31:23]: So I would say that it’s hard, but I think it’s also not. That hard that people think. It’s just about knowing the principles and spending enough time on it.

Miłosz [00:31:36]: Right, so if you were to boil down DDD to one primary theme, I would say something like it’s about understanding the domain you work with and then modeling it well in the code. Whether you call it DDD or not, it’s like a mindset. More than patterns, maybe?

Robert [00:31:57]: In other words, you should always do that. I mean, if somebody will tell you that, okay, you should not think about the domain, what problems you are solving in your software and how you are writing in an account like it sounds crazy.

Miłosz [00:32:12]: Yeah, it’s like you’re deciding. Yeah, I will create a website. I just don’t know which one. So it sounds like you do something for the sake of learning the technology, which is fine. Sure. But if you want to create a product, it’s not really helpful.

Robert [00:32:32]: My observation is actually that dark.

Robert [00:32:37]: Two kinds of people that are often visible in the area of techniques like domain-driven design, architecture, take anything.

Robert [00:32:46]: There are two comps that are totally opposites. One comp is saying that you should always use that. Because it’s great, etc. And the second is you should never use it. That is the worst idea in the world. And I think it’s pretty interesting that even if they are opposite, they have many in common because they are extreme in some way.

Miłosz [00:33:06]: So they treat it more like kind of religion rather than something you want to use to deliver a product.

Robert [00:33:17]: And in practice, probably you should be somewhere in the middle.

Robert [00:33:23]: It may sound obvious, but you should use proper technique for a proper problem.

Robert [00:33:29]: That’s it.

Robert [00:33:31]: It’s important when you see some religious people about techniques or tools that are against them or love to use it always that it’s the same coin, but just other sides of it.

Robert [00:33:46]: I think it— gets into one thing that you should watch out to not use any techniques or tools like DDD just for sake of using it. So you should identify the problems that you have and use the right tools for that. And the Winterwind design have couple tools that you can take out of that like a tool from a toolbox and just use what you need from that.

Miłosz [00:34:12]: There’s probably no project that will use every pattern described in the book.

Robert [00:34:16]: Yes, but probably you agree that every project should use some parts of domain-driven design. So it may sound controversial, but it’s getting into a point that you may be using some parts of domain-driven design. Even without realizing that. It may be because of reinventing the wheel. Again, it’s probably easier to just see that, okay, you have some tools like that, but many of them, some people just can find them, yeah, probably except for very trivial projects, but we don’t care about them here.

Miłosz [00:34:51]: Like, if you create something that’s very trivial and doesn’t need any kind of design and architecture, and you probably don’t need to figure out how to do it better.

Miłosz [00:35:05]: So we don’t worry about this here.

Miłosz [00:35:09]: Maybe also something very technical like Linux kernel. I don’t know. But probably also there should be at least one idea from DDD that applies there. Maybe value object.

Miłosz [00:35:21]: Yeah, but in general.

Miłosz [00:35:24]: If you boil DDD down to some very key ideas, I think you should be able to use them if you know what you’re looking for.

Robert [00:35:34]: I think it’s a similar situation like with the book Design Patterns. So pretty old one. I don’t remember actually how many years. It’s older than DDD. But I think 96 or 97. So still, pretty old book, but still pretty relevant and many of us may be using patterns from this book. Even without knowing that those are patterns from this book, because it was used by somebody else and we learn those patterns by replicating them. But still. We may be missing some of the context maybe of them. And if we read that, maybe we could understand them a bit more and apply them in the better situations also.

Miłosz [00:36:15]: I remember there was a joke or people making fun of people in interviews. When asked about design patterns, they will tell something like, ‘Oh, yeah.’ There’s the singleton, and that’s it. But that’s also kind of the point. You know, there’s an entire book, and doesn’t mean you need to know all the patterns and use them all in a project— even if you learn about some, like maybe you can use a strategy here or decorator in another place. I think it’s still a good result. You don’t need to force them and rewrite your entire project to use, I don’t know, abstract factory method or something.

Robert [00:37:00]: It’s always about identifying some problem that is really a problem, and not solving.

Robert [00:37:08]: Some imaginary problems. Yeah.

Miłosz [00:37:10]: Especially if you force the patterns. I think that’s the worst thing that can happen because it just leads to overengineering. You just read the book and you like the ideas, so you use them everywhere in your next projects and of course they don’t apply to all of them.

Robert [00:37:27]: And I think it. It plays nicely with a couple of things that we covered in the previous episode that we’ve mentioned: that being a principal engineer doesn’t mean that you’re forcing to use some patterns. You should instead be able to convince people what value those patterns give and and convincing them that it’s worth using some patterns there. Hmm. And at the end of the day, it’s also a matter of solving your problems, not imaginary problems.

Miłosz [00:37:59]: There is no one solution to suggest. Because there are so many factors in play, like the size of your team. If you are a 20-person team in a corporation, a developer working on open source software, or maybe you are a five-person startup, you will probably use a very different approach to software.

Robert [00:38:22]: And also, what’s the timeline for the project? Because if you are in a startup that you’re starving for, if you don’t know if you will survive half a year, and you’re doing refactoring, it’s probably not the best idea, but I mean, it depends, obviously, but it’s again going back to solving real problems that you have. So, if you are a startup that has a half year of runaway and you’re fighting for survival, you should choose your fight wisely, basically, and not put maybe refactor some code. That you didn’t touch for a year and put design patterns there. But maybe you have some part of the application that, for example, you are doing a pivot now and you need to— it quickly, but you’re wasting a ton of time to modify this code, maybe doing this refactoring and putting those design patterns may be good at the other, even if you’re fighting for survival.

Robert [00:39:26]: Because maybe it may allow you to build this new functionality or do this pivot within this time that you have left.

Miłosz [00:39:33]: Have some idea that this project will be for you longer. You probably need some better maintenance here. But but maybe some some areas are more less important. Maybe you will get rid of them in two months if it doesn’t work out. But because it’s sometimes difficult to decide.

Miłosz [00:39:57]: This might be a bit controversial, maybe, but I think you don’t need to go all in. Most of the patterns from DDD you could decide to use them in a more, let’s say, shallow version, other than deep version. So you basically think of the the first principles, the big idea behind the pattern. You don’t need to implement all of it and still get some good results. Because you often don’t know if the project you work on will be with you for longer, or maybe it’s just a prototype.

Miłosz [00:40:33]: But I think there’s a balance between having some very scrappy code and something that’s very enterprising. While design software.

Robert [00:40:44]: It’s actually an interesting thing because it’s one of the things that Domain Driven Design covers with core and supporting domains.

Robert [00:40:56]: It’s kind of forcing you to think, ‘Okay, is this code really important to my business, or is it less important?’ And it can kind of affect how we are thinking about this code.

Robert [00:41:09]: I know that many people like consistency in the code, like, ‘Okay, let’s do.’ Something in the consistent way everywhere. End. It may sound like a decent idea, let’s say, but it may be also dangerous because you at some point start to over-engineer some parts because some parts of the application are more important, more often touched, more complicated, and some are simpler. You may just lose a lot of time for maintaining this consistency for sake of nothing. Basically.

Miłosz [00:41:44]: So this episode is more like an introduction, but in the next episodes we want to cover more patterns from both shallow and deep end, so you can better decide what to use.

Robert [00:41:58]: But I think we should put big emphasis on the thing that people are very often missing. So the strategic part, because Tactical patterns— those strategic domain-driven design— are the patterns that are not that much about how you write the code, but it’s more about strategic thinking, like how the modules are interacting, how modules are split. Very often cross-team collaboration.

Robert [00:42:28]: How important are multiple parts of the code? And what’s interesting, everything what I just said, I think you should probably think about that in any project that you are doing. You’re using domain-driven design or not. It’s cool to have some tools that are helping you to do that, and some frameworks— let’s say it’s not a framework, but some way of thinking how to approach this. And from other side, you have technical patterns that you’re using in the code, and those ones are not always needed. And you don’t need to always apply them.

Miłosz [00:43:01]: So we won’t be diving deep into patterns today, but let’s look at the core idea of DDD.

Miłosz [00:43:12]: The domain model or model-driven design or knowledge crunching, as it’s specified by the book. What is it about?

Robert [00:43:23]: There’s one thing in multiple projects that is a bit like running business, like we said earlier. So it’s obvious that you should earn more money than you are spending. so you have a successful business. In software, it’s a matter of understanding what kind of problem you are solving and how it should work.

Robert [00:43:45]: It makes all of us like, yeah, we are starting a project. We should understand how it works and how we should implement that. From other sites It’s not happening.

Miłosz [00:43:59]: yeah because sometimes the business is complex by itself and even for simpler ones like consider e-commerce seems like a simple domain it’s just people People buy products and that’s it. But if you dive deeper, there’s capturing payments and delivering the products and getting the inventory somehow. So it’s more complex than it seems.

Robert [00:44:24]: How it looks in practice? People are thinking, ‘Oh, okay, it’s simple.’ We know what to do. Let’s go to implement that. Exactly. Two years later.

Miłosz [00:44:33]: Consider, if you wanted to to open an online shop.

Miłosz [00:44:38]: Oh. If you ask a software developer, they will probably tell you, ‘Yeah, you just need to host this WordPress website. With an e-commerce plugin and you’re done.’

Robert [00:44:52]: I think many software developers will say, ‘No, no, we need to implement it from scratch.’

Miłosz [00:44:57]: Oh yeah, that’s even better. But my point is, they probably won’t think about that— that you need to. Order the inventory from somewhere, store it somewhere, and then somehow send it out to people, which is the hard part of the business and your software probably needs to work with it somehow.

Robert [00:45:16]: It actually works nicely with this part of strategic patterns of domain-driven design when we are considering what’s our core domain and what’s the value for our company.

Miłosz [00:45:27]: And many projects are much, much more complex than e-commerce. If you consider accounting, for example, you can get into some very very specific edge cases depending on the country and if you do do it globally, then it’s yeah, it’s crazy amount of knowledge you need to understand and to implement.

Miłosz [00:45:48]: Complexity in those projects doesn’t really come from the technical challenges, but from the business itself. It’s just complex to run this kind of business, so the software becomes also complex.

Robert [00:46:03]: So, in other words, you have multiple dimensions, complexity dimensions, let’s say, to this product, and you can add more and more and more. So, for example, you’re starting with one country. Right. We need to scale globally. Okay. We need to support multiple countries, multiple continents. Great. Later, we need to support multiple types of product. So it can have multiple tax rates and multiple rules. It’s getting more and more and more complex.

Miłosz [00:46:28]: And it’s always complex even before you introduce any kind of technical challenges. Just not considering software at all. This is already complicated.

Miłosz [00:46:39]: So you want to model it well in the code, so at least you don’t need to deal with the additional complexity.

Robert [00:46:51]: And to make it worse, often the initial idea is also changing with the time. So I have two good examples of the companies that were like that. So Netflix.

Robert [00:47:01]: Maybe many of you know that Netflix at the beginning was an extreme platform. At the beginning they were just renting DVDs. So it’s interesting how it changed over time, how they were making business, and probably how much it changed how they’re operating. The other interesting business that changed over time is Instagram. Instagram wasn’t this kind of social media platform that you know now; it was more checking location applications. So I guess you were able to send people information like, ‘Hey, I’m here’ or something like that, not share pictures.

Miłosz [00:47:40]: Yeah. And so imagine in both cases.

Miłosz [00:47:45]: At some point, developers complained that the requirements weren’t clear. We didn’t know this component would become this thing it is now. Of course, it wasn’t clear because no one knew about it back then and they had to adapt somehow.

Robert [00:48:00]: Those examples were more extreme, let’s say, but most companies are changing their initial assumptions pretty much, so it’s also adding this complexity there.

Robert [00:48:14]: I also like example of the cyberpunk game that the cars are horses from Witcher 3 because they’re using the same game engine.

Miłosz [00:48:25]: Yeah, so that goes back to the discussion of evolving software. The first version is usually simple, right? Everything works fine. But then someone tells you, actually, we’re not renting DVDs. We’re going to be streaming videos.

Miłosz [00:48:42]: And at this point, you have to change your software to a completely different model sometimes.

Miłosz [00:48:52]: It’s great if you could do it easily, if you don’t need to spend one year to rebuild everything. What helps is also what we mentioned in the last episode about developers working directly with domain experts.

Robert [00:49:06]: So even if it’s a bit controversial, so we’ve seen how it can change things and again affect how you are implementing things.

Miłosz [00:49:17]: So we don’t need a product owner or product manager acting as a proxy?

Robert [00:49:22]: Always, because sometimes it makes sense, but sometimes it makes sense to just go direct.

Miłosz [00:49:27]: But the idea of the domain model is to create a model that represents the problem domain as well as you can at the moment.

Miłosz [00:49:37]: Developers can understand how it works, the more information they get from the people who work directly with the business. The better. The model doesn’t need to be physical, so it’s not a document or diagrams or anything else, although it helps, but it’s more abstract. It’s about understanding how it should work. And.

Miłosz [00:50:03]: The ultimate artifact is the code.

Robert [00:50:06]: And it also doesn’t need to cover everything. So I always like to compare this model part to a map. So map is not a perfect representation of real world, but it’s a useful representation of the real world. So you can look on the map and see the details that you care about and do the job that you need.

Miłosz [00:50:28]: And it’s also not a data model, which is a common misconception.

Miłosz [00:50:35]: Very often in software you can see models used for both the domain logic and some storage, for example, or transport.

Miłosz [00:50:49]: For developers this is unnatural.

Miłosz [00:50:52]: No. thing to do treat them as the same so we often start implementation with database schema, for example, because it feels natural to think of storage.

Miłosz [00:51:03]: But here we think about the more abstract domain model, how the thing works, what behaviors it has.

Robert [00:51:11]: So this is pretty big part. Paradigm shift and it’s also not consistent with how we’ve been learning earlier. So this is also the theme that was pretty common in the last two episodes. But how we are learning to code is very far away from the real world. And we are not prepared actually to work in the companies. And later it’s ending up with the.

Robert [00:51:37]: golden cage concept from the previous episode. Getting out of that is not simple, but I think it’s mandatory if you would like to be successful and survive on the tough market and work in the better places.

Miłosz [00:51:52]: Another tool from the previous episode is event storming, which is also closely related to figuring out the domain.

Robert [00:52:01]: So today we’re more focusing focusing on giving you some introduction.

Robert [00:52:09]: To— Maybe not Domain Driven Design really, but some parts of Domain Driven Design that we’ve been using in the project. Sometimes we’ll name it Domain Driven Design, sometimes not, because again, its patterns are secondary. I think it’s more important to focus on problems, not on the patterns, because when you are focusing on patterns, you are just with this hammer in your hand and looking for nails everywhere. So I think it’s more important to focus on problems that we’re solving and how we can do that. And yeah, this is something that we’ll be doing in the next couple episodes and it’s connected to the fact that now we are working on our new training the domain engineer and that will be available at the beginning of next year in January.

Miłosz [00:52:56]: Yeah, so in the next episodes we will look at more use cases for the strategic patterns.

Robert [00:53:02]: Yeah, so we’ll give you some more practical tools also, how to approach that.

Miłosz [00:53:08]: So let us know in the comments if you have any topics you would like to hear about. You can also check our ebook, ‘Go With the Domain,’ which is about working with domain-driven ideas.

Miłosz [00:53:22]: In Go specifically.

Robert [00:53:24]: Yeah, so we will be preparing agenda for next episode soon. So let us know on YouTube comments or Spotify comments. So read all of them and it will be. Cool to help you with some projects or problems that you have. We have many, many, many stories from projects that we’ve been working in, but we’d also love to introduce a bit more there.

Miłosz [00:53:50]: So that’s it for today.

Miłosz [00:53:53]: Subscribe to our channel so you can see the new episodes as they come.

Miłosz [00:53:59]: And hit that like button.

Robert [00:54:01]: And if you don’t want to miss when the domain engineer will be available, just Google the domain engineer and join our newsletter if you are not there. Yet. So we’ll send you an info when it will be ready. And you will also notified when new episodes will be out, because we know that sometimes notifications from multiple platforms are not sent. So when you’re in our newsletter, you are sure that you will always get the notification.

Robert [00:54:29]: Okay, so I think that’s all for today. So thank you, Miłosz, for your time.

Miłosz [00:54:33]: Thank you, Robert. See you next time.

Robert [00:54:35]: See you in two weeks. Bye-bye.

Becoming a Product Engineer: First Steps

Wed, 26 Nov 2025 12:00:00 +0000

Quick takeaways

Developers are often kept in “golden cages” - treated as coding monkeys receiving tasks without understanding the broader context or product vision
Requirements are usually incomplete or wrong - understanding the business context helps developers spot issues early and deliver what’s actually needed
Collaboration with stakeholders is crucial - developers have valuable product insights and should be involved in planning, not just implementation
Event Storming is a powerful collaboration tool - a lightweight workshop technique that brings developers, product managers, and stakeholders together to align on complex topics
Change takes time and trust - transforming company culture requires starting small, showing results, and convincing people through solving their actual problems

Introduction

In this episode, we talk about why software projects in regular jobs are delivered much slower compared to side projects, and what you can do about it.

We share our journey from building hobby projects as teenagers to working in professional environments, and the differences we encountered.

Developers are often kept isolated from product decisions and treated as “coding monkeys in golden cages” - just receiving tasks without understanding the why behind them.

We discuss techniques like Event Storming that can help break down these barriers, and improve collaboration between developers and product managers.

Notes

Event Storming - a workshop technique created by Alberto Brandolini for collaborative domain modeling

Quotes

Five days of coding can save one day of planning.

Robert

If you want to be promoted, for example, delivering is the very best way to do it. So instead of just waiting for requirements and complaining that it doesn’t work, you can try to help whoever needs what they need.

Miłosz

How it often looks like: it’s like companies are trying to make the programmers coding monkeys in some golden cages, basically.

Robert

Instead of complaining that we never get anything done, you tell them, yeah, this feature could have been one week, but we spent two months.

Miłosz

The worst thing that we noticed is that the performance of how fast we’ve been able to deliver new stuff was much worse.

Robert

But what if instead, you have this domain expert, this stakeholder meet with developers directly and tell them what they need. And then usually developers will tell them, oh my god, this is so complicated, it will take two months. But if we drop this one button from the dashboard, it will take one week.

Miłosz

Timestamps

Transcript

Robert [00:00:59]: I’m Robert.

Miłosz [00:01:00]: And I’m Miłosz. And this is No Silver Bullet Podcast, where we discuss mindful backend engineering. In this show, we share takes on software engineering that will help you grow into a principal engineer.

Robert [00:01:11]: In this episode, maybe let’s start with a bit of our story, because I think it’s pretty important to understand that, how we started to build some production systems and from where our experience came. So probably some of you know that we met with Miłosz in the high school, so a pretty long time ago. We were just talking before recording. That it was, I think, 17 years ago, so a pretty long time ago. Actually, I know Miłosz two months longer than he knows his wife.

Miłosz [00:01:42]: That’s pretty long, and the reason I know you longer is because we are already working on projects before everybody started. So we started with some toy websites and then grew into more complex projects.

Robert [00:01:59]: And many of those projects actually became pretty successful, so many people were using them. The downside was that, well, there wasn’t great business sense at the time, and well, a lot of people used that, but it wasn’t enough for making for a living of that. So we decided, okay, let’s go to university. Maybe we’ll learn something useful. Long story short, we did it. So we left the university at some point. So we quickly dropped out. Yeah, because, well, it was probably a bit boring since we’ve been already coding for multiple years and we’ve seen that, OK, maybe we’ll learn physics and mathematics, but… That’s not probably relevant in the software that we are writing.

Miłosz [00:02:41]: We still work on our projects in the meantime, but there was almost no money coming from it. We had to just start regular jobs, which was fun at the time, because we didn’t have this professional experience, let’s say that’s our own.

Robert [00:02:55]: I just remember that. During the time, the market also wasn’t that easy. I mean, we’ve been over decoding for a couple of years already, and I was already afraid that I will need to go back to my parents’ home because I I struggled a bit to find a job, despite having a couple years of experience already.

Miłosz [00:03:11]: Even the basic wage we made back then was great for our student life, compared to hobby projects.

Robert [00:03:21]: That’s true. But I think it wasn’t worth thinking about that because, since we’ve been working already on multiple projects, we’ve been working, so we have two people— team basically, so me and Miłosz. It was a bit shocking to start to work in the normal job, let’s say, let’s name it in this way. So, you know, in the way that you are no longer your boss, but instead you are getting requirements from somebody else. You are not inventing the project and product that you are building. It’s totally different dynamics. And for some people, I know it’s fine because they don’t need to touch this ugly business and product they can just code. But we’ve been different. We’ve always wanted to create something and invent something. And it was a bit shocking.

Miłosz [00:04:08]: The upside is you don’t need to worry about… If the mana is coming through. Someone else handles that for you. But alongside this, someone tells you, here’s this system that has been around for 10 years and you need to add some feature to it. And usually it’s not an easy task.

Robert [00:04:27]: Yeah, yeah, exactly. And you know, you have other challenges, like you need to collaborate with more people. So it’s no longer two of us, but the team is bigger, you need to take maintenance into account more because there’s 50% of a chance that it will be the person that will be maintaining it later. In this case, it’s totally different because you’re working with someone’s legacy code that some person who worked here 10 years ago wrote. This person no longer works there. Code that we have written 10 years ago is still running out of there.

Miłosz [00:04:58]: You have to agree with other people in the team how to do it, probably, because you are not the expert anymore.

Robert [00:05:04]: So it’s creating a lot of challenges.

Robert [00:05:06]: Those challenges are pretty important because the worst thing that we noticed is that the performance of how fast we’ve been able to deliver new stuff was much worse. And in this episode, we will discuss some tools and techniques that should help with that.

Robert [00:05:27]: Can give you some tools to handle situations when you are not working in a solo project, but in a team and you need to collaborate in some way and create software that is maintainable. In longer term than one month. So can you keep the same pace as when working with two people or just a solo developer? So if those problems sound familiar for you, please stay with us. So we will give you some hints how to overcome those problems when you are working in a daily job. So at the beginning, let’s think why projects in your daily job are delivered slower. So in other words, why, when you are working with your friends after work or on your pet project, you are much more productive? What do you think it is? So we already mentioned one thing: when you’re working on the project alone, you are basically your own boss and you’re working on your ideas, which is great. But the downside is that it’s also much harder to make business out of that. I mean, we are software engineers, we are good problem solvers, but from our experience, very often we are not best businessmen.

Miłosz [00:06:32]: Yeah and we often start side projects and you know, some websites maybe web apps, but often it’s just something for fun because it’s for learning. And this is how we often are learning, and it’s great. Yeah, but making it into a business is a much different story.

Miłosz [00:06:52]: Which might be also why developers usually in companies don’t have that much impact on the product side. So very often they are kept in a silo of software development.

Robert [00:07:10]: Yeah, I actually like to name it a bit more brutal— and they need a golden cage. So and how it often looks like: it’s like we are companies are trying to make our programmers coding monkeys in some golden cages, basically. I know it sounds brutal, but this is how it looks in multiple components, basically. So what do you mean by coding monkeys? So there are a couple elements of this problem, let’s say, and also a couple of levels of how companies are approaching this. So the most extreme scenario is when you’re this coding monkey in your golden cage and you’re just receiving tasks to code. I even heard about worse cases when some developers even received empty classes and they were diagrams, empty classes and some UNLs and just please feel the methods and don’t care about anything else. It’s similar to LLMs today, just like a human instead. Yeah, yeah.

Robert [00:08:13]: And funny thing, if many people were working like that entire life, well, I can imagine that they may struggle now a bit because… So this is actually, it’s not very performant at the end because the software that is created in this way, obviously the chance of working is very, very low and making some… value. But still, it can be replaced in some way.

Miłosz [00:08:36]: So engineers in this scenario are just kind of implementators of some idea. From the outside. I think especially in product companies it might be an issue because developers usually have great insight into the product area as well.

Miłosz [00:08:53]: It can work better in some very technical domains where you need developers just to work on some specialized algorithms or something like that. I guess even waterfall can work in such a scenario.

Miłosz [00:09:09]: But if it’s a product company, it’s much different.

Robert [00:09:12]: Some of you may wonder, ‘Okay, why is that?’ I can just sit in my nice cage and code. It’s great. But… Unfortunately, it’s pretty unproductive. If you’re working with software a bit longer, you probably already know that requirements that you receive, they’re in most cases… Maybe not wrong, but…

Miłosz [00:09:30]: Incomplete. It’s probably the biggest issue developers have in their day-to-day work. I can’t work on this. This is not clear. This is missing some requirements. And it’s discouraging because you can’t do anything with it. You can just, you know… tell your product owner or manager or whoever that, sorry, this is not complete. I can do anything with it. And then the product owner needs to go back to whoever knows what to do and figure it out and come back to you with clear requirements, which is never the end of it.

Robert [00:10:08]: And needless to say, it’s super unproductive. When you’re working alone, you can ask yourself, ‘Ah, okay, it doesn’t make sense.’ Maybe let’s do it in this way. When you have proxy… Project manager well, it’s very ineffective because probably this project manager also doesn’t know. So this project manager needs to go to another person, that person ask another person, and it can also probably explain why often those requirements are not perfect.

Miłosz [00:10:36]: And worst case scenario is probably when the product owner also decides who gets to do what.

Miłosz [00:10:45]: Split the tasks between developers so the workload is even across the team. So that’s probably the biggest anti-pattern you can see in project management.

Robert [00:10:57]: Yeah, exactly. Because, well, if the project manager is assigning tasks to each developer, probably he may not or she may have not enough context to know. Which person has most knowledge there, or maybe some other things that if team will be responsible for assigning those tasks. It will be much more effective because they have more context to do this decision better.

Miłosz [00:11:23]: Yeah, that’s a better approach because the team knows what they specialize in, so they can share the knowledge as well between the team members. Everyone knows a bit about what’s going on in the project. It’s not like there’s one person responsible for everything.

Robert [00:11:44]: And I think it’s quite interesting because it’s based on crazy and pointless ‘why’ to work in this way. But there are multiple reasons why, in many companies, it’s the default mode. I remember that.

Robert [00:11:58]: In many components, when I was starting at the beginning, it was unfortunately the default mode of working, and it was often some work needed to change that. I mean, it was possible and it wasn’t that hard at the end when you’ve shown what problems it’s generating, but… Yeah, unfortunately, it’s often the default way of working. Do you have some ideas maybe why companies are doing it in this, well, probably crazy way of assigning tasks to people and closing them into the cages?

Miłosz [00:12:32]: I guess maybe it’s just the way things always used to be.

Miłosz [00:12:38]: Just the default mode of, you know, we have those developers who are implementers and product people who come up with ideas. So the development team becomes the downstream of those ideas.

Miłosz [00:12:56]: Ping pong between each other.

Robert [00:12:59]: Yeah, I also found one interesting reason for that. And the reason was that developers are expensive. So their time is expensive. So it means that we should use as small amount of their time as we can.

Miłosz [00:13:14]: Yeah, basically use as much as we can for all the tasks we need.

Robert [00:13:20]: So basically, let’s optimize developers’ time so they can code as much as they can because, well, we’re paying them for coding. This is why we’re paying them that much.

Miłosz [00:13:28]: It’s basically like in a factory. You want to utilize all the devices to do the work for you.

Robert [00:13:37]: And you’re trapped, because, okay, from one side you are… ensuring that those developers are using 100% of their time on coding. But if you think about that a bit for a bit longer, you will notice that, okay, maybe they’re coding more. But useless things, because even if we will force them to code some stuff, and they will write them after it will be shipped to production after in this kind of companies, half year let’s say, you’ll see that, okay, it’s actually not what we needed. The worst part is it all feels like a good job. Yeah, and for the developers. You may also feel very productive, like I’ve been coding for 12 hours and it’s so great.

Robert [00:14:18]: Yes, but there is nothing to show for it. You more like wasted the time. Probably one sentence here will be very accurate. And I really like to remind it over and over. So five days of coding can save one day of planning. And this is basically that. Yes. So how…

Miłosz [00:14:35]: talking about or coming up with ideas how something should work feels much less productive than just typing away and creating this. stuff. So we are very eager to jump to implementation, maybe build some prototype. Sometimes it’s useful, but very often it would be better to spend some time. Trying to understand what you really need to build, what will be useful for the users, for example.

Miłosz [00:15:06]: But it doesn’t feel as productive as this starting coding. So developers don’t really have an issue with this most of the time.

Robert [00:15:15]: So let’s all go maybe one level up in our ladder of how good or bad putting developers into cages can be. So the second level. That I noticed in some companies is some a bit better situation. So in this situation, product owner is going to the team. With some requirements gathered from stakeholders. And in this case, the product manager or product owner is not the person who is creating the tasks, but it’s the person who discusses basically what we need with the developers and he’s working or she’s working with developers to learn how to implement that.

Miłosz [00:15:57]: So they present some high-level idea and then developers split it into smaller tasks or come up with implementation, maybe propose some solutions.

Robert [00:16:06]: So we no longer receive tasks. It’s no longer assigned to people. We can talk to our closest stakeholder and discuss what we would like to implement.

Miłosz [00:16:17]: You get even better knowledge sharing this way. So everyone is on the same page. Also easier to work with code later.

Miłosz [00:16:24]: Because everyone has insight how this will impact the end product. You don’t just work on some obscure subtasks on the other end of it. You see the full picture. But there’s also a challenge here, right? It’s not really common knowledge how to do it.

Miłosz [00:16:44]: It can take long hours to come up with the final solution.

Robert [00:16:49]: And can go into never-ending meetings.

Robert [00:16:55]: My observation, obviously, it’s always about balance. But my observation is that often it’s when you have hours of discussion. It’s basically alternative because sometimes even better alternative. I mean, the worst alternative can be that, okay, you are going to saying that, okay, ‘We are discussing already for too long.’ Let’s go. Let’s do what we think should be done. Yes. And later, you’re not spending hours on planning, but you’re spending weeks on implementing something just later to see that why we did it in this way. It’s crazy. It doesn’t make any sense. And nobody’s using that.

Robert [00:17:33]: So I think it’s one popular trap that we’re trying to optimize this discussion part. Sometimes, as you said, we don’t have good tools how to discuss requirements with product owner and/ or product manager. We’ll get into that soon. So we’ll give you some tools.

Robert [00:17:50]: Yeah, it can lead to that, but it’s, I think, important to ask ourselves the question, what’s the alternative here? And often, the alternative is not the best.

Miłosz [00:18:00]: outcome of this can be simplification.

Miłosz [00:18:03]: I think you need skilled developers that can have this. Product vision a bit and can talk with product owner, for example, and suggest some solutions. But because often the reason the meetings drags on so long is that we talk about two big problems at once. So we want to create this full system or big feature up front. And there are many edge cases. Developers love poking those edge cases and they will come up with many many issues. So then you have like 20 things you need to tackle before you start the development because there are so many edge cases to cover. And very often, if you start with a simpler version of the same feature, it may be much, much easier to discuss and start. And also very often you… you later realize and actually, you know this is like 80% good, maybe we don’t need most of what we thought we needed in the beginning.

Robert [00:19:09]: I think it’s some kind of meta advice that it’s already coming in multiple episodes that we had earlier. So some golden standards that we have tried to have in teams that we’ve been working in. To be able to deliver each task within one day and to be able to deliver some, let’s name it, story within one week. Because, if you’re implementing, you’re creating some functionality into something existing. But it helps us to… Simplify those discussions because, if you are discussing some feature for multiple hours, and you would like to implement within one week, you already see that okay it cannot happen. Because, if you are discussing some feature for multiple hours, the implementation of that will just take much longer. So it’s good because you can start a discussion like, ‘Okay, it’s already scope creep. Let’s try to figure out what we can do. Some kind of how we can make some kind of MVP out of that, and start with that, and iterate on that. I said that it’s working for features that you’re already building on already existing functionality.

Robert [00:20:17]: Because, in this case, you can do it faster. It’s a bit different when you are doing kickoff of new project. Because, if you are doing kickoff of some project, sometimes you need a bit more time to have something that is functional. So, but I would say that you should also watch out to not exceed, let’s say, one month. For example, and have in one month some MVP that already delivers something, and somebody can already use that, and you can already receive some feedback on that. And again, I think it may sound very obvious, but…

Robert [00:20:48]: It just works, basically. So it’s one of those advices that are too simple to be true, but this is how it works.

Miłosz [00:20:57]: Splitting work is not easy, for sure. We’ll probably need another episode on this. But coming back to this… with product owner. This is probably an upgrade on the previous approach.

Miłosz [00:21:09]: Product owner deciding on everything themselves, but it’s still… not perfect, right? Because this is a single point of failure. This one person, they understand the domain, they talk with stakeholders or users.

Miłosz [00:21:24]: But they still don’t know everything. So they are kind of a proxy between the development team and whoever needs the product built.

Robert [00:21:34]: And if you are working this way, probably more than once you have situation like you are asking some question to your product owner, he or she is seeing you for a while and I don’t know, I need to ask this stakeholder. I will go back to you in a week.

Robert [00:21:52]: And it’s good because if this question is there, because maybe it will save you one week of implementation that it will be useless at the end. So that’s good. But from other side, you need to often wait for some time for having those.

Miłosz [00:22:06]: Yeah, I remember those planning sessions where we were deciding what to… Take on for the sprint for the next two weeks, and of course, we poke holes in some plant features. We see some magic cases that are not solved, and the product owner says, ‘Yeah, you’re right.’ So yeah, I need to go figure out what to do. Meanwhile, here are some well-defined but very unimportant tasks you can do in your sprint. So you postpone the important work this way, and because you don’t have anything better to do, you start working on something smaller but probably something that’s not really game-changing for the company.

Robert [00:22:50]: But sometimes it’s good, because very often you have this backlog of things that we should do later, a . k . a.

Miłosz [00:22:57]: never. Sure, but this way you never do these game-changing features. Probably the product needs the most. That’s true.

Miłosz [00:23:08]: Because they are so important and risky, no one wants to just… start doing them with some simpler approach. It feels like we need this perfect implementation or specification for it. So you wait until you have it planned perfectly.

Robert [00:23:28]: So what’s the crazy alternative for that?

Miłosz [00:23:30]: Because I know— because we tried this crazy alternative.

Miłosz [00:23:33]: Yeah, so if you go one step further, I guess you could have… developers talk with stakeholders directly? Oh, no, it sounds so expensive.

Miłosz [00:23:44]: Or, you know, just domain experts in general, whoever this is, I don’t know, maybe your users or your CEO or whoever.

Robert [00:23:54]: That are often the source of those requirements. So there are people with knowledge, there are people with some interest, let’s say, but basically… those are people who give you the direction of work that you are working on. And those are the people to whom your product owner is going to ask how to do something. Exactly.

Miłosz [00:24:15]: So it might be your VP of sales or your operations specialist in another team.

Robert [00:24:23]: Or even your CEO. Depends on the company.

Robert [00:24:27]: Um, Or it may be also an operation theme. So it really depends on the functionality that you’re implementing.

Robert [00:24:34]: Usually, if you’re implementing some functionality, there is some person that is mostly interested in that and the idea maker.

Miłosz [00:24:44]: go to your product owner and they will tell them I need this new dashboard built and they will reply okay I will talk with my developers in two weeks later.

Miłosz [00:24:58]: I will let you know two weeks later what we think. But what if instead, no, you have this… This domain expert, this stakeholder meet with developers directly and tell them what they need. And then usually developers… will tell them, oh my god, this is so complicated, it will take two months. But if we drop this one button from the dashboard, it will… take one week. And then the stakeholder says, oh, really? I don’t need this button at all.

Miłosz [00:25:31]: Wow, impossible. Now we save two months of development and also two weeks of planning to figure this out.

Robert [00:25:39]: And this is not a story that we invented. It’s actually what happened multiple times for us when, basically, we could spend ages to implement. Something that the stakeholders would say that it’s not that important. But if we have proxy as product owner, it will never happen because the communication between developers and stakeholders will never happen. And to be exact, so it’s not about gathering all stakeholders in the conference room and discussing. Everything because it will be super inefficient. We’ll go into tools very soon. It’s more about having them meet with developers in some way. At some point. And also not doing it for every feature. So to be exact here. So we’ve been doing it for some bigger project kickoffs. So we’ve been not doing it for… every story that we’re implementing because it would be super expensive, especially that our stakeholders were living in a different country. So it meant that gathering, for example, five stakeholders, and discussing everything with them, would cost flights, hotels, etc.

Robert [00:26:46]: So it was pretty expensive, but still it will work for us and we’ve been doing it multiple times because, if it wouldn’t, it will just…

Miłosz [00:26:55]: Yeah, but this is a bit extreme scenario, right? When we fly over people to meet in the conference room. But this same idea can also be applied on a more simple level.

Robert [00:27:04]: It’s easier if you have stakeholders nearby, let’s say.

Miłosz [00:27:09]: Yeah, but even if you have someone in another team that uses your software and they have some issue, you can just have a call with them and have them show you directly what’s the problem they have. It’s the same idea applied to a more shallow level, so you don’t need your… Thank you. product owner to be a proxy between you two. You just talk directly to them and you understand, okay— this is the issue they have. So now I will figure out how to fix it instead of waiting for perfect requirements to figure out what to do. But I think the basic idea here is that it’s difficult to come over, having developers think about the solution and offer some simplification. And I remember some product owners trying to do it themselves. They somehow… In a way they knew the software developers were working on and they tried to figure out themselves how complex it would be to add a feature here or there. It’s a disaster always. I guess today they will ask AI which feature is the most complex and how I can simplify it.

Miłosz [00:28:27]: Because it won’t work, because you need this expertise to understand what’s going on really. That’s why you need to close the gap, have developers in a discussion.

Robert [00:28:40]: Yeah. So to maybe recap this. So I would say that there are three or four levels of how you can basically be engaged in developing the feature.

Robert [00:28:56]: The top level is pretty close to how you are working on this feature alone. In this situation you are together working with stakeholders— people that basically can make decisions if the feature can look in this way or not, because this is very important. So to have people that you can ask, okay, can we… skip this and they can say yes, no, and you can go further and not wait one week for decision. It doesn’t need to be like that for every story. It’s probably better for kicking off a big feature. So when we’ve been working on some features like some bigger epics, let’s say, that we’ve been working for Hazier, for example, or some new functionality. This is what we’ve been using.

Robert [00:29:41]: One lever lower, you have situation when your product owner, product manager, is gathering requirements and you are working on them together.

Robert [00:29:51]: In case of doubt, the product owner can go and ask some stakeholders if it’s problematic or not. It works nicely for some stories because usually you can wait for a while and you’re planning it in advance, so it’s not an issue that it can be a bit delayed.

Robert [00:30:09]: I would probably not recommend the ‘golden cages’ approach when basically you are receiving some specification or some task from somebody else and you are just just writing this without really affecting how it’s implemented because it’s very ineffective.

Miłosz [00:30:24]: So we said that it’s better from product perspective.

Miłosz [00:30:29]: Work directly with domain experts and so on. But why should developers care? Because, for many developers, they are fine being just an engineer and waiting for requirements. Is there anything for them in this scenario?

Miłosz [00:30:48]: What do you think?

Robert [00:30:50]: So, I think that there are two things that you should keep in mind. And, as you said, so… Many developers, most, hard to say, but many developers don’t care that much about, let’s say, component success. They are just here working to code and they don’t care that much. But I think it’s not that simple because… You could be such this kind of developer in the market when everybody could get a job easily. But as we already said, it’s not like that anymore. And you need to be a bit more outstanding to get job easily. And to understand job market now, it’s probably important to understand about what companies… care when they’re hiring developers. And well, they care about hiring people that will be useful for them. Let’s put it in this way.

Miłosz [00:31:44]: So basically, if you are able to deliver stuff that they need, they’re kind of happy and you need to make help them make more money, basically, so they can share some of it with you. Now you know there’s less funding than 10 years ago, it was very easy to Oh.

Miłosz [00:32:03]: get VCs to invest in whatever idea you have. So you need to actually deliver something more often. And it’s quite a rare skill to work directly with product. And I think for me the funny thing is: We somehow don’t test this on interviews very often. Also, no one teaches this.

Miłosz [00:32:27]: Universities or even your job, you’re mostly taught to write code. I would even say that we are probably more prepared to be those coding monkeys. Yeah, exactly. If you go for a tech interview, you will encounter the slit code exercises or you will be asked to write some algorithm.

Miłosz [00:32:51]: On a piece of paper or on a whiteboard. Perfect for coding monkeys.

Robert [00:32:54]: Those are requirements. Don’t ask questions, just implement them.

Miłosz [00:32:59]: Yeah, the technical details. But then you start… Working and turns out those things are really not that much used. They are not that important. You will spend more time figuring out what to build and what not to build.

Robert [00:33:15]: And often you’re hiring this kind of people that are great in algorithms. And later you are finding out that they are expecting them to give them tasks, requirements, and they can just code that. And obviously there are companies where it works, but in the product companies… We already covered why it’s not working.

Robert [00:33:36]: It’s also pretty hard to change your mindset if you’ve been working for entire career in this way. So this is obviously the important thing to think about. I mean, if you’re working this way.

Robert [00:33:50]: And you will be working this way for your entire… Okay, maybe not your entire life, but for longer time, it’s harder later to switch. And companies are also— now better in picking, if people actually have some skills to be a bit more engaged in the in the way how product is developed.

Miłosz [00:34:13]: It’s harder to hide your incompetence behind some algorithm. and technical stuff. For many product companies, it’s not that important. For the technical skills you have, I ignore it.

Miłosz [00:34:30]: You know, data structures you can implement on a whiteboard. They are not that much relevant. They need to deliver something quickly to the market, and iterate over it, and do it over and over. So, if you can communicate well, you can talk with domain experts, you can figure out what they need to build, and you can deliver it. And that’s someone companies probably want to hire.

Robert [00:34:54]: And even if somebody will manage to be hired, many companies are also pretty open to let some people go if they will not meet some requirements of how they expect that some people will be operating.

Miłosz [00:35:10]: And on the other hand, if you want to be promoted, for example, delivering is the very best. Probably why to do it. So instead of just waiting for requirements and complaining that it doesn’t work, you can try to help. Whoever needs what they need. And in many companies, probably in most companies, this will help you get promoted because they will tell your manager or whoever that you do a great job. It’s really quite simple.

Miłosz [00:35:45]: I think we can say it works.

Robert [00:35:49]: And we mentioned earlier that nobody is actually teaching us how to do that. So how to work with requirements of product owner.

Robert [00:36:01]: Nobody’s giving us some tools how to do it efficiently. I mean, There are a lot of resources how to build classes, how to do architecture.

Robert [00:36:12]: How to, I don’t know, use vMotion to be super fast in your IDE. And it’s great, but from other sides, I’m not sure if it’s giving that much return on investment. It’s easier to learn, for sure.

Robert [00:36:26]: And you know, it’s coding. It’s—uh— more cool definitely than working with product. But from other sides, I know that many people are also complaining how much time they were spending on meetings.

Robert [00:36:41]: Actually, okay, maybe if you’re spending a lot of time on meetings, maybe it’s worth to spend some time to understand what you can do to do it more efficiently because it’s also a skill and often you may say, ‘Okay, it’s the responsibility of the product owner, I don’t care.’ Yes, but unfortunately it often doesn’t work this way. And in many companies that we worked earlier, if we wouldn’t initialize to do some changes if it would work like it was working earlier, like people will be coding monkeys in case. The default mode. Yeah, yeah. So I think it’s important to get some initiative and to go outside of this comfort cage or comfort zone. Oh, it sounded so coaching.

Robert [00:37:30]: Thank you. Yeah, it is what it is. I mean, sometimes you need to do some things that you are not that comfortable with, and it’s fine.

Miłosz [00:37:37]: Yeah, but I think it’s also important to mention it’s not for… for company’s sake or just because it’s some moral thing to do. It’s basically how you can grow your career this way. Because you become someone who can deliver stuff, and this is what companies want to pay for. Exactly. So I would say do it for your own sake, for your career’s sake. And it will affect your productivity much, much more.

Robert [00:38:08]: I can attest that you can have a bit less of bad meetings that you have. And if you have some tools how to do that. Okay, so we’ve already mentioned a couple of times about that we have some tools, techniques earlier, and probably you are waiting to learn what those tools or techniques are. are. So you can try to use them, experiment with them, and maybe improve a bit how you can do that.

Miłosz [00:38:35]: So, is there any tool you recommend for this?

Robert [00:38:39]: I think I’m not sure if you agree, but I’m quite sure that one of the tools that created most impact in the teams that we’ve been using was brainstorming. When we’ve been working some things that at the beginning was working this default mode close to golden cages, it helped us to… to trigger some discussion between developers and non-developers and to think more how to implement those features.

Miłosz [00:39:11]: Yeah, so maybe if someone doesn’t know event storming, we can… Uh, give a quick TLDR: this is a brainstorming session where you gather developers and domain experts in one room and have them use sticky notes. I know how it sounds—yeah, but basically, the idea is to have this lightweight conversation about complex topics so everyone is on the same page.

Robert [00:39:42]: Even if it sounds, again, maybe I like, again, some coachy, agile, Scrum Master-ish thing.

Robert [00:39:52]: It’s actually a very good technique created by Alberto Brandolini, who is a software engineer. He’s not some agile coach or whatever.

Robert [00:40:02]: software engineer, so it’s very close to implementation. So the artwork that you’re creating is very close to implementation and it can be mapped to the code. Nicely and it gives us some nice framework to discuss some functionalities that we’d like to implement. And we’ve been using it in multiple contexts. So we’ve been using it for this smaller planning for these weekly stories, but we’ve been using it also for some bigger strategic features that we’ve been spending, for example, here. And it’s not about creating a big upfront design. It was rather to have some high-level idea on how everything interacts, how we would like to interact with other teams, to see complexity where it’s there. Because very often… There is one problem with stakeholders. They are going to you with some functionality and they are saying that ‘Why so long?

Miłosz [00:40:57]: Yeah.’

Miłosz [00:40:58]: At least you have a way to have everyone start to discuss a topic in a different way than just sitting and looking at a presentation. And at first everyone is a bit confused, but then very often you have these lively discussions about the product.

Robert [00:41:20]: I can also recommend you to maybe Google event storming so you can see how the artifact at the end looks like. But the idea is that basically you are using notation of domain events.

Robert [00:41:32]: In the simplest kind of session and trying to model how our system works. And it’s nice because it’s creating some visible artifact that’s showing the interactions in the system, but it’s also showing the complexity there. I mean, if you’re working on this artifact together, with product owner or stakeholders, it’s kind of closing all those questions: ‘why is it that complicated because if system is complicated, it’s usually visible in the board— that’s a very visual representation.

Miłosz [00:42:04]: And this is not a technical representation, right? So it’s not about diagrams or database schemas.

Robert [00:42:09]: It’s something that stakeholders can also understand, but it’s also cool because you can also translate it later to your code very directly.

Miłosz [00:42:17]: So very often, if you do it for an existing system that has been around for a while, you will see how developers use a bit different names than other people in the company and start to close this gap.

Miłosz [00:42:33]: It’s very interesting when you ask your colleagues from other teams to map out how it works from their perspective. Hmm-mmm. It’s like you see a bit different system than you are used to. It’s the same thing, but they use a bit different names. They don’t know all the details. And if you do it together, you get a lot of insight from this.

Robert [00:43:01]: And as I said, so for us, it was game changer in multiple things that we’ve been in. And from my perspective, it’s really a shame that it’s not adopted much wider. So, the domain-driven design community is probably best in adaptation of that, so a lot of domain-driven design practitioners are using that by default. I would say that outside of this cycle, it’s not that widely adopted. Again, I think it’s a shame. I’m not sure actually why, but maybe it’s the thing of the forum that for a lot of people it’s like, ‘ah, some agile or coaching stuff, sticky notes again.’ Maybe it’s… because of some cynism. I don’t know.

Miłosz [00:43:43]: You also need someone who facilitates this session so it doesn’t end up with… People discussing stuff in smaller groups and not participating together or just going to some off-topic. So it’s not that easy. Always to run it you have to figure out how to do it a bit, but I think it’s worth it. In a way, it’s probably better than having one person speak and everyone sleep on the chairs in the conference room. After the meeting, you decide to run another meeting because you don’t know what to do next.

Robert [00:44:20]: I think you need to also have some more open-minded people for that. It’s great if you would like to try that, but you’re not always working in teams when you have good climate for running it. I know how hard it sometimes can be.

Robert [00:44:37]: It can be the reason. And in this episode, we’ll not cover into details how to run this kind of sessions, because probably it will be… pretty long. There’s also one very good resource from which we’ve been learning how to do that. This is the ebook from the creator of the technique. It’s named Introducing Event Storming by Alberto Brandolini. We’ll link this in the episode notes.

Robert [00:45:07]: This is what I recommend you to check. But we have some tips how to try to convince your teammates, your product manager and people around to try that. Probably the first tip will be to try it maybe on a smaller scale. So don’t start with this big session of planning for a half year, getting your stakeholders from abroad. And don’t start with booking.

Miłosz [00:45:38]: plane tickets?

Robert [00:45:39]: Yeah, so I would recommend to start with maybe those planning sessions that you are doing with the rest of your team, maybe with product manager. And try to learn how to run those sessions. Because we’ve been running sometimes some bigger session with 20, 30 people. It’s much, much harder than with just your team.

Miłosz [00:45:57]: That’s for sure. Especially if you have someone who’s skeptical of this idea, they will just try to discourage everyone to continue because, oh, we know how everything works already. Why do you do it again?

Miłosz [00:46:14]: I remember sessions like this.

Robert [00:46:16]: But also don’t try to look for excuses like, ‘I’m introvert, I don’t like to do that.’ I’m also, and I did that, and you can just do it. No worries.

Miłosz [00:46:25]: It can be tiring, but there is some end goal in all this. You want to figure out the domain. It’s not just a meeting for the meeting’s sake.

Robert [00:46:36]: It’s tiring, but from other side, I think it’s also very satisfying when it was successful and later you are sitting to implement that and you can later feel how this implementation is just flowing. And it’s also causing a lot of questions and solving a lot of problems that we’ve been mentioning at the beginning of the episode.

Robert [00:46:57]: So, if you would like to start with brainstorming, the first good place to try to adopt that is within your team. So, run some smaller sessions. We recommend in-person sessions since the dynamics are a bit better, but I know that also a lot of people doesn’t have this privilege, let’s say, to work in the office. So, if you’re working in a hybrid way, we would recommend to gather and do this in person because it’s working just much better. Because if you’re doing it on Miro or any other online tool, it’s just harder to maintain everybody’s attention because everybody will try to open some other tab. You can do that, but it’s super hard.

Miłosz [00:47:40]: It also happens in in-person meetings when people come to the session with their laptops. Yes, yes, and they are always very confused when you ask them to close them and stand up.

Miłosz [00:47:53]: The first sessions are always fun.

Robert [00:47:54]: But this is mentioned in the ebook, so we’ll not go super deep into that. And it makes sense. So the first thing you should also discuss it with your team. So you should have some better understanding of what problems you may have with delivering and planning.

Robert [00:48:13]: You can use them to try to convince people that. Let’s try to maybe do it in a different way and try to get buy-in from your team to try that.

Miłosz [00:48:23]: I know it’s often difficult, and we often get asked how to convince my team to start using and driven design. This is a similar idea here.

Miłosz [00:48:35]: I think maybe you can find, if it’s not your entire team, a people like likely-minded as you who want to do something better and try to figure out together how you can improve how you deliver software, basically, and like we’ve been mentioning at the beginning—so, in our podcast, we would like to give you some tools that can help you to be promoted to principal engineer, for example.

Robert [00:49:04]: So getting buy-in of people is a very important skill to be principal engineer.

Robert [00:49:11]: We believe that if you’re a principal engineer, you’re not the person that is forcing people to do something. It’s rather showing people some benefits of your ideas. They will just commit to your ideas and they will be happy because it’s giving some solution for the problems that you have.

Miłosz [00:49:28]: Yeah, so if you want to start, you can just take your team, invite them to a session, try to figure it out together, or you can lead if you feel like it.

Miłosz [00:49:44]: Hopefully, you come at something that’s better than your current way of working and this way they see the value. And similar with your… or product person, product owner, or product manager. You can invite them to the session. They should be probably also eager to find better ways to work for all of you because it’s also there in their interest, unless they are, you know, kind of micromanagement freak.

Robert [00:50:17]: But from other side, this micromanagement is often coming out of something, and at the end of that, if you… go deeper, it’s ending up that they would like to deliver stuff faster.

Miłosz [00:50:28]: If you use this card, it’s nice to yeah and it’s it’s usually quite easy to point out the issues you already have. So you can look back at what happened in your recent plannings, let’s say. You can point that out. But this feature took us two months because of all the issues we didn’t know, of all the gaps in the requirements and so on. Maybe we could do it faster the next time. Maybe we can learn from it.

Robert [00:51:07]: It’s also one important argument that is pretty close to the time. So it’s money. I mean, you can say, ‘Okay, last time we one month, which is this amount of money, because this is the average salary of the developer in our teams, if we could spend maybe this next time on planning it better, we could just save this money at the end.’

Miłosz [00:51:34]: Probably the best way to introduce new concepts is to build. Trust first. So, if you just just join the team and you see how that everything is not done the way you would do it— Probably telling the team on the third day that they should reorganize is not a good idea because you don’t know anything about the team and the company. But if you deliver consistently, you build trust, and then people will be more eager to listen to your ideas. You can gradually introduce new ways of working.

Miłosz [00:52:16]: So I would start with this. One common issue with meetings or sessions is the never-ending discussions.

Miłosz [00:52:26]: They tend to make the meetings hours long.

Robert [00:52:30]: And there are some tips about that in the ebook, but I think it’s also worth mentioning because, even if you will read this ebook, it’s easy to get into some never-ending discussions anyway.

Miłosz [00:52:44]: Especially if you have… If you run this big picture strategic sessions with 10 or more people in the room and you start mapping the entire flow on the board, people tend to run into discussions because you know, for every system you have in your company, there will be someone who says, ‘Oh yeah, this is the issue I need this fixed’ and

Miłosz [00:53:08]: It can take a lot of time and it will derail you from the main topic. So it’s useful to have someone act as facilitator and cut that. the discussions early.

Robert [00:53:20]: I’d probably use better work to maybe park it because, if you cut it, somebody may feel like, ‘why are we cutting that?’ It’s probably better. So, what we’ve been doing usually during the session was using the red sticky note. So, we’ve been writing down the problems that we’d like to park, maybe later during the session, maybe after the session, and put it to the board. So, the person that has this problem will feel that, ‘okay, it’s also something that it’s not disappearing, it’s there.’ We won’t forget about it. Yes, yes. But important. Let’s go back to that. Because we’ve already mentioned the trust earlier. So, if you will be parking things and not going back to that later, you may also have problems with this trust part.

Miłosz [00:54:07]: Yeah, so later we had separate discussions, maybe in smaller teams. Maybe not everyone needs to be involved in all of the red cards. Maybe some of them we will… decide just are not important and forget about them or ignore them. But for most, we need to revisit and discuss this again. But this way, we will focus on one thing at a time.

Robert [00:54:31]: Another thing that I think is also in the ebook, but worth mentioning. So this is the time boxing problem. We’ve already mentioned it earlier, but I think it’s one of the biggest issues. So it’s important to emphasize it and mention it again.

Robert [00:54:48]: Getting some balance between cutting discussions but also not skipping discussions that are important. And it’s hard to give some good algorithm how to park some discussion and when to follow it. It’s probably some skill that you need to develop with time, but, as we said earlier, sometimes cutting some discussion may mean that it will go back to you during the implementation and making, changing some. During implementation, it’s much, much more expensive than during planning because you need to go back to planning. Maybe something that you already implemented is not compatible with that and you need to undo that. It’s important to keep it in mind. Experiment. So maybe sometimes try to discuss a bit longer.

Robert [00:55:39]: Maybe sometimes try to cut, and also set expectations that, usually, when we’ve been doing some planning sessions, some vendor storming sessions, we’ve been like, ’excuse.’ After that, because it’s it’s been sometimes hours to do that, but at the end it was worth— I mean, very rarely— we had like, ‘okay,’ we’ve been spending on even storming too much, and we designed it too well.

Miłosz [00:56:04]: We could better deliver something this time.

Miłosz [00:56:08]: This comes a bit back to the topic of cutting scope. If you plan huge features, it will be very difficult to discuss all of it. Yes, but again, cutting scope is not always easy, depending on the project.

Robert [00:56:24]: And again, it’s mentioned in the e-book, but in the… book, you can read that event storming has multiple phases. So at the beginning, you’re generating a ton of ideas. And with the time you’re removing more and more of the things that are not relevant. And usually, at the end, maybe not at the end, but close to the end of the session, you should do some reality check and to think: ‘Okay, is what we plant here deliverable? In depends if you’re adding some feature or doing something big, maybe a week, in worst case one month.’

Robert [00:57:04]: If it’s one month, probably it will be in practice a bit longer. But if it’s longer, probably you should look at it and see what we can cut out of that.

Robert [00:57:15]: To make it functional, usable, let’s say, and work on this basically. It’s not obvious, but we’re often forgetting about that and we are trying to implement a feature that It should take half a year, and it’s going to be two years later.

Miłosz [00:57:34]: Yeah, because no one challenges it. You start planning. Some feature that comes from someone from another team or your CEO or whoever and no one challenges you.

Miłosz [00:57:47]: You need the entire thing. Maybe you need just 80% of it or 50%, and it will be much simpler to deliver.

Robert [00:57:55]: Okay, so we’ve given you some, let’s say, brief introduction to event storming. Again, it’s not something for episode because I think even one episode will be not enough for that. But if it sounds interesting for you, please leave us a comment. Maybe we’ll do some episodes just about event storming. But. Yeah, I hope that it will give you some overview why it makes sense to try to think out of this thing out of the cage.

Miłosz [00:58:29]: Maybe let’s reiterate some ideas. What can someone do if they see this is an issue, if they want to work better with the product?

Robert [00:58:40]: Yeah, so I think at the beginning, you should assess in what level of cage you are, let’s name it. So, if you are receiving tasks and it’s assigned to you, or you are maybe closer to the way when you’re planning tasks with your product owner but you’re never discussing with stakeholders.

Robert [00:59:05]: Tchum! Think on which level you are. Mm-hmm. Uh, later. Discuss with your teammates if they also see those problems. You should be able to identify the problems with feature planning already. On this episode, so it should give you some weapons.

Miłosz [00:59:24]: Yeah, try to figure it out together. Just not try to force. Your teammates to do some of your ideas right away. Try to think together what are the issues you want to improve.

Miłosz [00:59:41]: Can write down specific problems you have or even situations. So I remember we had a very issues like this— one task has been on hold for one month because of something, I don’t know, some miscommunication or whatever.

Robert [01:00:00]: Or maybe you lost a lot of time on implementing something that used to be useless at the end and your stakeholders said… This is not what we asked about.

Miłosz [01:00:10]: Yeah, we delivered something and then it was scraped because no one needed it. And… probably the more specific it is the better because you can then rise it with your manager or product owner or whoever is in charge Instead of complaining that we never get anything done, you tell them, yeah, this feature could have been one week. but we spent two months.

Robert [01:00:38]: In other words, use the language and arguments that other side care about. You know about what product managers will care. your teammates probably they care about their car so you can also say that okay maybe would like to if would like to get better job later or be promoted, maybe we should try to think about that. And again, if this convincing people to change approach to use your approach. It’s a very important skill of principal engineers.

Robert [01:01:10]: Being principal engineer doesn’t mean that you are forcing people to do something. It means that you are convincing people that your solution will help them with their issues. And that’s it, basically.

Miłosz [01:01:21]: And if it really does, then you get trust.

Miłosz [01:01:26]: You need to accept that some people just won’t care, and it’s also fine. Don’t try to force them to do anything, whether they are developers or not. And I would instead try to find people who care and try to work with them. So if your product owner doesn’t care, doesn’t want to change how they work, maybe to go to their manager and tell them there’s an issue. Basically, find someone who cares about work getting done in this company.

Robert [01:01:58]: If it’s not possible due to political reasons, just maybe do it within your team.

Robert [01:02:05]: We don’t recommend that, but it’s better to do event storming session within your technical team without product owner than not doing it at all.

Miłosz [01:02:16]: There’s also the last advice we didn’t mention in this episode yet, which is change your job.

Miłosz [01:02:22]: It’s always the solution.

Robert [01:02:26]: But again, please don’t use it also as an excuse to not include your product manager, because you would be surprised. I mean, how often they care more than you. You think. They just have a bit different goals, but…

Miłosz [01:02:39]: And they also might have been burned working with developers in the past. Exactly. If they worked with someone who’s just like, yeah, I don’t care. I will code and give me what to do. They might not be eager to work with you directly. So this goes both ways.

Robert [01:02:54]: Yeah, but from our experience, it’s not the case in 96% of cases. It’s very, very rarely a case that it’s… those are people that I don’t care. So please try and you will be really, really, really surprised. Also, I think it’s important to set expectations.

Robert [01:03:16]: There are no magic techniques that will make all the planning accurate because ideas will always change during implementation. We are going closer to planning better.

Robert [01:03:31]: It will also always take some time to do planning. It’s not magic tool. There is no magic tool that will totally remove meetings. It’s just about making it more efficient. Also be prepared that, okay, if you, for example, decide to use event storming. It will be not super successful from the first session, but it’s also a pretty simple technique, so you’ll be surprised that the result will be pretty good anyway. But also set some expectations. Be prepared to experiment and also improve after each session. Maybe do some evaluation after every session to ask people, okay, did you like that? What you didn’t like? And improve, improve, and improve.

Miłosz [01:04:13]: Especially if you change company culture, which is very, very hard to change. And especially if it’s been like this for a long time.

Robert [01:04:22]: But it’s possible. We did it more than once and later people were eager to work in this way. So it was cool. And it was also cool later to transform it to other companies and also have some success stories that helped us too.

Robert [01:04:39]: Be in different places later. Okay, so I think it will be all for today. So if you have any questions about windstorming, about getting out of the cage or any similar topics, let us know in comments on Spotify.

Miłosz [01:04:54]: Yeah, also share your stories maybe. I’m sure there are a lot of stories how projects are run. Let us know in the comments how it works. Works for you.

Robert [01:05:05]: If you have any questions, just ask us. We are answering all the comments, so we’ll be also happy to help you.

Robert [01:05:14]: Would be interested in listening about event storming, for example. Let us know so we can maybe also record entire episode just about event storming.

Miłosz [01:05:23]: Hit that like button and subscribe to the channel so you don’t miss the next episodes. We plan to release a new episode every two weeks now.

Robert [01:05:32]: Yep. So, thank you, Miłosz, for your time. And see you in two weeks.

Miłosz [01:05:35]: Thank you, Robert. See you.

Robert [01:05:36]: Bye.

Durable Background Execution with Go and SQLite

Mon, 20 Oct 2025 00:00:00 +0200

A DNS outage is like a flash flood. It hits you and disappears as if it were never there. “It was probably the DNS” became a meme. But, half the time, we do not know the exact cause. The outage is seldom long enough to diagnose, sequester, replicate, simulate, and test the failure conditions. How can we be certain that this type of failure will not compromise our systems again in the future? The DNS can fail spectacularly in a variety of ways.

A better way to address intermittent failures, such as DNS outages, is to accept them as unavoidable occasional execution disruptions. We might not be able to address each of them exhaustively, but we can design our system to handle almost any interruption of its internal logic. We can harden the system against the disruptions to attain durable execution.

What is Durable Execution?

Durable execution is a guarantee that a program output is correct regardless of failures anywhere in the execution process.

Competitive business applications require durable execution. When it is absent, an employee will spend an hour or two fixing each incorrect output when it is detected. In large business applications, incorrect output often goes undetected for weeks, resulting in expensive chaos around holidays.

Are we talking about bugs? No. Bugs are the program. Some intermittent bugs erode durability, but the durability loss is most often a result of an interruption of the execution process. The interruption can be caused by a network outage, service failure, security breach, or a delinquent update. All of those reasons are common. Durable execution does not care what caused it. It is a promise to recover from almost any failure or combination of failures and deliver the correct output.

Event-Driven Architecture (EDA) is the industry best practice for supporting durable execution. However, most EDA production systems are not provably durable. What happened?

Note

In the second guest post on our blog, we welcome Dima Kotik, who contributed the SQLite Pub/Sub backend to Watermill. Please give Dima a warm welcome! 🎉

Miłosz & Robert

Durable Execution Requires Immediate Input Storage

There is a difference between a guarantee and a wish. A guarantee has some proof, a history of recovery from a variety of system crashes. It begins with emulating crash conditions in integration tests. After writing many such tests, you will realize a simple but profound principle: durable execution begins with writing execution state into storage. There is no other way to proceed, because anything present in random-access memory or ephemeral storage is liable to disappear if the system crashes.

The first step of a durable system is to stash its input before any processing takes place after parsing and validation. Security is more important than durability, so input validation takes precedence.

Stash the Input After Parsing and Validation

Wait, what? I have to commit all the system input immediately into storage? Yes. Is that even possible? Not all at once, but this is not as daunting a task as one might imagine.

If you are using event-driven architecture with an publisher/subscriber management library like Watermill, you are already halfway there. Watermill is one of the most popular event buses in the Go ecosystem.

Watermill is an unusually flexible library in that it can bridge events from a large number of different backends. Your core business logic may stream events to a PostgreSQL database, and your input may initially be stored in a SQLite databases of your edge nodes to achieve durability without sacrificing performance. Watermill can link the two to tango together seamlessly.

Unfortunately, many business applications that utilize Watermill continue to use the Go channel ephemeral backend here and there despite the documentation urging developers not to do it. This backend is a basic in-memory implementation based on Go channels intended for testing purposes. The events are queued into an asynchronous channel. They will disappear when the system crashes. They will clog up the server with a long queue if the execution process slows down for any reason. They will compete for resources with a critical server process.

In other words, do not write input into random-access memory. You are asking for trouble.

There is an easy persistent drop-in replacement for an ephemeral backend. It is not a silver bullet, but it is an easy first step towards durable computing. Try the new Watermill SQLite backend. It does not require setting up any additional infrastructure, and it is more performant and reliable than any other storage method because it does not need a network connection or a custom transport protocol. It writes to a file on a disk. It does not require CGO.

Suppose we are processing book orders, and we would like to store the orders as events in an SQLite database:

db, err := sql.Open("sqlite", "orders.sqlite3?journal_mode=WAL&busy_timeout=1000&cache=shared")
if err != nil {
 return err
}
defer db.Close()

db.SetMaxOpenConns(1) // driver limitation
publisher, err := wmsqlitemodernc.NewPublisher(
 db,
 wmsqlitemodernc.PublisherOptions{
 InitializeSchema: true,
 },
)

msg := message.NewMessage(
 watermill.NewUUID(),
 // serialized order event
 []byte(`{"order_id": "1", "title": "1984", "customer": "Queen Elizabeth II", "address": "Buckingham Palace"}`),
)
if err := publisher.Publish("book_orders", msg); err != nil {
 return err
}

For cloud instances, ensure that you are writing the SQLite file to a persistent volume, which is mounted explicitly. The root file system is typically ephemeral. You may want to eventually migrate the collected input into your main database after thinking about performance implications.

The matching subscriber is not any different from normal Watermill usage other than the backend initialization:

subscriber, err := wmsqlitemodernc.NewSubscriber(
 db,
 wmsqlitemodernc.SubscriberOptions{
 InitializeSchema: true,
 },
)
if err != nil {
 return err
}

messages, err := subscriber.Subscribe(context.Background(), "book_orders")
if err != nil {
 return err
}

for msg := range messages { // consume messages from the channel
 processBookOrder(db, msg)
}

The SQLite publisher and subscriber are drop-in replacements. They can be also used without Watermill router. In other words, you can just use low-level Publish and Subscribe methods.

Full examples are provided in documentation. Nothing else needs to change in your code to achieve this first step towards durable execution: write the input events to persistent storage.

For complete durability, however, the event handlers for each step of processing the book orders must have two provable qualities:

Idempotency: the handler can consume the same event multiple times without generating more than one downstream event.
Atomicity: the handler either completes successfully or fails completely. It must never succeed partially.

The combination of these properties with persistent storage ensures durable execution. When the a step in business logic fails for any reason, it is attempted again from the last safe system state.

Prove Idempotency with Event Duplication

Idempotent event handlers must not change system state when processing identical events. In other words, any repetition of input any number of times makes no difference to the system. We can achieve this effect by de-duplicating events, but it is best that the program logic itself tolerates repetition.

First, ensure that the event message is acknowledged only after the all the business logic has been executed successfully:

func processBookOrder(db *sql.DB, msg *watermill.Message) error {
 // use only one database transaction per event handler
 tx, err := db.BeginTx(context.Background(), &sql.TxOptions{
 Isolation: sql.LevelReadCommitted,
 })
 if err != nil {
 return err
 }
 defer func() {
 if err != nil {
 _ = tx.Rollback()
 msg.Nack()
 } else {
 err = tx.Commit()
 // acknowledge the message ONLY AFTER successful transaction commit
 // what happens if the the execution fails here?
 if err != nil {
 msg.Ack()
 } else {
 msg.Nack()
 }
 }
 }()

 // process input using the business domain logic
 // ...
 return nil
}

What happens if the transaction succeeds but the message acknowledgment fails on line 16? If the handler is idempotent, the event will be reprocessed, but the system state will not change. An easy way to achieve this is to constrain the database table field order_id to be unique. Then, drop the error if it is caused by a unique constraint violation:

// Example for SQLite3 "modernc.org/sqlite/lib"
if err != nil {
 if liteErr, ok := err.(*sqlite.Error); ok {
 code := liteErr.Code()
 if code == lib.SQLITE_CONSTRAINT_PRIMARYKEY || code == lib.SQLITE_CONSTRAINT_UNIQUE {
 // duplicate order id is normal behavior
 // it means that the event was already processed
 // ignore this error
 return nil
 }
 }
 // ...
}

// Example for PostgreSQL's "github.com/lib/pq" driver
if err != nil {
 const UniqueViolationErr = pq.ErrorCode("23505")
 if errors.Is(err, UniqueViolationErr) {
 // duplicate order id is normal behavior
 // it means that the event was already processed
 // ignore this error
 return nil
 }
 // ...
}

What if there are multiple SQL table key constraints? The best solution is to have an idempotency test in place. Then, methodically groom the business logic until it tolerates repetition and passes the test. Use duplicator handler middleware to clone messages and check that the final program state is the same as running the test without the duplicator.

Prove Atomicity with Chaos Engineering

Once we achieve idempotency, how can we prove atomicity for every step of the business logic process?

At-least-once atomic execution is theoretically facilitated by Watermill, but there are subtle ways of subverting it. If you rely on low-level primitives, always acknowledge the message after all the business logic finishes. And, ensure that business logic is granular enough that all of it runs anew in case of an interruption.

A simple rule to follow is that all state changes in one event handler must apply within the same database transaction. If you have two database transactions, you will never achieve durable execution. If you are publishing another event inside the event handler, that event publisher must operate within the same transaction as the event handler. The outbox pattern is essential when making any additional changes to the database together with publishing the event.

How do you prove atomicity with a test? Write a chaos middleware to pair it with the retry middleware for your testing setup:

func ApplyChaos(h message.HandlerFunc) message.HandlerFunc {
 failNextOperation := true

 return func(msg *message.Message) ([]*message.Message, error) {
 result, err := h(msg)
 if err != nil {
 return nil, err
 }

 failNextOperation = !failNextOperation
 if failNextOperation {
 return nil, errors.New("world chaos cancelled the event handler")
 }

 return result, nil
 }
}

This middleware will break the event handlers half the time. The retry middleware should ensure that the operations complete even with half of the system experiencing outages.

How do you prove that an event handler is atomic? Write another middleware that pollutes the message context:

func PolluteContextWithChaos(h message.HandlerFunc) message.HandlerFunc {
 return func(msg *message.Message) ([]*message.Message, error) {
 polluted, cancel := context.WithCancel(msg.Context())
 defer cancel()
 go func() {
 <-time.After(time.Millisecond)
 cancel() // simulate premature cancellation
 }()

 msg.SetContext()
 return h(msg)
 }
}

If all your input and output operations are context-aware (they should be - it is best practice), the premature context cancellation will simulate an execution interruption inside the event handler. This simple example cancels after one millisecond.

Simulate a variety of pseudo-random cancellation patterns. Never use true random, because it will make the tests brittle. Check that the output is the same, and you are well on your way to a durable system.

Conclusion

Business applications need durable execution. Programs that have an innate ability to recover from almost any error or crash are a joy to work with. Durability can be achieved with proper system design and a rigorous testing discipline.

You can see how durable execution can get complicated quickly, but the first step is the easiest. Write the input event to a persistent volume. It will give you some durability right away. You can work out the rest of it later. SQLite Pub/Sub backend for Watermill is a great choice to try this. This new driver works without CGO and, like Watermill, is free and open source.

Cloud providers have begun to offer distributed SQLite clusters for use cases that offer some durability and the fastest possible read speeds. Those projects are a strong indicator that SQLite-powered storage is a viable and increasingly desirable option for durable execution:

LiteFS keeps all the classic SQLite conventions like the single-writer constraint. This gives good reasons to anticipate that in the future durable execution could also become fast even with global transactions.
Turso comfortably runs on edge or mobile devices with poor connectivity.
Cloudflare durable objects are an SQLite persistence companion to Cloudflare Workers.

There are two disadvantages to those commerical offerings compared to Watermill over SQLite Pub/Sub: (1) the vendor lock-in and (2) some difficulties in binding database transactions to event handlers reliably. This means that Watermill might get you farther past persistence towards idempotency and atomicity on the journey to provable durable execution. Give it a try!

SQLite Pub/Sub, Quickstart, and more — Watermill 1.5 Released

Wed, 24 Sep 2025 00:00:00 +0200

It’s been almost a year since the last Watermill release post, and we have a bunch of news to share!

In case you’re new here, Watermill is a Go library for building event-driven applications the easy way. It’s like an HTTP router but for messages. Watermill is a library, not a framework, so there’s no vendor lock-in.

Watermill is now close to 9k GitHub stars and 79 contributors, just on the main repository. It supports 13 Pub/Subs including Kafka, Redis Streams, NATS, Google Cloud Pub/Sub, Amazon SQS/SNS, SQL, and RabbitMQ.

See the documentation for more details.

Let’s look at what’s new.

SQLite Pub/Sub

Watermill now officially supports the SQLite Pub/Sub. The messages are moved between publishers and subscribers using tables in an SQLite database.

A huge thank you to Dima Kotik who contributed the new package: github.com/ThreeDotsLabs/watermill-sqlite. See the documentation.

SQLite Pub/Sub is simple to set up, as you don’t need to run any external services: everything is just a file on disk. It supports persistence, so you don’t lose messages when your application restarts.

So far, we recommended PostgreSQL or MySQL Pub/Sub for when you don’t want to set up a separate message broker. Now it’s even easier to start with SQLite.

It can also be a great alternative to using the GoChannel Pub/Sub in smaller projects.

We’ll soon publish Dima’s article about when it shines. Stay tuned!

The implementation has two packages: modernc.org/sqlite and zombiezen.com/go/sqlite. You can choose either one when creating the Pub/Sub.

Here’s an example of creating a Subscriber with the modernc driver:

import "github.com/ThreeDotsLabs/watermill-sqlite/wmsqlitemodernc"

// ...

subscriber, err := wmsqlitemodernc.NewSubscriber(
 db,
 wmsqlitemodernc.SubscriberOptions{
 InitializeSchema: true,
 Logger: logger,
 },
)

messages, err := subscriber.Subscribe(ctx, "messages")
for msg := range messages {
 fmt.Printf("Received message: %s, payload: %s\n", msg.UUID, string(msg.Payload))

 msg.Ack()
}

And then publishing messages:

import "github.com/ThreeDotsLabs/watermill-sqlite/wmsqlitemodernc"

// ...

publisher, err := wmsqlitemodernc.NewPublisher(
 db,
 wmsqlitemodernc.PublisherOptions{
 InitializeSchema: true,
 Logger: logger,
 },
)

msg := message.NewMessage(watermill.NewUUID(), []byte(`{"message": "Hello, world!"}`))

err := publisher.Publish("messages", msg)

We consider this a beta release: the full test suite passes (and it’s pretty extensive!), but we want to gather more feedback from real-world usage. If you try it, please share your experience. You can use the main repository’s issues or join our Discord.

Watermill Quickstart

We recently launched Watermill Quickstart, a free hands-on training that teaches you how to use Watermill in your project.

We keep looking for the most effective way to learn programming topics. It’s clear that reading docs and books is not enough: you learn best when you actually practice by writing code, failing, and fixing it.

The common advice for newcomers in any technology is to “create a pet project.” But coming up with an idea and sticking with it long enough to learn something isn’t trivial.

The Quickstart guides you to build this kind of project. You code in your own IDE, so the next time you want to use what you learned, you already know the setup and have the experience.

If you try it, let us know what you think!

Watermill Quickstart

Watermill v1.5

While there are no major new features in the main Watermill package, we reviewed a ton of PRs and merged many improvements and fixes. You can see the releases page and the full changeset on GitHub. Thank you to everyone who contributed, and please keep the PRs coming!

Let’s look at some API changes and deprecations.

CQRS Marshaler updates

cqrs.ProtobufMarshaler is now deprecated and replaced with cqrs.ProtoMarshaler. Ideally, we’d remove it completely to get rid of the github.com/gogo/protobuf dependency, but that would be a breaking change. We recommend using ProtoMarshaler, which uses google.golang.org/protobuf instead.

There is now an easier way to decorate marshallers. If you want to add some extra logic when marshaling or unmarshaling commands or events, you usually wrap the marshaler in a custom struct.

Now you can use cqrs.CommandEventMarshalerDecorator to do that with less boilerplate.

cqrsMarshaler := cqrs.CommandEventMarshalerDecorator{
 CommandEventMarshaler: cqrs.JSONMarshaler{},
 DecorateFunc: func(v any, msg *message.Message) error {
 event, ok := v.(Event)
 if !ok {
 return fmt.Errorf("%T does not implement Event and can't be marshaled", v)
 }

 partitionKey := event.PartitionKey()
 if partitionKey == "" {
 return fmt.Errorf("PartitionKey is empty")
 }

 msg.Metadata.Set("partition_key", partitionKey)
 return nil
 },
}

Small API changes

You can now create messages with context using message.NewMessageWithContext.

msg := message.NewMessageWithContext(ctx, watermill.NewUUID(), payload)

This is a trivial addition, but we hope it’ll help make it clear that it’s the message that carries the context. For newcomers, it can be confusing that Publish() takes no context argument. This API is similar to net/http with NewRequestWithContext. A tiny change that should improve the developer experience.

Similarly, we deprecated the Router’s AddNoPublisherHandler in favor of AddConsumerHandler. When we initially designed the Router, we assumed the default behavior of handlers would be to always publish messages. In practice, it’s the opposite: most handlers consume messages and don’t publish anything, or publish the message in other ways. AddNoPublisherHandler sounds a bit awkward, so we decided to rename it.

Context in GoChannel Pub/Sub

There is now an option to propagate the message context to the handler when using the GoChannel Pub/Sub.

This has been something many people have asked for, especially for passing some arbitrary values through the context. You can enable it by setting the PreserveContext field to true.

pubSub := gochannel.NewGoChannel(
 gochannel.Config{
 PreserveContext: true,
 },
 logger,
)

Since GoChannel works in-memory, you can access the context you set when creating the message in the handler.

However, keep in mind this is not how most Pub/Subs work. When using Kafka, Redis, or any other “proper” Pub/Sub, the context is lost when the message is sent over the network.

Use it only if you know what you’re doing! If you switch to another Pub/Sub later, this will break.

Watermill SQL v4

We finally released watermill-sql v4.0.0 after a long time as a release candidate.

You can now use it with packages other than database/sql, like pgx (Postgres driver) or ORMs. This includes support for transactions, which is key for the Outbox pattern. Shoutout to @julesjcraske for contributing these changes!

Subscriber and Publisher constructors now take a Beginner interface argument instead of *sql.DB.

subscriber, err := sql.NewSubscriber(
 sql.BeginnerFromStdSQL(db),
 sql.SubscriberConfig{
 SchemaAdapter: sql.DefaultMySQLSchema{},
 OffsetsAdapter: sql.DefaultMySQLOffsetsAdapter{},
 InitializeSchema: true,
 },
 logger,
)

Migration: Wrap your *sql.DB in sql.BeginnerFromStdSQL(db) when passing it to the watermill-sql API.

Pgx has its own BeginnerFromPgx(db) constructor. When working with transactions, use TxFromStdSQL(tx) or TxFromPgx(tx) instead. If you use another driver or ORM, you can implement your own adapter that implements the Beginner or Tx interface. See the implementation.

The other major change is that we decided to make SchemaAdapter and OffsetsAdapter interfaces more flexible. Instead of raw arguments, they now take a params struct and always return an error. We will stick to this approach for most APIs going forward. While more verbose, it’s easier to extend without breaking changes.

type SchemaAdapter interface {
 InsertQuery(params InsertQueryParams) (Query, error)
 SelectQuery(params SelectQueryParams) (Query, error)
 UnmarshalMessage(params UnmarshalMessageParams) (Row, error)
 SchemaInitializingQueries(params SchemaInitializingQueriesParams) ([]Query, error)
 SubscribeIsolationLevel() sql.IsolationLevel
}

type OffsetsAdapter interface {
 AckMessageQuery(params AckMessageQueryParams) (Query, error)
 ConsumedMessageQuery(params ConsumedMessageQueryParams) (Query, error)
 NextOffsetQuery(params NextOffsetQueryParams) (Query, error)
 SchemaInitializingQueries(params OffsetsSchemaInitializingQueriesParams) ([]Query, error)
 BeforeSubscribingQueries(params BeforeSubscribingQueriesParams) ([]Query, error)
}

Migration: If you use the default schema or offsets adapters, you don’t need to change anything. If you have custom implementations, update the method signatures to match the new ones.

Watermill GoogleCloud v2

The Go Pub/Sub client library v1 will lose support next year. We updated watermill-googlecloud to use the new v2 library.

There’s only one breaking change: SubscriberConfig no longer exposes the SubscriptionConfig field. Instead, it now has the GenerateSubscription function.

The function returns *pubsubpb.Subscription, which mostly maps 1:1 with the old SubscriptionConfig.

Migration: Replace SubscriptionConfig with GenerateSubscription.

 googlecloud.SubscriberConfig{
 ProjectID: "tests",
 GenerateSubscriptionName: subscriptionName,
- SubscriptionConfig: pubsub.SubscriptionConfig{
- RetainAckedMessages: false,
- EnableMessageOrdering: enableMessageOrdering,
- },
+ GenerateSubscription: func(params googlecloud.GenerateSubscriptionParams) *pubsubpb.Subscription {
+ return &pubsubpb.Subscription{
+ RetainAckedMessages: false,
+ EnableMessageOrdering: enableMessageOrdering,
+ }
+ },
 Unmarshaler: unmarshaler,
 },

We considered mapping the config to make this change non-breaking. While it’s possible, we wouldn’t get rid of the old dependency, and we’ve seen reports that the old and new clients don’t work well together. So this means a major version bump, but the migration should be straightforward.

Hall of Fame 🏆

We had 43 unique contributors to all Watermill repositories since the last release post! 💪 Thank you all for your help!

Docs refresh

We updated watermill.io a bit, so it’s easier to tell what the library is about. It should also be easier to choose the proper learning path.

Once again, give the Quickstart a try and let us know how it went.

We consider making free training like this for other open-source libraries as well. If you have any suggestions, please share!

Shipping an AI Agent that Lies to Production: Lessons Learned

Thu, 07 Aug 2025 00:00:00 +0200

The peak of hype isn’t the best moment to reflect on AI. Will it take your job, or is it the next fad like NFTs? Are AI startups ridiculously overvalued, or are the companies that sleep on AI doomed?

Time will tell. LLMs are far from perfect, but I’m excited they’re here anyway. Not because of a silly promise to make me 10x more productive, but because they can solve some problems that were previously unsolvable. If you enjoy building software like me and Robert, it’s a new tool worth trying. And there’s a chance AI experience helps you create a better product or land a good job.

Did you ever get stuck debugging your project at 2 AM, and went to sleep frustrated? We spent a few weeks adding an AI Mentor to our learning platform to help our students avoid this.

As always, it wasn’t as easy as most tutorials show. Trivial use cases, like a weather-checking assistant, are fun to make, but aren’t much more useful than the default chat interface of any model.

I’ve heard before that implementing an AI feature is easy, but making it work correctly and reliably is the hard part. You can quickly build an impressive demo, but it’ll be far from production grade.

We can now confirm this is true. Plus, we’ve seen something not talked about that much: how an AI feature can fail in production and mislead users.

I want to share what we learned in detail, hopefully saving you time on research. Let’s set aside the world-changing stories for a moment and see what we can build with this stuff.

Note

If you’re just starting out

I know many developers stay away from AI because they consider it overhyped, unethical, or useless. They also feel massive FOMO because there’s currently no other topic in tech.

If you feel this way, I’m not trying to change your mind.

I encourage you to approach it like an engineer trying out a new technology. Look for what problems it can solve, while most people are busy applying AI to everything.

Once you use it in practice, you’ll have a good idea where it’s useful and where it falls short. You’ll make up your mind much better this way than reading hot takes from CEOs and tech bros.

Why AI Mentor?

We’re not VC-backed and don’t seek to sell our company, so there’s little value in slapping “AI-First” on our homepage. We wanted to use LLMs to solve a real problem our users had.

Our Academy platform is all about learning by building real-life projects. One of our challenges is helping students finish the training. If you have ever tried an online course, you may know this problem — you put it off after a few days and never return, distracted by something else.

One reason to put learning off is that you get stuck. While working on complete projects, it can be difficult to spot a bug. Example solutions don’t always help, since each project is unique. We help students move forward by reviewing their solutions.

Often, it’s a small typo or mistake that’s easy to catch with fresh eyes. There’s no learning value in fighting with such bugs for hours. Frustrated and stuck, some people give up.

Last year, we thought LLMs could help. You can feed them entire codebases and ask them to point out or even fix errors. And they are always available, in contrast to our limited time for customer support.

Back then, the models were expensive, far less impressive (GPT-4 had just been released), and there were few sources to learn from. But most of all, we lacked time and postponed the project. We came back to it earlier this year.

(If you want to see it live, the first few modules of Go in One Evening require no login.)

Milestones

We decided to start with an MVP and focus on the fundamentals, so we’re not distracted by nice-to-haves. We picked a few milestones and planned to implement each one in production before moving on to the next one.

A single “Help me!” button on the website for Go in One Evening. Users would receive a single message with a hint on what’s wrong with their code.
Conversations. While a chatbot wasn’t our main goal, asking a follow-up question about the solution could be helpful.
Enable the Mentor for Go Event-Driven.

Go in One Evening is a simpler training with shorter exercises. We assumed solving the issues found there would be easier compared to Go Event-Driven, where students build more complex projects. (The final project is about 4k lines of Go code.)

The mockup of our initial idea.

The end result.

The “Help Me!” Button

We created a proof of concept that should be easy to extend.

We started with two HTTP endpoints:

a POST for requesting help (for now with a constant “Help me” message)
a GET streaming all messages over Server-Sent Events (SSE).

In theory, a single endpoint could do both, but we wanted it to be asynchronous by default. Users can click the button and refresh the page without losing the response. Separating writes from reads is a good practice, especially for long-running tasks.

The downside is a bit more complex setup, but we use proven tools that we know well. We prefer event-driven patterns for asynchronous architecture. We use Watermill and Google Cloud Pub/Sub to trigger the Mentor and send updates back to the user.

All of our events end up in BigQuery, so we get storage for conversations out of the box. This is important because we need the history of messages to include in the next prompts.

High-level overview.

This seemingly simple setup already has some challenges, even without considering the AI part:

How to handle race conditions between the frontend app, the Pub/Sub, and the SSE handler?
How to handle errors and retries?
How to deal with BigQuery’s insert and select latency?
How to approach message ordering?

Events

Event-driven patterns turned out to be a good fit for working with LLMs. Behind the scenes, there are many moving parts, and most need no input from the user. Failures and retries often happen, but they are all hidden behind a single response.

It’s why we don’t recommend using a single HTTP handler for the whole process. There are delays and interruptions, and you need to handle them, so your users don’t see the mess.

Most LLM APIs support streaming chunks of messages. You can show them on the UI before the model has finished generating the response. SSE is a perfect fit for this.

The first response of the SSE handler is a list of all messages (for the given user and exercise). Then, we stream all chunks over the Pub/Sub to the SSE handler and then to the browser.

We had a great experience working with SSE before, and watermill-http supports it out of the box, so it integrates nicely with our architecture. We’ve improved the implementation by adding support for generics (although it’s not public yet, we may release it at some point).

It’s all hidden behind a high-level interface. The HTTP handler receives a channel of incoming events and a channel to send responses back to the client.

func (h *Handler) HandleEventStream(
 r *http.Request,
 events <-chan *events.MentorMessageSent,
 responses chan<- any,
) {
 // Send the initial response with all messages
 messages, _ := h.getAllMessages(r)
 for _, msg := range messages {
 responses <- msg
 }

 // Process incoming events
 for event := range events {
 // ...

 responses <- response{}
 }
}

Note

If you want to learn more about Server-Sent Events, we have a dedicated post about implementing it in depth: Live website updates with Go, SSE, and htmx.

Calling LLMs

We initially used langchaingo for calling the LLMs, but the API felt a bit bloated for our needs. We created a small wrapper on top of the official SDKs to be able to switch the models quickly.

The design of most LLM APIs is very similar. You can describe a prompt call like this:

type Model interface {
 Prompt(history []Message) (Message, error)
}

It’s a function that accepts a chat history and returns the next message in the chat.

In practice, we ended up with a few more details:

func (l *LLM) Prompt(
 ctx context.Context,
 history []Message,
 user User,
 opts PromptOpts,
) (*Output, error) {
 // ...
}

type PromptOpts struct {
 MaxOutputTokens int
 Model ModelName
 Tools []ToolSchema
}

func (o *Output) Listen() (<-chan Chunk, error) {
 // ...
}

func (o *Output) WaitAndGetAll(ctx context.Context) (string, error) {
 // ...
}

This is a tiny abstraction over the LLM API, just enough to make it easy to switch between models.

Similarly to SSE, Go channels are the perfect API to stream chunks. It works well with the whole asynchronous model, and it’s great to have support for it on the language level.

Prompts & Context Engineering

We spent a lot of time iterating over the prompts and the context we fed to the model.

For example, we wanted the Mentor to hint at the right direction, but not simply give the complete answer. We ended up with quite a long prompt for something as simple as “Help with this exercise”.

We have many files related to each exercise (the user’s solution, an example solution, and many others). Even though models now easily handle long contexts, we still need to be careful about how much we feed them.

Deciding what context to provide needs careful balance. If you dump too much and the context window grows, the prompts get more expensive, and the models are less likely to follow the instructions. If you provide too little, the model may not understand the problem and give a useless answer.

There are really no rules that work for every case. One thing we found helpful is strong language in the prompt. The models seem to follow wording like “ALWAYS” or “NEVER” better than “should” or “shouldn’t”.

Free-form chat

After the initial response, users can ask about the solution or other things related to the exercise.

This part was quite straightforward since we had already made the MVP ready to work with threads of messages. The key here is getting the historical messages to include in the future prompts.

In our case, we fetch the chat history from BigQuery. We retrieve what we can concurrently, so the time to the first response chunk is short.

Solving Complex Projects

This approach worked pretty well, even though we used a single prompt. For shorter code snippets, it could guide students on what’s wrong with the solution.

But for complex exercises, it failed miserably. With big projects, the model often made mistakes. It tried to fix issues that didn’t exist or completely missed what was wrong.

We improved it a bit with better prompts and more context about common concepts. But for the most complex examples, it wasn’t enough. So we experimented with a more advanced approach, which you could consider an agentic system.

Fixing the Solution

The tricky part of generating hints is that we can’t be sure whether the model is right.

It’s how your AI feature can (and likely will) fail in production. To counter this, you need to be able to verify the model’s output.

In our case, we built a system similar to coding agents like Claude Code. We use it to fix the user’s solution before it gives any hints. If it succeeds, we have some confidence that the model is on the right track.

This approach solved the issue of the model making wrong assumptions. If it tried one approach, and it didn’t work, it would realize the issue is elsewhere.

Among other things, the agent can edit the user’s submitted files. These changes happen only in the Mentor’s context — the user doesn’t see them. It also plans what to do, can summarize the changes, and a few other tricks. Different models excel at different tasks, so we can use the best one for each step.

In the code, an agent is pretty much a for loop plus tool uses. It’s a bit like a regular chatbot, except the model “talks” to itself. You keep the history and add messages with feedback on the environment. The context grows with each iteration, so it makes sense to limit the attempts.

Parallel Runs

The new approach sometimes brilliantly succeeded. But other times it was stuck, failing to edit a file, or just not getting the original issue.

The good part was that it could now tell the user it didn’t know the solution, rather than making things up. That’s a huge improvement — if someone is already stuck, we don’t want to mislead them further. But it wasn’t able to fix the solution on each attempt. You had to retry a few times if it failed.

We noticed that if the model didn’t succeed in the first few attempts, it would only get worse. It made more sense to start over.

So we started running a few agents in parallel, each trying to fix the solution. Once one succeeds, we stop the rest.

Managing concurrent code like this is a pretty common pattern in Go. We feel it’s a great fit for AI engineering thanks to its concurrency model (compared to other languages, like Python).

wg := sync.WaitGroup{}
wg.Add(workers)

for i := 0; i < workers; i++ {
 go func() {
 defer wg.Done()

 for cfg := range configsChan {
 ctx, cancel := context.WithTimeout(ctx, workerTimeout)
 defer cancel()

 select {
 case <-ctx.Done():
 return
 default:
 }

 result <- Solve(ctx, mentorContext)
 }
 }()
}

go func() {
 wg.Wait()
 close(result)
}()

for {
 select {
 case <-ctx.Done():
 return nil, ctx.Err()
 case r, ok := <-result:
 if !ok {
 return nil, ErrCantHelp
 }

 if r != nil {
 return r, nil
 }
 }
}

We use many combinations of models to increase the chances, hence the configsChan. Different models excel at different tasks, so we can use the best one for each step. Also, if a cheaper model manages to finish the task fast, we can save some money on the expensive ones.

Often, a single call couldn’t reliably figure out the solution. This shotgun approach improved the chances. The tradeoff is cost: we pay for all started parallel runs, even if we finish them early.

Don't miss new posts.
Join over 18k subscribers of our newsletter and get a free e-book!

Go With The Domain Three Dots Labs

🔒 We do not send spam. You can unsubscribe at any time!

QA: Tests & Evals

In many projects, testing is an afterthought. It turns out it’s crucial when working with LLMs. And testing AI products can be a nightmare.

It doesn’t seem that different at first. Calling a model is similar to using any other API, so you need to prepare test cases with inputs along with the expected outputs. But there are some caveats.

Challenge #1 is judging the responses. An optimistic scenario is when you make the model return a structured output, with a well-known list of valid values. Good old unit tests can work well if you have enough test cases.

But if you let the model respond in free-form text, like in most chatbots, it quickly gets complex. You need an evals system where you judge if the output is correct. You could either use a human to do this or another LLM call.

Whatever you choose, expect to read lots of data. You need a realistic set of inputs and outputs, and you need to review and label them.

Once we kicked off the project, we spent a few days manually going through possible test cases based on production data. It seemed unproductive at first, but if we skipped this part, we would have no idea if what we built works. It also helped us understand the problems people actually face.

Challenge #2 is time. If you want to use the smartest or thinking models, a request can easily take tens of seconds. But even with faster models, running a few prompts in a sequence is far from unit test speed. This slow feedback loop is annoying to work with, especially if you tweak the prompt slightly and want to see the impact right away.

Challenge #3 is cost. The best models are also the most expensive. If your context grows (some of our evals have context with 100k+ tokens), each test run costs real money.

Because of #2 and #3, forget about running a big set of evals in CI on every commit (unless you have too much VC money to burn). You can run some of them, or do it only once in a while. In our case, the trivial test cases (fast and cheap) are not that interesting, because even the less capable models solve them easily. The longest and the most expensive ones are what we care about the most.

In regular tests, you can mock the expensive or slow resources. It makes little sense here, as you need to validate the same models you’ll run in production.

Since we run parallel workers for each mentor call, it also impacts the evals. While not a big difference in the production environment, it makes evals much more expensive. It was also easy to hit the API limits.

So, in the evals, we run the workers sequentially, starting with the cheapest ones. It slows down the tests, but saves us some money (this makes a difference, especially for bigger contexts).

On top of all this, LLMs are unpredictable, so you may want to run a few iterations of each eval for more realistic results. This multiplies both time and cost.

You need to balance many factors.

We didn’t use any framework, but a script written in Go that runs the evals and collects the results.

Because evals take a long time to run and could be interrupted by some error, we implemented a cache and retry mechanism, similar to what go test does. It’s frustrating to spend five minutes waiting for tests, only to realize you forgot a simple change and have to start over.

A few of the evals we run for the Mentor.

Failing in production

After rolling out the first version of the Mentor, we monitored the responses to see how it did. Shortly after, came this:

Mentor hallucinating. This is wrong!

I started sweating when I saw this. People pay us to teach them, so it’s a huge risk if they learn obviously wrong things.

In this case, the student tried to understand if assigning a slice to another slice in Go results in a deep copy. The model happily agreed, even though it’s not true.

We take teaching people seriously. I almost wanted to scrape the entire feature when this happened. We contacted the student right away to clarify the issue. Thankfully, they appreciated the detailed explanation and weren’t angry.

To counter this, we switched to a better model for conversations and added more context to the prompts (details below).

The worst part about hallucinations is that the model is very confident about being right. When you just start working with LLMs, you’re surprised how often it happens. You need to be ready for it, because it WILL happen in production.

This is why I’m not comfortable learning something in-depth just from an LLM. If it’s something you can quickly check, it’s not a big deal. But learning fundamentals the wrong way is another story.

RAG and Sources

Initially, we assumed we didn’t need to use RAG (Retrieval-Augmented Generation), and we simply always added some documents to the context.

But after the first issues, we decided to implement a simple RAG with things like Go documentation, our blog posts, and other resources. We have a simple database of content in Markdown format.

The RAG hype presented it as some mindblowing technology, but it was rather simple to implement. It took us just two days to have an MVP in production. It helped that we could split the work and implement two parts in parallel: building the database and querying it in the mentor code.

As a bonus, we also link to the sources in the response, so students can read more about the topic.

The list of useful links.

Predictability

If you expect a structured output from the model, like “true” or “false”, or a set of answers, testing is much easier. But dealing with free-form answers, like with most chatbots, is entirely different.

One day, I started tweaking the temperature setting, assuming lower values should work well for our needs. Then, I realized there’s no way to tell unless I compare multiple responses manually or write an eval that focuses on this difference. And even then, I’m not sure if the reason was my change.

It’s very difficult to get predictable results from LLMs. Small changes in the prompt have effects on the whole pipeline. The more complex the system, the less predictable the outcome, and the harder it is to test.

After you change the prompt, you need to run the evals. But probably not all, since they are slow and expensive. But then, if you test only a few, you don’t know if it covers all cases.

If the response changes, you can’t be sure it’s due to your changes. Maybe it’s just a different random response. Or maybe the model has been updated overnight (you have no visibility into it). To have a better idea, you could run more iterations of each eval, but this increases both time and costs.

The developer experience of changing the prompt and waiting for results is quite poor. It’s nothing new: it’s like working with a slow or flaky CI/CD pipeline, where you commit changes and can’t spend the next five minutes on anything productive. The context switch is painful. You can counter this with parallel runs, but you can hit usage limits this way, and it’s then easy to burn money within a few seconds.

How you model your evals is key. We ended up with two kinds of evals:

Strict evals, where we expect a given percentage of scenarios to pass. We can use them reliably in the CI pipelines.
Graded evals, where we run a set of scenarios and manually compare how the results changed according to grading.

Shifting the Mental Model

How do you cope with software that doesn’t follow instructions?

Often, you like the LLM’s output 90% of the time, but the remaining answers are confusing or outright wrong. This can be frustrating if you’re used to working with code.

There are many ways to reduce hallucinations, but there’s no way to avoid them completely. Realizing this helped me change how I think of LLMs.

I see people go to two extremes:

Treating LLMs as sentient beings, trusting the model to do exactly what they ask, if the prompt is good enough: “Bro, I found a way to make GPT never lie to me, just use this prompt.”
Skeptical about using LLMs entirely, making fun of prompt engineers having to beg the model to do what they want.

A middle ground I found is to treat an LLM as a function that is excellent at parsing vague inputs, and the output is often brilliant, but sometimes nonsense. I treat prompts as code, but with less strict rules than I was used to.

With this mindset, it’s easier to be pragmatic and remember not to blindly trust the output. But also not getting discouraged after something breaks.

✅ Tactic: Expect LLMs to fail

LLMs are impressive but unpredictable. Don’t treat them like the code you’re used to.

Be ready for failures and hallucinations. If you can, verify the output.

Agentic systems or autonomous agents?

The top 2025 trend is running autonomous agents that make decisions independently.

But thinking of an agent as a sentient being that can figure out what to do in every scenario is a trap. Code always works in contrast to an unpredictable LLM.

We didn’t need to go all in on the agent running all workflows by itself. For example, we can just decide (in code) what steps the coding agent should take next.

When writing regular code, you need to control the agent loop more directly. It’s tempting to think you can just include in the prompt what the agent should do in every scenario. But the reality is, you have no guarantee it will follow your instructions exactly.

Our agent isn’t super complex, but it doesn’t need to be. If you can predict when to hand over to regular code or another LLM call, just do it in the code. No need for an agent framework where agents know about each other and can decide who to “talk” to.

Especially if you can’t verify the output, how can you trust the agent to make the right decisions?

This agentic hype feels like the early days of microservices when we told ourselves the story of independent entities talking to each other. Then everyone jumped to implement the solution, without considering whether it actually solved anything.

Many teams learned the hard way that distributed systems are difficult to get right. Even with classic, perfectly predictable code. I’m not eager to add unpredictable models to the mix.

✅ Tactic: Use LLMs where they shine

If you can handle the problem with regular code, there’s no need to use LLMs.

Don’t think of agents as sentient beings that can figure everything out by themselves.

Where is the complexity?

The length of this post shows how much goes into one AI feature, and not all of this effort is related to LLMs. Most of it is good old software engineering and product design.

I will risk saying that building the Mentor was 80% regular software development and 20% AI-specific parts. And 80% of the latter was creating and running evals.

Most of the complexity is in the orchestration. All well-known patterns and good practices still apply. This agent-orchestrating code is your domain logic.

Once again, Go shines for such use cases. It’s trivial to run concurrent workflows, and channels are a great way to communicate between different parts of the system. The logic is easy to follow. (And Go is easy to learn!)

Models

We accidentally picked a hot moment in model releases.

We started with Gemini 2.0 Flash — it had a 1 million token context window, a generous free tier, and was relatively cheap overall. Then, Gemini 2.5 Pro came and made Flash look silly. It had a much better understanding of code. Then GPT-4.1-mini came and made Pro look expensive and slow.

In retrospect, it’s good we didn’t implement this feature one year ago, since the models got so much better over time.

Picking the right model is a balance between speed, cost, and quality. You need to be able to switch models easily for tests, so make sure your system is not tightly coupled to a specific model.

We eventually landed on OpenRouter. You can use the Go OpenAI SDK to call any provider’s model, which is great by itself. It also has no strict rate limits (we hit limits running evals on OpenAI). Finally, it centralizes billing and lets you set limits and budgets (something unheard of for Gemini — crazy!). The downside is some extra cost and latency.

Limits & Costs

One feature of our internal LLM library is limits.

We designed it so that calling an LLM is not possible without using the limits. Even tests track limits in-memory.

It’s too easy to lose track of how many tokens you use. One day, we spent $100 running evals. After that, we made the cost clearly visible in all of our tooling, including tests running in the CI.

A cost spike because of running evals.

Gemini is an especially awful experience. There’s no way to set budget limits, and the usage shows up with a delay. Thankfully, other providers offer basic services like budgets or prepaid payments. As mentioned above, OpenRouter also works great for this.

We can set limits on a few different levels. The primary one is per user. It’s not a perfect solution, since more complex projects can quickly exceed the limit. It’s something we will try to improve in the future.

The costs of running in production so far are just a tiny fraction of our infrastructure costs. The surprisingly expensive part has been running the evals. The faster you want them to run, the easier it is to burn money quickly. Thankfully, this happens only once in a while when we change the prompts or add new evals.

Observability & Tooling

As in any complex system, debugging AI features is difficult. Good tooling helps.

Tracing is a great way to understand how the system works. We didn’t use anything AI-specific for tracing, as we like simple tech stacks and proven technology. We already use tracing in our project, so it was just a matter of adding spans and propagating them.

A cool thing about tracing is how it shows which parts run concurrently.

We also created a back-office dashboard to track the cost and usage, and a set of CLI tools to easily manage evals.

Moderator

We wanted the Mentor to avoid unrelated topics, as it was more likely to give incorrect answers then. We set some guardrails in the prompts, but it was easy to work around them.

Our second approach was a separate moderator prompt.

It has two functions: filter out irrelevant topics and decide if the user needs help with the exercise or just asks a question. If it’s just a question, we don’t need to fix their solution, so it’s a much simpler and faster prompt.

It’s a classic example of how asking the model to do too much at once is problematic. Similar to classic software, the separation of concerns is a good idea. One prompt should focus on one task.

Moderator stopping the conversation early.

Encouraging students to ask for help

Implementing the Mentor was one thing, but it wouldn’t be effective if students didn’t know it existed.

People are already fed up with companies shoving AI features their way, and since most chatbots are dumb, it’s not tempting to click the button.

We track the failures and show small hints in the CLI. Plus, we use the same SSE endpoint to stream hints and show an indicator on the website.

Sometimes, students still ask us directly for help. We don’t want to point them straight to the Mentor, as our support is part of the product. Now, we can use the Mentor ourselves to find what the issue is, and then we just need to explain it to the student. Finding a small typo is much easier this way.

The UI

We’re not frontend experts, so I will shortly mention the UI parts for completeness.

We use React and have vibe-coded most of the Mentor’s frontend code. We’ve been working with frontend for many years, but it’s not something we enjoy. Generating the code was a great way to start: we understand enough to know what we want, but we don’t have to write it ourselves.

We find AI-assisted coding works well for software that has no “strict behavior”, and you can quickly tell if it works, like UI and video games. For example, if the generated CSS is silly, but looks fine, it’s not a big deal. Especially if it’s just a single component. Of course, if you go too far with it, it can turn your project into a mess.

We use a polyfill for SSE. It supports features not in the official spec, like passing custom headers in the API (Authentication), and it works out of the box.

One thing to consider with a chat interface is that you need to animate the chunks when streaming (delay the letters slightly). If you make them appear as they come, it doesn’t look as good.

At some point, we added status messages to show that the Mentor is working on the response. Sometimes it takes a few minutes, so it’s easy to lose patience and think it stopped working. These messages are not fake — we stream them from the backend via the same SSE endpoint.

Outcomes

We ended up with something good enough to make it public and observe what happens. We see people use it, and most of the time, the response does the job.

What we haven’t tried yet:

Running an open-source model on our own.
Fine-tuning models.
Judging the responses by another LLM on the fly.
Pre-computing hints before someone asks for help.
Partially reading files instead of dumping the whole codebase to the context.

Even without these, we feel the result is pretty impressive compared to most chatbots.

The key lessons learned for us (hand-crafted by a human so you don’t need to paste this post into an LLM):

LLMs are not magic, but they can solve some problems that were previously unsolvable.
Building an AI feature is easy, but making it work reliably is a whole different story.
The complexity is in the orchestration, and the old software engineering practices still apply.
You don’t need a framework to get started. A tiny wrapper around the LLM API is enough.

You need to watch out for:

The hype stories don’t mention the full picture. Try things yourself!
LLMs WILL hallucinate in production, be ready to handle it.
Testing AI systems (evals) is difficult to get right and can be expensive.
It’s easy to go over budget. Implement limits early.

If you want to see the Mentor live, you can try the free part of Go in One Evening (no login required).

AMA #1: Clean Architecture, Learning, Event-Driven, Go

Wed, 02 Jul 2025 16:00:00 +0000

Quick takeaways

Go in the AI era: Go is excellent for AI applications, as the built-in concurrency makes orchestrating parallel API calls much easier than languages like Python
Architecture philosophy: Clean Architecture isn’t always necessary. Start simple and add layers only when you feel the pain of complexity, not because someone said you should
Career transitions: Switching roles within tech is easier internally. Moving from sysadmin to developer works better within the same company where people already trust you
Go design patterns: Small interfaces near usage is the Go way. Duck typing allows you to define interfaces where they’re used rather than in separate layers
Distributed systems: Async communication often solves sync timeout issues. When dealing with chains of service calls, consider using messages instead of increasing timeouts

Introduction

In this special 10th episode, we answer community questions in our first AMA format before taking a summer break.

We discuss Go’s role in AI development, Clean Architecture implementation, career transitions in tech, and distributed system timeouts.

After the break, we’ll switch to pre-recorded episodes with improved production quality. We still plan running some live episodes like this one, so stay tuned!

Show Notes

The new Domain-Driven Design in Go training we announced
Team Topologies book mentioned for organizational architecture
Join our Discord community for more technical discussions — we’re happy to help!

Quotes

Architecture sounds like something complex that architects do. But it’s basically stuff like “if you have 10 services connected with synchronous calls, what happens with timeouts?”

Miłosz

Five days of coding can save one day of planning. And it’s often the case. And even if it’s not visible on the first sight, it’s often visible during review that is going for ages.

Robert

I really like this idea of fixing something once you feel the pain. If something is not painful maybe it’s fine even if it’s not the best. Maybe it could be refactored, maybe it could be split to modules or whatever but if it’s not painful you probably have more stuff in the backlog to take care of.

Miłosz

In many cases we hired people that didn’t write a line of code in Go earlier. It wasn’t an issue in the end, because those people were productive very, very quickly.

Robert

Often people are like, I am just a developer, give me the requirements and I will transfer them into code. So yeah that’s not really helpful for most companies, because they need people who can understand the business domain, can help people figure out what the product needs.

Miłosz

Go is also a great language to be generated, thanks to the syntax that is simple, But it’s also very strict. So, for example, if you have some agent that is generating your code, it’s easy to verify that.

Robert

Timestamps

Transcript

Robert [0:00]: All right, so welcome in our first AMA episode.

Miłosz [0:04]: Hello everyone.

Robert [0:05]: Last episode before short summer break. So we have a couple of questions to go over today. And just as a reminder, so if you have any extra questions, just feel free to leave it on the chat. We’ll also be happy to answer that.

Miłosz [0:21]: This episode is a bit special because this is our 10th one and we will end the live format this way.

Robert [0:29]: Or at least partially. I mean, I think if you like this AMA format, there is a chance that we may have more episodes like that. So we’ll ask real questions and we’ll run it live. But in general, it will be no longer a live podcast, it will be a podcast. So we’ll basically pre-record most of the episodes and you’ll be able still to watch them on YouTube, so it will not change. And yeah i think it should improve the sound quality the video quality because well it’s not the simplest way to make it live and to make it sound great because it’s live so it cannot i mean it’s easier to tweak sound after recording obviously it’s a bit harder during the recording the second thing is obviously video uh the third thing is that a bit less obvious that is that we We don’t have our own room to record, so sometimes we need to improvise a bit to record in different places. Like today, so when you’re longer with us, you might notice that we’re in a bit different background. This real background, so…

Miłosz [1:39]: I hope the next episode should be much more professional,

Robert [1:43]: Let’s say. But you will see. It’s a surprise after vacation, so after we’ll be back, I guess, in September.

Miłosz [1:50]: Probably, yeah.

Robert [1:51]: Yeah, yeah. So this is the thing. So yeah. Improve and yeah it will be also a bit i think um content quality should be also higher because when we are recording it live you know it’s also a bit challenging to go with the flow and try to avoid a lot of pauses and try to maintain the flow obviously when it’s not live it’s much much easier because we can pause we can just cut it later we can think about topics so but yeah i think it was It’s a nice experiment to try with the live format and, again, it’s not like we’re dropping that. It’s nice that we have already working setup for that. We have the space, but it’s just a bit changing of the bars.

Miłosz [2:33]: We also received a lot of feedback. It would be useful to have some diagrams, some visual aid to what we are discussing. So maybe we’ll also try something like this in the future. So it’s not, you know, so you have a video with some helpers or something more interesting like coding, not only talking.

Robert [2:53]: Yeah, but no worries. So if you are listening for that in the podcast applications, also we don’t want to make it hard to listen, just listen. So, you know, sound is always the highest priority here. What else? Yes, I think that’s all about the format and I think we should be able to go to the questions.

Robert [3:16]: Or maybe you have something to add before.

Miłosz [3:18]: Yeah, if you have any questions, leave them in the chat so we can pick them up.

Robert [3:29]: All right so questions so the first question is what do you think the future holds golang developers in ai forward world so i think we can have i actually have pretty a lot to think to say about that because we have the mentor functionality in our online trainings and from what we see how people are using that it’s super useful and especially compared to some other ai solutions because i think it will be not only my experience that very often when you see some ai functionality it’s just unpolished i mean you can have impression that somebody just get the api of open api of open ai or cloud or whatever just put some empty simple prompt and that’s it or like for example if you have iphone i mean it’s not the best implementation in the world with everything and the AI implementation.

Miłosz [4:24]: As a user, it’s hard to get right. You can guarantee the answers.

Robert [4:31]: I think it’s also easy to have something that works, but it’s much, much harder to have something that helps. But I think it’s a bit off topic. The question is more about Golang developers in the AI of our world. So what do you think about how Golang fits with this AI world?

Miłosz [4:50]: I would expect most developers to use APIs, just call the RLMs over an HTTP API. And something that probably everyone does in Go anyway. So technically it’s not that different. And if you write more agentic systems, like we did with Mentor, The hard part probably is the orchestration of it all. So there’s a lot of loops and parallel function calls. And I think Go is actually great here. We’ve seen it work well when we designed the mentor feature, because goroutines give you very easy-to-use tools to create parallel requests.

Robert [5:47]: And to cooperate with Python, so I think we are probably pretty good people to talk about that, because we professionally work with Python for some time, and doing this parallelization in Python is hell, in a very big simplification. And Go is basically created for that. So running something in the background is just a matter of using the Go keyword. And it’s simplification obviously, but compared to Python, it’s like deadly easy.

Miłosz [6:20]: Other than that, you will use probably the same tools you use now for development with LLMs, just calling them, and you know, it won’t be that much different.

Robert [6:35]: But I think this parallelization, this is the key, because when you’re building some agentic flow, usually you’re doing some operations that are slow. They are not CTO-bound operations, but they are still calling some API, they are doing some operations, and just doing it synchronously, it’s super, super slow. And this is why I think Go is just perfect language. I think it’s not only my opinion that Go is a perfect language for building AI or LLM or Gen AI applications.

Miłosz [7:09]: Because it works. We will have a separate post on our blog soon. It writes up in detail how I do it. In short, there are a couple of complex parts, right, working with AI. One is prompt engineering. The hardest probably for me was writing evals, so testing the AI code. It’s super difficult. And the last one probably is orchestration of it all, which is the same code you are used to writing. It’s kind of like business logic for your AI agent.

Robert [7:46]: The only difference is that the result that you expect is not that strict. As you mentioned, the problem is that if you have written some prompt, it will maybe work in 90% of cases, but maybe just 10% is the hardest when it’s doing some bad things that totally unexpected, and you need to spend a lot of time to convince it that, no, no, no, it’s not about what… This is not Go-specific, but Go, I think, is helpful for that because, for example, the Go test package, it works pretty nice with that. So you can do some simple wrappers and write evals for that. So that’s pretty cool.

Miłosz [8:28]: In general, expressing business logic in Go is quite easy, or at least works for us well. Because we have this explicit approach. You have no magic in place. You can just re-read code and you know what it does. So it helps when you define Domain objects, business logic and such, and similarly, when you write this agent logic, similar idea. So it’s much easier this way.

Robert [8:56]: I think it’s one important thing to mention here that if you are talking about, writing some Gen.AI applications, it’s not data science. I mean, because often when somebody is talking about machine learning, AI, Python is mentioned, but Python is great for data engineering, for data science, but using general AI, I think it’s pretty far away from data science. It’s more backend at the end. I mean, you’re doing some API, you obviously additionally have prompt engineering, but I mean, it’s big simplification, but it’s very, very close to backend in general. And a lot of good practices that you know from backend systems are applying one-to-one.

Miłosz [9:46]: You don’t need much background about machine learning or RLMS in general to start.

Robert [9:52]: I think probably a lot of people can compare it to driving a car. It doesn’t need to be mechanic to drive a car. Maybe if you would like to go to F1, maybe you need to have some mechanics background to understand very special details of the car, but it’s just or top level.

Miłosz [10:14]: Can we comment on the audio? It’s not clear. Pull your mic closer. Or even maybe using some specialized tools of API.

Miłosz [10:29]: It’s basically the same idea.

Robert [10:38]: Okay, so we mentioned Go, how it’s compatible with AI for our world. I think it’s one more thing, because AI is very wide, ARIA, let’s say. So we have building some JN AI applications. We have also Vibe coding. We have generating some code on production. I think it’s also very interesting, we will think about Go, because Go is also a great language to be generated, thanks to the syntax that is simple. but it’s also very strict. So, for example, if you have some agent that is generating your code, it’s easy to verify that. There are no many options actually how it can be generated. Like other languages, you can do things in 10 ways. That’s making generation a bit harder and it’s also harder to get the syntax right. But in Go, thanks to having it pretty simple, I think I also seen even some studies based on that and covering that.

Miłosz [11:36]: You’ve got types also in compiling, which is a big plus compared to dynamic languages.

Robert [11:43]: Yeah, yeah. So, and I think it’s also out loud that, okay, you can probably easily generate JavaScript that will compile, let’s say, but it doesn’t mean that it will work properly. And for example, with Go, this is the language that for the first time I have the ceiling that, okay, I’ve written something, it’s compiled, and it works. With languages that are script languages, it’s not that obvious, because it can execute, but at some point it can show exception that, okay, it doesn’t make sense, sorry.

Miłosz [12:15]: Yeah, you’re relying far more on testing.

Robert [12:18]: Yeah.

Miłosz [12:20]: That’s why I remember working with Python. We need far more assertions about what, for example, what some method returns, which is just not an issue with strongly typed languages, which is, no, it’s like nothing new, but if you generate the code, the less time you need to spend reading it, the better probably. So you don’t need to analyze, you don’t need to test such as edge cases, because the compiler does it for you. That’s a B+.

Robert [12:52]: Yeah, so Tldr, in our opinion, and not only our Go is great for that. Oh, and maybe one last point about that, it’s performance, because also some people might say, okay, Python is slow, but I know maybe Rust will be faster. But remember, so what’s the battle like in your Gen.AI applications? It’s not CPU. It’s usually I.O. call done to some external server, so basically it’s doing some network calls. And for that, actually not CPU matters that much. It’s more about goroutines. And Go is I think even more efficient than Rust in running goroutines because it’s not using OS, I mean, it’s using operating system threads, but it has also its runtime.

Miłosz [13:38]: I guess you can do the same lightweight threads or coroutines in Rust but with probably not such a built-in API. But it’s also something that many people overlook. They focus too much on the performance. I’ve seen some AI frameworks that claim that they are the fastest framework in the market. And as I said, it makes zero sense because you will wait a couple of seconds for the alarm to respond anyway. So if you optimize nanoseconds or milliseconds of your framework, virtually no gain. It’s similar to how people tend to benchmark Go routers for HTTP, which apart from extreme cases is not that important. If you if you talk about network and you use JSON Marshalling and you will be kind of slow but good enough anyway so the nanoseconds don’t matter that much yep ah great

Robert [14:57]: Okay so the next question uh so the next question that you received is about how software development looks in go is it just a driven development uh so tdd yeah so it’s interesting question so i would be interested to hear more about the background but probably will not know but yeah, I think it depends a lot and it’s not like everybody does TDD in Go. I would even say that probably more people could do that.

Miłosz [15:34]: Yeah, I think TDD is very popular in Go, but probably similar to other high-level languages. I don’t think there’s a big difference there. It’s just TDD is not, no, it’s language agnostic, so it doesn’t matter that much.

Robert [15:54]: And probably compiler adds one extra step because you need to make this code compiled at least, but it’s a different level.

Miłosz [16:04]: Do you use TDD?

Robert [16:06]: It depends. So yeah, I think it depends on what kind of code I write. So for code that I know how it will behave, and it’s kind of clearly defined what I expect for that, it makes sense, because I can start with tests earlier. One nice thing about TDD is also that it’s helping to make nicer interfaces of the things, because, okay, you know what’s the expected interface, and you’re writing code for that.

Miłosz [16:40]: Or designing the API. Yeah, a nice tip.

Robert [16:44]: But, for example, when I am experimenting, or I don’t know what I would like to achieve, or I know it may be a bit controversial, but I know that for this functionality I don’t need tests, because it’s just obvious enough, and it will not add enough value. It also doesn’t make sense.

Miłosz [17:06]: I actually find it helpful when working with AI to generate code. Sometimes I will use AI to generate something like, you know, a very specialized algorithm. For example, the other day I had to process a file to, I don’t remember, remove some lines or whatever in a very weird way, and this actually is pretty cool that you can define the function interface, write the tests for it, which are very simple because you have just input and output, then use AI to generate the body of the function.

Robert [17:43]: And AI has nice specification basically.

Miłosz [17:46]: Yeah, you can even give it the tests. And what I like about it is that if it’s a small function and I can define the tests well, I don’t really need to care if it’s cover or corner cases. Basically, I don’t need to care how the code works. I can treat it like an external library when you use an external library you don’t read all of it to make sure it’s nice and I don’t know,

Robert [18:15]: More optimal sometimes better to not know yeah.

Miłosz [18:20]: Exactly and so this is similar so this is kind of TDD, I would say because I focus more on the tests and I can write down first and then generate it and now I can just take

Robert [18:33]: AI TDD.

Miłosz [18:37]: I can take a look on the generated code, but I’m like, okay, if it works, I’m fine. Whatever.

Robert [18:44]: I think also to make it super clear, so there are multiple, let’s say, classes of tiers of code. So it’s some component critical code. Probably this approach may be not best, but sometimes you have code that you don’t care that much. It’s some simple functionality.

Miłosz [19:01]: You don’t want to maintain over time.

Robert [19:03]: Yeah, maybe you’re just writing some ugly tool to do something once. And I would say that even just writing tests for that, it’s a lot.

Miłosz [19:15]: For me, it’s like algorithms in general. Like when you have to parse something, when you go line by line and save some results over time. I had a couple of examples like this recently. and this works pretty well because you can do it but it will take probably about an hour or an hour yeah unless you like doing this it’s

Robert [19:37]: Also like a cool it’s a strange hobby but I know that people like that.

Miłosz [19:42]: It’s like a cool puzzle to solve although people complain about hacker rank exercises so maybe not everyone likes that but yeah it’s But coming back to the question about what is the software development, how does software development look like in Go? I would say it’s like any other high-level language probably. There’s no specialized methods.

Robert [20:16]: But also, from my experience, it’s also pretty, let’s say, opinion, but I have also some feeling that compared to some other languages, in general, some library code quality is a bit higher than in other languages. So maybe it’s also thanks to the language, to how strict it is, but this is my feeling that I have sometimes also. I don’t know, maybe it’s subjective, but it’s something in that.

Miłosz [20:50]: Yeah, perhaps.

Robert [20:53]: Probably it’s not related to this class-driven development.

Miłosz [20:56]: There are some approaches, but I’m not sure if I would count them as a methodology, compared to the TDD. Like you know avoiding frameworks could be one example but it’s more like what the community likes to do

Robert [21:11]: Not.

Miłosz [21:12]: Really a methodology

Robert [21:14]: But it’s also coming from the language design yeah it’s I mean language is designed in a way that it’s, good that you don’t need to go into frameworks because you have some interfaces already defined that you can integrate with so it’s a nice thing It’s some special thing about Go, but it’s not a big deal for others. One good thing. I guess that the person that asked this question may not… The person mentioned that this person is learning Go, but also keep in mind that Go has one big nice advantage. You can learn Go very quickly and be productive pretty quickly. So when, for example, we had our teams, in many cases we hired people that didn’t write a line of code in Go earlier.

Miłosz [22:05]: Because it was difficult back then to find people like that.

Robert [22:09]: Yeah, but from the other side, it wasn’t an issue at the end, because those people were productive very, very quickly. And it’s not like there is some magical technique that you need to learn to use Go. It’s basically, you can learn the basics of the language and use the principles that you use in many other languages.

Miłosz [22:33]: Well, the risk here is that if you hire a team with people from different backgrounds, we have a diverse team, which is nice, but also people might start writing Go, like they used to write Python or Java or whatever. So they need to be kind of aligned. So we spent some time to help them learn.

Robert [22:58]: And I think this is an argument that we heard very often, that in terms of complex codebases that people were, copying some patterns blindly from Java or from C Sharp, but at the end it was very complex. So yeah, it’s probably something to watch out.

Miłosz [23:17]: Yeah, it’s a classic discussion around clean architecture. Is it still Go or is it Java? you should start doing it.

Robert [23:29]: Stop doing Java from Go.

Miłosz [23:35]: Well, I guess you can apply it to any language, really.

Robert [23:39]: That’s true.

Miłosz [23:41]: Go too far.

Robert [23:44]: That’s true. Okay, so another interesting question that we had was about switching a role. So the person was a sysadmin, and the question is actually how to… So we can probably generalize that. So how you can basically switch a role when you are, for example,

Robert [24:02]: sysadmin or maybe data scientist or something else and try to go into software engineer path. So maybe do you have some examples of people that did this transformation and how they approached it?

Miłosz [24:21]: Yeah, I’ve seen it happen within teams. It’s much easier to do it within the same company than just trying to be hired right away. It’s basically like starting in software development in general. Everyone knows being a junior is not ideal because we need some skills, to take proven skills into a portfolio to be hired, and it’s the same idea. But if you are already hired in some kind of team that works closely with software, it might be easier to start helping this team a bit, learning how they work, I might be doing some part-time work there for a time so I can see if you like it and if they like it. Yeah, I had a friend who was in the chat switch from front-end to back-end this way. Kind of similar, I guess. Like sysadmin to back-end and front-end to back-end I guess, pretty close, maybe front-end is a bit closer because it’s software development, but still you need some skills you didn’t have before.

Robert [25:49]: And I think, so what you said, it’s some kind of internal team transfer and if you are kind of person that people know within the company that they can trust you and you are nice, it’s a bit easier to do this transfer because if you are joining a new company, it’s harder because it requires much more it involves much more risk for this company, so hiring, this kind of person let’s say also it’s possible, but I think it requires much more time to prepare portfolio, maybe some project that you work in and yeah, it’s probably easier than just going outside of tech to tech obviously if you are within the tech.

Miłosz [26:32]: So it’s much easier because you already have some contacts with the people you can spend some time with them see how they work I’ve seen people do it from like a technical support role for example because this was area where they had to know the technology very well already to help clients and they were in close contact with developers so it was quite natural to just over time move towards development and remember,

Robert [27:03]: Be nice to other people because if you are not nice for those people, it’s much harder to do this move. But unfortunately, sometimes if you are doing this kind of big change, sometimes you need to expect some salary downgrade.

Miłosz [27:20]: Unfortunately. Yeah, for sure. And title downgrade probably too.

Robert [27:26]: Yeah, but it may be worth sometimes, especially if you don’t feel in your role that it’s something that you would like to continue with. And yeah, I think also one nice tip maybe to try, okay, if you are looking for a company where… Try to find a company when you can use both parts of your skill sets, for example your sysadmin. So obviously it’s not always easy to find this kind of company, especially if market is harder, but sometimes you may find a company when they are looking for a software engineer with sysadmin skills, for example, and it can be nice advantage. Or for example, if you’re switching from front-end to back-end, you may look for a full stack role and try to do less and less.

Miłosz [28:14]: So this is the idea of this T-shaped skills, right? They are super proficient in one area and a few others, just a bit. But it’s great if you have a team and you have someone with, for example, cloud skills or SysApp skills. So the rest of the team doesn’t need to go that deep in this topic. And this person usually is happy to handle them.

Robert [28:39]: And yeah, this is the thing that also you recommend to do to basically go very wide with the skills. Obviously at the beginning you can focus on one area, but, At some point, I think it’s nice to try other areas, for example, for a back-end developer, try a bit of front-end, a bit of data science, a bit of sysadmin, a bit of devops. Why devops is probably not the best term for that, but okay, let’s use it. And it’s nice, because maybe you may find a role that is different and you like more, for example. And I think some projects that you are doing after hours, it’s pretty good for learning that. But you might discover that, okay, maybe there is some role that you prefer more. And I think also this kind of people are also very valuable in the team, because if you need to collaborate with other teams, so with some sysadmins, with frontend, with some cloud ops or whatever, you have more knowledge and you can, you know, have a discussion and, you know, because if you are working as a developer, you often may need something from your cloud operations team. And if you have no idea how it works, it’s, you know, sometimes hard to request something.

Miłosz [29:59]: Yeah, so you have more skills in-house. You don’t need to rely that much on external teams.

Robert [30:04]: And it’s also nice because if you are working in a startup, you can just deliver more things because sometimes you may have some small front-end functionality or maybe some simple data science work and you don’t need to outsource it to other team, wait ages for somebody that will do that. You can just do it yourself. I think it’s great. And this is, for example, what we are doing in Trios Labs. So we are kind of lucky that we did many areas so we can basically work to people and we can do a lot of stuff because we can have some experience so we are not front-end experts we are not data engineering experts but we know the field enough that we can do it in the level that is and.

Miłosz [30:48]: Just use it to beat something end to end so we are not stuck or leverage some tools to help you do it. Because if you would need to hire someone to do it for you, that would be super expensive.

Robert [31:05]: Expensive and the overhead is also pretty big, because if you’re doing something alone, you can just do it, ship it, and it’s working. But if you are outsourcing it to somebody, it’s probably 10 times more expensive in terms of time.

Miłosz [31:20]: And the communication overhead is not that it’s an issue to talk with people, but there’s always this scoping and discussions and agreeing on things, so this will take ages.

Robert [31:36]: It’s not only for indie companies like ours, it’s even for bigger companies where you have teams and requiring something from other teams is just not the most efficient way. And there is a follow-up question for this question. If it’s worth to invest in some older technologies like Laravel or PHP. And I think it’s maybe also an interesting way actually to do some switch because obviously working with older technologies, okay, it’s maybe not the most interesting thing in the world. But from the other side, the upside is that less people are keen to work with that. So basically, the competition is…

Miłosz [32:22]: And there is Legasse to be worked on.

Robert [32:24]: Yeah, and you can actually learn a lot on the Legasse. Usually, I remember when I learned a lot, it was the hardest projects. The projects that are greenfield and everything is great, you will not learn that much during that.

Miłosz [32:37]: I would probably consider if the language is growing or declining rapidly. I guess if you consider Java let’s say or maybe PHP too there’s many many projects building those languages. They probably don’t grow rapidly right now. I didn’t check the indexes but I just guess. It might still make sense to go get into them because there will be lots of work in a way. But if you compare it to something that’s declining by, I don’t know, Peril? Maybe Ruby? Yeah, so that’s something that you need to watch out for probably. Not to spend too much time learning something that will be obsolete.

Robert [33:28]: Yeah, yeah.

Miłosz [33:29]: But if you are just getting into tech and you learn one language well, it will be worth it probably anyway, because then you can pick up the next one much easier.

Robert [33:39]: To switch between programming languages than between titles, basically. And yeah, probably one more dimension there, so local market. So if you’re looking for hybrid or on-site, it’s also specific.

Miłosz [34:07]: Job market in general. Check what the market looks like, which is also not ideal, because it can change in a few years, but probably not that fast. So we should be safe at least for five years or so. For example, with Go it was a bit risky when we started working on it full-time because there weren’t that many jobs, but also it was we learned quickly, so it wasn’t a huge investment.

Robert [34:41]: And the competition, if some companies started to ask about Go wasn’t that big.

Miłosz [34:47]: Oh, yeah.

Robert [34:48]: As we said, so we’re later looking for people keen to switch to Go because there were no Go people in the market.

Miłosz [34:56]: Right, yeah, and that’s also interesting. Sometimes, you know, in a situation like there’s a niche language which some companies use and they are desperate to hire, and there are a few developers who are also desperate to be hired because they want to learn in this technology. But they don’t always match each other, depending on the location and so on. So, one more thought. Maybe it’s easier to stick to the proven but old technology than learning something fresh. You know, something that’s not stable yet. Because it might never get traction.

Miłosz [35:41]: I think maybe Elixir is one example. I’m not sure how the job market looks like for it, but I remember learning it around the time I learned Go and And I think betting on Go was probably a more pragmatic choice.

Robert [35:57]: Yeah, definitely.

Miłosz [35:58]: I think by now, Go is much more popular.

Robert [36:02]: Yeah, I remember that a long, long time ago, also, it started to be some Scala hype. And I guess that you can still have some Scala job.

Miłosz [36:12]: But… Probably Kotlin killed it a bit, right?

Robert [36:17]: It’s a bit different, but it also had some specific use cases. But I think the entry point also wasn’t the best thing. But at some point, it was trying to get some traction. People went into that, but I would say that, okay, maybe in some areas it’s still there, but it’s not longer hot.

Miłosz [36:38]: So you have those early adopters, companies who jump on it and hire people, but there’s not enough of them. Over time, we just kind of die down.

Robert [36:52]: It’s just a reminder about the thing that we are advocating all the time. Don’t focus on learning specific technologies. Obviously, you should have some baseline, but we covered it more in an episode about efficient learning. If you focus on the things that are language agnostic and timeless, it will work for you for much longer. Because if you focus on learning some frameworks, languages, obviously you need to have some base, but it’s not the most important thing. And this was also the key when we were looking for people to our Go teams, that we’ve been not asking about how to implement something in Go, because we just know that it doesn’t matter that much. What mattered for us was how they are solving problems.

Miłosz [37:42]: How they

Robert [37:43]: Can for example get some requirements and stuff like that and some general.

Miłosz [37:50]: Stuff Firmina shared in the chat that Java killed Kotlin oh yeah I don’t know so I will trust you’re right long late leave Java I agree and Firmina also asks if one year and a half is enough time to let it go So, I think day and a half is good enough.

Robert [38:17]: One evening I think it’s enough.

Miłosz [38:19]: Yeah, it’s ideal. What’s the next question?

Robert [38:27]: The next question is about clean architecture. You would like to take that.

Miłosz [38:35]: Okay, so this is about where to define the data interfaces. Should you define it in a data layer or in the implementations? And this actually is something that where the language matters. At least if you compare Go to, let’s say, Java. And one cool feature of Go is implicit interfaces.

Robert [39:03]: In other words, duck typing. So, if it’s ducks, it’s a duck.

Miłosz [39:13]: In Java, in many other languages, you have to explicitly say this class implements this interface. In Go, we don’t need to do it. And it has this nice advantage that but you can keep the interfaces close to where you use them, so probably near the… not near the implementation of the interface, but near the thing that uses it. Usually the application layer or use cases or whatever you want to call it. Some people don’t like that, but I think it’s quite cool. you just need a good IDE to show you what implements an interface. That’s a bit hard. If you are a diehard emacs fan, that might be a problem. Or maybe you need some plugins to navigate. But other than that, it’s quite easier to keep the interface close to where it is injected. or used. And this solves this issue of a separate layer for the interfaces, like ports and adapters, where you need to keep them in one place. In Go, you don’t need this layer at all. It’s much simpler.

Robert [40:41]: And I think it’s one big advantage of this duck typing and this non-explicit implementation, is that it’s much easier to implement interface segregation principles. Basically, you no longer need to have interfaces that have 30 methods, for example, for some big repository. Because instead, you can just have multiple interfaces with the methods that you need in this use case. And you can inject this repository with multiple methods there. But you are making the contract of the interface much smaller. Because you are just defining which methods are required. And it’s useful for refactoring, because sometimes you have a repository with that many methods, you can just cut it to multiple repositories and just inject it a bit differently. And it’s nice. And it’s also nice for testing, because when you’re also writing tests, when you have some stubs or mocks, you can just stub or mock smaller interfaces.

Miłosz [41:40]: Yeah, that’s also great. Small interfaces and close to where they are used. That’s the idea of scenario. We covered this in more detail in the episode on Aclia Architecture. If you want to check it out, it’s the third one,

Robert [42:01]: I think. And just a reminder, so if you have some other questions, listen on the chat, so we’ll also cover it. Like, what’s the joke? Okay, so let’s maybe take a question from the chat. if MocnyŻuk asked the question. So how do you handle timeouts in distributed system in sync communication, HTTP, GRPC? So probably it can be answered from multiple levels, let’s say. But in general… How it should be handled. So I guess the question might be, okay, if it’s about distributed systems, I guess it may be about some chain of calls. So one service is called by something and it’s calling other services. So in Go, for example, it’s handled out of the box with context. So when you have the client that is calling your server and the connection is closed, so it’s propagated basically through context and it should be propagated to HTTP Hender. It should be propagated to other requests that are going further. And if you are passing context, for example, to your database queries, it should be also cancelled and it’s doing some kind of chain of cancellation and at the end the connection is closed and everything is great.

Miłosz [43:22]: What about more like a high-level approach? How do you define timeouts? Let’s say you have a service that calls another service. Do you set a constant timeout for it?

Robert [43:36]: So usually i would say that at the beginning you have some endpoints from your message handler and usually it’s good to have some default timeouts let’s say like 30 seconds let’s say for http request and the idea is pretty simple pretty similar because you have this context cancellation that is propagated to everything else basically yeah.

Miłosz [43:59]: I would keep it very very short or sync communication, especially for internal services. Because for external ones, external APIs, you sometimes can’t control it. And then I would consider hiding it behind a message. Basically publishing a message, and then this message triggers this endpoint, and if it times out, it will just be destroyed eventually. If it’s internal, I would keep it super short. Because long requests are usually a pain to handle.

Robert [44:38]: But it’s important to not forget… So if you are talking exactly about Go here, so passing context, because sometimes I notice this problem that somewhere this context wasn’t passed, and context background was passed, and it may be problematic when you are doing some I.O. Call, like if it’s HTTP or some database call, it can run for much longer than you would.

Miłosz [45:05]: Like to run.

Robert [45:06]: And even if you have middleware with the timeout, it doesn’t matter because it’s not propagated and it will run, maybe not forever, but for pretty long.

Miłosz [45:20]: Yeah. And working with timeouts, there’s one more difficulty. If you have a chain of services that call each other and they are slow, and then the timeout happens on the end, don’t really know what happened in the system, the last one. Maybe it was just network failure or whatever.

Robert [45:43]: Especially you don’t have transactions. Because if you have transactions, if you have transactions, again, we are talking about Go, but context cancellation should cancel that and it should roll back things. Yeah.

Miłosz [45:55]: So it’s often where people will consider using a distributed transaction, maybe.

Robert [46:03]: Or at least some message in the middle, maybe event or command.

Miłosz [46:08]: Yeah, so there’s something to watch out for. Ideally, those endpoints will be idempotent.

Robert [46:14]: You should try it.

Miłosz [46:16]: If it happens.

Robert [46:17]: Or ideally, the operations will be fast, and the things that you’re doing synchronously, maybe it can add something to the queue, and it’s just responding pretty quickly, and everything else is done in the background and maybe you can.

Miłosz [46:32]: Okay, I do have a follow-up. Let me read it. I have a scenario where a given call is propagated for various services via REST API. Therefore, you need to set up smaller and smaller timeouts the deeper you go in the tree of services. Oh yeah, okay, I see your point. So you have a long chain of services that call each other. And I don’t like the approach to use increasing timeouts from top to bottom. That’s why this question originated. How it should be handled? Right, so you have this chain. The first service needs to be aware. There will be lots of calls.

Robert [47:17]: So I assume that if you are mentioning that it’s not like the entire chain is executed within one second, something tells me that it’s taking a couple seconds or up to 30 seconds, for example.

Miłosz [47:30]: Yeah, maybe minutes.

Robert [47:32]: So, I don’t know. So it’s hard to say without seeing the system, but it’s maybe a case for some more asynchronous communication, for example.

Miłosz [47:43]: I would consider it,

Robert [47:44]: Yeah. Yeah, because in this case, you may have a problem of cascading failures as well. Or also other solution that I know that for other people is crazy. Maybe we don’t need as many microservices. It depends, because I don’t know this case. maybe every team has its own service and its multiple teams and its very big product. That’s fine in this case. But if it’s, I don’t know, one team has 20 services, maybe it doesn’t need to be 20 services. Again, it’s hard to say. I’m more saying, more mentioning, more alternative, let’s say, to consider that, but…

Miłosz [48:28]: But in general, if you have this spider web of service communication, I would probably try some asynchronous approach, because there are many issues there. For example, each service needs to react to some error. I’m not sure if this is like one service calling 10 or 10 in a sequence, but any time any failure happens, maybe it’s timeout, maybe it’s a network issue or some domain error. It needs to handle it and it propagates it up the stack and then the next service also needs to handle this somehow.

Robert [49:06]: And if every service in the chain is storing something in the database, I hope in transaction, but from my experience, something tells me that it may be not always the case.

Miłosz [49:17]: It’s more risky. But the transaction spans only one service anyway, so I would consider doing it with a message. Publishing a message with the first service, and then all the others reacting to it somehow.

Robert [49:32]: So, start with the question. Do you need that many microservices, PCS.

Miłosz [49:37]: Might be too late for that.

Robert [49:40]: Yeah, okay. That’s true also.

Miłosz [49:42]: But the interesting approach has the downside that you don’t get the error right away. I mean, you don’t know the result of the other services, what they execute.

Robert [49:56]: Also, one more alternative, because, maybe it’s also not that big issue, and maybe you can increase this timeout a bit more.

Miłosz [50:09]: Yeah, it’s not stuck out the most naive.

Robert [50:12]: Yeah, I mean, it’s a naive option, but we also know how it looks like. In perfect world, it would be good to join microservices, use asynchronous architecture for the places where we need but sometimes it’s not important enough and we don’t have enough time yeah.

Miłosz [50:32]: Depends how many activities are on there because if you add a new service to the chain every week, it will get much worse with times but if it’s something you don’t need to touch over the year whatever, we can just bomb the time out So,

Robert [50:52]: I hope it didn’t help. But yeah, jokes aside, so, well, favorite face. So, it depends, but…

Miłosz [51:03]: Yeah, I would consider if async approach can work here. That could simplify it.

Robert [51:08]: Even simple ones. So, also, the thing that we are often advocating for is, even if you don’t have any message broker, you have Postgres. You can have a very simple message broker on Postgres and it will be also fine for the beginning.

Miłosz [51:22]: I love how helpful we are that someone asks about sync communication specifically and our answer is to use async.

Robert [51:32]: Yeah, but this is the case of end with.

Miłosz [51:37]: Yeah, but other than that, there’s not really much you can do. You can bump the timeouts, sell it everywhere to high value and live with it maybe it’s not an issue or redesign the system work in a different way So, some frontline service that would send an event and then waits for the finished event. If you wait for a finished event, it’s kind of like a timeout. You just do it over messages.

Robert [52:16]: So, it depends if you’d like to wait for this event in context of request. So, basically, usually you do it in the way that you’re sending some event, and the request is finished.

Miłosz [52:32]: And you can, for example, have a status in the database that each service writes to. And then you can display it on some kind of UI.

Robert [52:41]: But, again, it’s not for free.

Miłosz [52:44]: It really depends on the use case.

Robert [52:46]: On the complexity of the system, how big is the team…

Miłosz [52:50]: If you want to share more about the exact use case you have, You can join our Discord and we can help you figure it out more in detail because it’s hard to…

Robert [53:00]: Yeah, I think the size of the team, the scale, it’s important here, because I think the answer depends on this context a lot, basically.

Miłosz [53:13]: Hello, full dump. Nice to see you. Thanks for joining us. And drop your questions if you have any. Okay, should we move on to the next question, is also about Clean Architecture, I think?

Robert [53:31]: Yep, that’s true. So, my team started using Clean Architecture about a month ago, and it has been quite a learning experience. But overall, we are seeing the benefits as we wrap our heads around it. One thing we got stuck on sometimes is when to determine the boundary between application and domain layers. This is the question that we all usually have. Or question, what is domain logic? This is also a pretty interesting one. So we’re using SQLRS in application layers. Sometimes the handlers are so simple for crowd operation that we don’t have any dependency on the domain layer. Other may be than entities. We have other handlers that do a lot more and that’s ended up with calling domain-specific logic. Is it common? So to recap, I think it’s about if we need to have domain logic basically in our applications or we can simplify that because it’s just some simple crud and it doesn’t make sense to do some domain magic there.

Miłosz [54:41]: Yeah, I remember the long discussions about this we used to have.

Robert [54:46]: What is domain logic?

Miłosz [54:47]: Yeah, what is application logic? So, maybe we can start with that you don’t need a domain logic in every service you have.

Robert [55:01]: Even if you have domain logic, because you can probably say domain logic about many things, like you’re doing if in SQL query, it’s domain logic.

Miłosz [55:10]: Yeah, so how I like to think about it is like, instead of four layers or whatever you might have, just think about it as a split between implementation details and the logic. That’s a very basic split. You want this domain code or business code away from the SQL queries. And then, if you find it useful, you can split this logic into pure domain logic and all the other orchestration, application things that join things together. And it’s not like a huge deal, but sometimes they work with very complex domains, where this domain layer will be huge, because the business rules are huge basically. And you need to model what your software does. And there’s no other way than writing it down in the code. So it helps to have a separate layer that’s not polluted by any database, calls, or whatever it’s just pure functions and structs and lots of tests for it yeah,

Robert [56:27]: And I think for what we advocate here is trying to be mindful about that and asking yourself a question whatever I’m doing, is it helping me or not? For example, okay, I’m not using domain layer I have a lot of logic in my application layer okay, maybe it’s not helpful maybe I should have domain layer or maybe the opposite side for example, I have domain layer with encapsulation and a lot of stupid mapping for CRUD so I’m just mapping fields between layers and not doing anything for that I don’t have tests for that and I see that it’s not giving me any value okay, it should be not there and it’s not easy to give a good answer if you should have a domain layer or not. You need to learn that. Oh, by doing. And I don’t have a better answer than just writing this kind of logic and over. And yeah, you will have some puzzle.

Miłosz [57:24]: This is the hard part that you can’t really define that. And I remember having the same issue because we have some meeting where we agree, okay, guys, this is application, domain logic, blah, blah, blah. We set this you know the system in place so everyone knows and then you have this edge case like when you for example like this person said they do just crud and then you start to wonder you know should it be there should it be there mm-hmm It’s hard to give one answer to this, but most of the time it also doesn’t really matter that much.

Robert [58:01]: Just try and be mindful. I know it’s not easy, and I think it’s also not easy with Skunko’s fallacy, because if you already spent a lot of time to build your beautiful domain, layer, and at the end you see that it’s just useless, it’s a bit hard to delete that, I know.

Miłosz [58:21]: So if you could define very strict rules for your application, we wouldn’t need probably admin developers because we would just wipe code or we would just use those builders for software.

Robert [58:40]: Or you will use spreadsheets. So I think it’s interesting, heuristic, or kind of. Like, if you’re building a crowd application and you’re feeling that this is enough, maybe you don’t need an application for that, maybe just something can do a super big spreadsheet doing that. Okay, it’s maybe not most optimal, but sometimes it’s more than enough for the beginning, especially if you’re working in a startup, because you might not know if it will be useful or it will be trolled away. and, you know, the spreadsheet for the beginning. Right, fine.

Miłosz [59:16]: I think when you said that in the beginning, it was important to you. Just think of what you get. If you are just starting out, do you need this full design of all the layers and splitting? Probably not, because you don’t really know what will be there. But if you have application layer and so handler, and you find 500 lines of code there with 50 ifs and lots of conditions, lots of arguments and so on. Then it kind of feels natural to just extract this thing somewhere else. It can be another function, another struct, or if you have many places like this, just do a separate layer of it. And that’s basically the idea.

Robert [1:00:03]: The downside is that often we don’t have enough time to do that, so the pain is starting to be bigger and bigger and bigger. But the more complex it is, the harder it’s refactor that. But from the other side, I think it’s also nice, because if something is bad enough, it’s easier to propose changes.

Miłosz [1:00:22]: You feel the pain, so you know it will help to change it.

Robert [1:00:25]: For example, Mocnej Zhuk asked about these timeouts. I’m guessing, but it may be two scenarios here. That those timeouts aren’t painful and it may be worth investing in migrating, for example, to event-driven architecture, but maybe it’s not painful enough and just increasing timeout, it will be good enough and, okay, we have some bigger problems. And the same is with refactoring. So, for example, if your model in the application layer is complex enough and it’s slowing down your implementation a lot, you might have time to actually refactor that. And obviously, it would be great to be right at the beginning and start with domain logic and have tests from the day zero, but also let’s be realistic that you may be wrong with your assumptions, that you may assume that, okay, it will be crud, but at the end, the complexity will explode. It happens. It happens even for us when you are working up. But the reason for that is not that it’s sometimes some lack of skill, but it’s often lack of understanding. That if you are starting to work on something you may not have full understanding of the problem space and you will discover it during implementation and there’s just no way that you guess that okay maybe you will have luck and you will.

Robert [1:01:51]: Implementing in a domain logic. But you may have a bit less luck and you assume, okay, it’s simple, complex, it will explode. And even if you do your best to guess it should have domain layer or not.

Miłosz [1:02:07]: You will be not always right. Next month, your boss can come in and say, actually, you know, we decided not to do this.

Robert [1:02:15]: Or we signed a customer that needs this, this and this.

Miłosz [1:02:18]: Yeah, can I just revert what you did last month? Because we need to change the domain model. And it’s actually fine, you know, when this happens. So your domain model is never ready. You need to adjust it over time.

Robert [1:02:34]: So yeah, TLDR is totally fine to not have domain layer, just put everything to the application layer.

Miłosz [1:02:41]: I use some of it and some DOM.

Robert [1:02:43]: Exactly, exactly. And what’s even better, and it’s not like if it’s good or not, but it’s, I would say, necessary for applications to have modules that have domain layer and have modules that doesn’t have domain layer. And it depends a lot on the application, but usually this is a good idea, because trying to keep consistency for everything, it’s usually a bad idea. Because usually you have these modules that, even if you have very complex applications, not everything is critical.

Miłosz [1:03:13]: Although it will be much simpler to have one standard for everything.

Robert [1:03:18]: Yeah, but it will be just very expensive compared to how much you will gain from that.

Miłosz [1:03:28]: So I really like this idea of fixing something once you feel the pain. If something is not painful maybe it’s fine even if it’s not the best maybe it could be refactored maybe it could be split to modules or whatever but if it’s not painful you probably have more stuff in the backlog to take care of and

Robert [1:03:54]: You will never finish.

Miłosz [1:03:55]: Everything yeah, you will just keep growing it that’s normal, that happens in every company every project Oh, I think we can answer this one.

Robert [1:04:09]: But we have another one about clean architecture. Oh, okay. So, what about clean architecture and small teams with complex projects? So it’s a question about the episode that we had. So the most popular thing that we have. So 10k views on just YouTube. So yeah, this is probably why it generated…

Miłosz [1:04:32]: So there we mentioned that architecture shines for bigger teams and complex projects. So if you have a smaller team or a simpler project, better to avoid it. The question is, what about small team and complex projects? Yes.

Robert [1:04:51]: Yeah.

Miłosz [1:04:54]: Yeah.

Robert [1:04:55]: So it makes sense. So we are a two-people team, and even if we are a quite small team, we are using some lightweight implementation of Clean Clean Architecture, because if you were already doing that, it doesn’t add that much overhead, and you are doing it properly, and it can basically help in many, cases, even if you’re a small team. Again, the cost is not very big if you’re already proficient with doing that, and you can just have benefits out of that. It’s also some natural way of doing things, but again, it’s about doing it in a lightweight way.

Miłosz [1:05:40]: I even do it for my own projects, very small ones, because it comes kind of natural. But yeah, the key point here is to do it in a very lightweight way. For example, not inject everything from an interface like we often see in Dan. That can be crazy.

Robert [1:06:02]: Also, the thing that we mentioned in the previous question, you don’t need to have domain layer with the clean architecture. It’s not a must.

Miłosz [1:06:10]: You don’t even need application layer if you don’t know.

Robert [1:06:13]: And probably you shouldn’t have more than four layers, because we’ve seen the comments on Redis that do.

Miłosz [1:06:19]: You remember how it was? Yeah, it was like Clean architecture is over-engineering. You shouldn’t have more than four layers in your project. We’re just free. I don’t know what some people do, but it sounds like over-engineering then, for sure. If you manage to keep it simple, you can do it for small projects, for sure. For small teams, complex projects. Sure. Why not? I’ll just watch out for making it overcomplicated for no reason. Once again, this question of why it helps can be helpful. I want to understand why you do it, not just do it because someone says, this is a nice idea. So for example, do you run integration or component tests? If you do, the bendless injection is what you need basically to do it in the same way. If maybe you don’t want component tests at all, maybe then you don’t need the prevention injection at all. Let’s try to understand if it makes sense or not.

Robert [1:07:32]: And it was an interesting follow-up question and all that. So if it’s something like under-engineering, not spending any time on architecture at all.

Miłosz [1:07:47]: Yeah my favorite quote here is that there is no there’s no such thing as no design there’s only bad design that seems similar I just don’t spend any time on proper design or any kind of architecture, you’ll end up with this big ball of mud. And sometimes it’s fine if it’s some project you do over the weekend and we’ll scrape or your vibe code.

Robert [1:08:19]: Or it’s proof of concept that you will hopefully throw away after proving if it works.

Miłosz [1:08:27]: But yeah, keep in mind it will happen. So is it what you want? If you want to maintain this thing over time, maybe just spend some time, a few hours to draw some diagrams, try to sort it out in your head how it should look like.

Robert [1:08:45]: And I think it’s one interesting thing that we covered a lot in the… One interesting thing and side effect of this node design that we covered a lot in the previous episode, how it affects code review. So if you didn’t make any design in your team, so The code review, so the PR that you may submit, may, let’s say, surprise some people. And surprising PRs are usually not best because it will trigger a lot of discussions, that resolving will take ages, that fixing will take ages, and probably it will take more time that you will spend on design. And I know nobody likes meetings, but I think I like having big PRs that are discussed for ages less than meetings. And I like the sentence that five days of coding can save one day of planning. And it’s often the case. and even if it’s not visible on the first site.

Robert [1:10:01]: It’s often visible during review that is going for ages. And later people are blaming that code review is awful because you have a lot of the comments, you need to resolve that. Let’s just push everything to main. But it’s not a problem of code review. It’s a problem of surprising people. And don’t surprise people with your code review.

Miłosz [1:10:24]: Architecture sounds like something complex that architects do. But it’s basically stuff like if you have 10 services connected with synchronous calls, what happens with timeouts? That’s basically it. Try to think about it, what will happen, how to handle this and then prepare for it somehow or either fix it or just agree it’s fine and note it, document it somehow.

Robert [1:10:55]: Or a different thing that I also remember that we faced one time is not thinking really how teams are split so let’s imagine that you have some process, and let’s imagine that the beginning of the process is handled by team a later team b is doing something the process later team a is doing something again and later some team c, yeah yeah yeah and it’s pretty problematic because nobody can really own some big chunk of the process because it’s really chopped for small parts that are between teams and it’s, hard to do some kind of handover of big.

Robert [1:11:42]: Let’s say, events in this process. And again, it’s a matter of a bit of design and thinking actually how the process looks on the high level. And obviously, it can be fixed later, but it’s also problematic because often it means that, okay, if this part of the process is handled by this team and it’s between parts that are owned by this team A, maybe this part of the process that is in the middle should be also owned by team A. And, okay, it can be done, the service can be migrated to other teams, but it’s expensive, because it was written by other teams. Probably it will be harder to maintain, because you have no idea how it works. Maybe this team will be also not happy about that, because they will say, I already have 50 services to maintain, we don’t have capacity for that. and yeah it’s like.

Miłosz [1:12:39]: A very high level architecture like an organizational there’s this team topologist book that talks about this I think it’s quite nice if someone wants more on this topic you can check it out

Robert [1:12:52]: And we also mentioned in some previous question that even if you are not manager if you are not architect it’s nice to have this knowledge from other areas for example like team topologist that is, a bit higher level, but from the other side, you can also propose better things and be promoted much faster, because very often those people that are on higher, levels, they may not know that.

Miłosz [1:13:20]: Yeah, that’s true. We can help them fix things, which can also help you in your career. Yeah, so it’s nice to have those more orthogonal knowledge. That’s pure development. That’s a good example of this, too.

Robert [1:13:39]: Yeah, and this is the thing, that if you are going to more senior roles, you’ll notice that, okay, writing software is important, but there are many things around that. And to really be beyond this senior level, you need to understand much more things and also go beyond technical things, basically.

Miłosz [1:14:01]: Yeah, and AI just confirms it. Writing code is cheap somehow I don’t hear many great stories about you know generated codes we are not probably doing much more meaningful work thanks to it to figure out anyway what you need to do in the first place and I think it’s

Robert [1:14:26]: The thing that people that are most senior can get more out of that basically.

Miłosz [1:14:34]: Yeah, exactly. So you get boring parts away, but you still need to figure it out on a high level and design it properly.

Robert [1:14:45]: Alright, cool. So I think we are shortly going out of questions, as long as you not put something in the chat. Or as long as you don’t have something to add about previous questions that came into your mind.

Miłosz [1:15:04]: So not as long in the chat if you have any more questions.

Robert [1:15:08]: Last chance before our holiday season.

Miłosz [1:15:12]: Yeah. So we can share a bit more about our future plans.

Robert [1:15:17]: Yeah, so short term, resting a bit. So I think we had a super busy time now with Go Event Driven V2. So about doing the big update, also giving you support.

Robert [1:15:33]: Any horror stories?

Miłosz [1:15:39]: We need a separate episode about startup horror stories. Oh, yeah. But I also want people to share theirs because I love reading this.

Robert [1:15:50]: One of my favorite ones is about sending a couple million SMSs to one person within one minute. Remember, add some trodling.

Miłosz [1:16:04]: Thanks for the suggestion for me I think we need a separate episode about this and I hope to hear people’s stories as well

Robert [1:16:16]: So after resting what are, our next plans I think we can officially confirm that we know what will be our next training that a lot of people were asking about. So do you want to share these news?

Miłosz [1:16:34]: Yeah, so we plan to release our next training, which will be the Domain Driven Design in Go. And there will be also a hands-on training like Go Event Driven and Go in one evening. Let’s focus on Domain Driven Design. Or I would

Robert [1:16:51]: Even say that obviously Domain Driven Design, it’s some way how to go there, but from the high level it’s more about how to build application with complex domain logic. So many questions you heard about this complex domain logic is for a good reason, because applications that are successful often have these parts that are very complex and require a bit more love than the crowd applications. And those are usually the unique part applications that are making money for the company. And the interesting thing is that, companies are kind of happy to hire people that can work with that because, one thing is that they need to usually touch this code update this code to make more money but from other side it’s also hard to find people that they can do that because it requires a bit different skill set to do that and and.

Miłosz [1:17:50]: Often people are like i am just the developer give me give me the requirements and i will transfer them into code so yeah that’s not really helpful for most companies, because they need people who can understand the business domain, can help people figure out what the product needs. Basically, product engineering. So this training is about that.

Robert [1:18:14]: We’re pretty lucky that we had a chance to work with some pretty weird domains that well, people try to use CRUD approaches sometimes there and it was actually quite interesting to see how this trying of simple approaches totally failed.

Miłosz [1:18:33]: Or maybe they thought they are using DDD, but DDD was their own idea which also didn’t end up well.

Robert [1:18:41]: Yeah.

Miłosz [1:18:42]: Yeah, so we this would be also hands-on training, like the previous ones. We learned a lot from people’s feedback. We had over thousand trainees in both trainings.

Robert [1:18:58]: Over 2000. I think two and a half thousand people.

Miłosz [1:19:04]: So we received lots of feedback. We had some things we wanted to improve. That’s why we prepared the second edition of Go Event Driven. That should be much more fun to go through and easier in general. So we want to take those learnings to the next training as well. Make it more focused on what’s the most important and focus on real life scenarios

Robert [1:19:34]: If you don’t like coding, sorry, it’s very hands-on.

Miłosz [1:19:40]: You need to code. There’s no videos to watch.

Robert [1:19:43]: And yeah, I think that this is a pretty common thing. You need to learn by practice, when to go simple way and where to go complex way. This is actually what was helpful for us. just working with many projects in a daily job, after work, and trying to get this muscle memory to have some idea what approach to use there. And this is why we are advocates of hands-on, because this is the only way actually how you can gain this skill. And you can get in work in the projects and do some crazy over-engineering stuff at the end, or maybe it’s a bit better to learn it in a controlled environment.

Miłosz [1:20:33]: So you don’t need to confess to your manager why you need two months of rework and refactoring. So you can experiment on your own project or our guidance. So this will be in Go, initially, but we probably will have more languages later, but that’s probably not the start. Just as with the other training, we will have a pre-sale, which will give the best price. So basically we want to check if there’s enough interest before we start working on it. And of course you can ask for refund and so on. That’s the same rules as we did in

Robert [1:21:19]: September so for now a bit uh a bit of rest a bit of thinking what we’ll do uh in quarter.

Miłosz [1:21:27]: And also there’s also one thing we want to complete which is go with the domain ebook so we have some cool ideas about to release the second edition or the 1.0 maybe edition is a better name

Robert [1:21:43]: Yeah because there are still some chapters that were not finished.

Miłosz [1:21:47]: That’s long overdue for a few years, so finally you should have some time to do it.

Robert [1:21:53]: But you know, probably it’s also good to share that if you have projects in your work that are taking more time than your thoughts, you’re not the owner. We also have this problem, but for a good reason often. And yeah, as we said, in September we’ll also do some updates of the NoSilverBullets form. It’s a bit secret, so you’ll see on September, but I think it will be quite cool. That’s it, I think.

Miłosz [1:22:33]: I have no more questions.

Miłosz [1:22:36]: Firmina is laughing at us, so that’s time to wrap up.

Robert [1:22:42]: Yeah so thanks for being with us and see you in two months yeah thank.

Miłosz [1:22:47]: You everyone for joining us next time

Robert [1:22:50]: See you next time ciao.

How to Create PRs That Get Merged The Same Day

Wed, 25 Jun 2025 16:00:00 +0000

Quick takeaways

Prioritize reviews over new work - treat PRs as work that’s almost done and needs to be pushed to production quickly
Big PRs create a deadly loop - when reviews take ages, developers make even bigger PRs to avoid multiple long waits
Knowledge sharing is the hidden benefit - code reviews aren’t just gatekeeping, they spread understanding of how things work across the team
One-day cycle time is possible - start work in the morning and merge by end of day with proper team culture and practices
Split work vertically and horizontally - break features into small slices and layers that multiple people can implement in parallel

Introduction

In this episode, we discuss how to make code reviews fast and effective by keeping pull requests small.

We explore why big PRs are problematic, what causes them, and practical strategies to create PRs that can be merged within a day.

Instead of waiting days for reviews with 200 comments, we focus on techniques that help teams achieve smooth, fast review cycles where work flows quickly from code to production.

Show Notes

Feature flags - technique for merging code that’s not ready yet
Mob programming - multiple people coding together with one driver
Clean Architecture - helps with splitting features layer by layer
Linters - automated tools for catching style issues instead of nitpicking in reviews
Draft PRs - for getting early feedback on experimental work
Shadow mode - running new code alongside old code to compare results
The Phoenix Project - book mentioned about bottlenecks in teams

Quotes

If you have a couple developers that are independently working on some features, they are actually not a team. They are a set of individuals working on something. And at the end, there is no shared responsibility to deliver some features.

Robert

It’s a well-known phenomenon that if you have small PR, people will find three defects, but if you have huge PR, they will just write LGTM.

Miłosz

I like the phrase that five days of coding can save one day of planning. And I think it’s often true and it’s often a problem in multiple teams.

Robert

I sometimes see people mention that code reviews are a sign of lack of trust. As in, “You don’t trust your teammates because you need to review code they write.” I think it’s funny because I find it the other way. So I actually trust them to review my code because I don’t trust myself enough to know I don’t make any mistakes.

Miłosz

I think it’s one of the most important skills for product engineering, and I think it’s super rare as well. Because most developers, when they get a clear description of a problem from a stakeholder, will just say, okay, we can do it, because they want to be the hero.

Miłosz

I know that a lot of people are like, let’s skip planning, let’s just write that and ship that and it will be everything fine. And I know that I was also like that when I was younger and less experienced, but later you can kind of learn that, okay, it can give you some acceleration at the beginning, let’s say, but later it can be very problematic.

Robert

Timestamps

Transcript

Robert [0:00]: Ever waited days for Code Review only to get 200 comments and endless discussions in these PRs, so maybe you wonder what’s the reason of this kind of big PRs and why it’s happened that such big PRs are happening.

Robert [0:18]: And what’s often the problem, it’s getting worse. I mean, when you are waiting ages for PRs to be merged, it’s creating some kind of dead loop, let’s say.

Miłosz [0:33]: It gets even worse.

Robert [0:35]: It gets even worse, because if you know that you wait ages for merging a PR, you are just making them bigger, because you know that if you prepare 10 smaller PRs, you will wait longer for those PRs. Today, in this live podcast, we’ll discuss what are the problems that are causing that PRs are big and some strategies that we used in our teams to make those PRs merged much faster. So by much faster, I mean how to create PRs that are merged within a day. So let’s imagine a scenario when you are starting to work on some task in the morning and it’s merged by the end of the day. Sounds like a perfect word that doesn’t exist, but our experience is a bit different.

Miłosz [1:24]: Can’t be possible. This is our last live episode, so feel free to leave comments in the chat, or we’ll pick them up during the discussion, or at the end we’ll have a Q&A section. So let’s kick off with why we need to review code code at all.

Robert [1:45]: Yes, I think it’s important to start with the fact that a lot of people are tracking code review as gatekeeping, so basically to ensure that changes that are not, expected are not merged. And I think it’s bad. It’s a bad sign if code review is kind of gatekeeping. So from my experience, if code review is gatekeeping, it usually means that you didn’t make a good job earlier on agreeing how things should be implemented. So it’s a matter of some good practices in the team, but it’s also a matter of designing things. So in other words, if you spend time earlier on planning that, it should be not a surprise what’s in the PR.

Miłosz [2:37]: So if someone opens a PR and they start wondering what exactly it is about, then maybe it’s too late. Maybe they don’t have enough context and they want enough to review it in a reasonable pace. I sometimes heard of people starting PRs with checking out the code and running the applications, trying to understand what’s going on. I think we can agree with this. This should be an anti-pattern, unless something unexpected happens, like your team is busy with something else, I don’t know, you ask someone else from outside the team to review your code. But most of the time, you should know beforehand what the PRs will be about.

Robert [3:30]: So, if it’s the case…

Miłosz [3:32]: It’s a perfect word.

Robert [3:34]: That can exist, but we’ll go into that a bit later. But if gatekeeping is not the reason why you are doing code review, it should be some reason why you are actually doing code review. And from my perspective, one of the most important reasons why you are doing code review is the knowledge sharing. And I think it’s not mentioned very often, and I think it’s some kind of hidden advantage that maybe we don’t think that often. But this knowledge sharing is important because when you’re doing review, you’re learning how something is implemented, and it’s helping to spread this knowledge how something is implemented in the team. And it’s pretty important because if you’re spreading this knowledge, it’s easier later to extend this functionality by more people in the team.

Miłosz [4:25]: Because you know where to look for changes if you need to extend them or implement your own feature.

Robert [4:31]: And while usually it’s not the best idea that one person has some area in the code and just this person is touching this code, well, I’ve seen it sometimes in my life. And okay, while in the short term it can be a good strategy because this person really knows how this code works, but at some point it may not scale if it will be more functional to implement there. And the problem starts to be a problem at this time.

Miłosz [4:58]: Also depends how big your team is. For smaller teams it’s probably optimal because if you have a small PR, everyone can take a look and have a rough idea what you are doing. The bigger team is probably a bit more complex, because not everyone can see each PR in progress.

Robert [5:22]: Definitely. But still, if you have a bigger team, we’re not talking about, for example, five people. Because I think if you have five people, it’s a bit easier to share this knowledge and be in the stage that everybody knows what’s done. But for example, if you have 15 people, it’s a bit harder and obviously it doesn’t make sense to not share for everybody. But I would say that if you have some subset there, that makes much more sense.

Miłosz [5:49]: I sometimes see people mention and code reviews are a sign of lack of trust. You don’t trust your teammates because you need to review code they’ve write. I think it’s funny because I find it the other way. So I actually trust them to review my code because I don’t try to trust myself enough to know I don’t make any mistakes. So it’s good to have another pair of eyes. Just take a glance and very often they will have some context you missed. If we had no comments on PRs, we would just stop doing them. But they come useful.

Robert [6:30]: In other words, you should not look on PRs like gatekeeping, but rather something positive that can help you to improve your stuff. I think it’s also important that it shouldn’t require from the reviewer to verify the changes, because I also heard that some teams were doing it like that, that somebody is doing or doing review and verifying if it’s worked properly. I think it’s probably more your responsibility to ensure that the functionality that you implemented is working.

Miłosz [7:02]: Sounds a bit like QA, but anyway, you should be doing QA on your own code. so all right um

Robert [7:12]: Okay i think let’s maybe start with why having big prs is a problem because i think if you are working with software most of people agree that it’s big elephant in the room that big prs are problem but i think a lot of people are not doing anything about that like yeah big prs are bad, but whatever.

Miłosz [7:34]: Or they just say out loud, okay, guys, let’s not do it again.

Robert [7:38]: Let’s not do big PRs.

Miłosz [7:40]: But then it always happens. For me, the first one is the most obvious one. It’s just hard to review. It’s a well-known phenomenon that if you have small PR, people will find free defects, but if you have huge PR, they will just write LGTM. And, yeah, I think we know this paint well. It’s very easy to just glance at a huge change set and not go very deep into the details.

Robert [8:17]: Obviously, we’re losing in this scenario the knowledge sharing that we mentioned at the beginning. If you have really huge PR, there is no way that you’ll read that and that you’ll be able to learn from that how it works.

Miłosz [8:32]: Exactly. And then all the processes like reading, understanding code, comments, and whatever else goes on in the PR just takes longer. And no one likes long discussions especially asynchranos so if you take a huge PR and then on top of this you have 50 comments that will take ages to review and if someone spends weeks or longer working on this PR and then you drop 20 comments on it they are probably not eager to to listen to what you want to say, your suggestions.

Robert [9:19]: You’re just thinking, I would like to get it out as soon as possible.

Miłosz [9:23]: I just want to be done with it and push it through to prod. So, yeah, it’s probably not encouraging to change this. It becomes kind of like an artificial review process but doesn’t give you much.

Robert [9:38]: Probably in this case, it wouldn’t change much if you would not have any cool review. The chance we can push to mine and… Yeah, exactly. Yeah, and another problem with big PRs is that, as you said, when you have the situation when, we are working on this PR for longer time, it’s a big chance that you receive 20 comments and okay, it depends on the team, but usually it may end up with a big rework of the job that you’ve made. So in other words, you will lose a lot of time on writing this code and you’ll lose a lot of time on rewriting it later.

Miłosz [10:20]: And you are very late in the process because someone spent already a week or two writing it. And now if you want to rework it now, it probably will be a huge change.

Robert [10:32]: And even worse, probably in the meantime, when you were working on that in your basement, people were doing other changes and you probably will have a lot of conflicts. It’s even more funny if other person in the team have PR like yours and this person merged it earlier. Okay, if you are not a lot of people in the team, you can always not review this person’s PR, so you are safe that it will not introduce conflicts. But you see the problem, the conflict of interest. I mean, it’s also problematic for you if the other person is merging some big PR, so you are not doing a review for this person because it’s too late for your PR.

Miłosz [11:10]: Or you’re raised to do it first, so you don’t need to deal with conflicts.

Robert [11:14]: Or you can add a lot of comments, so this person PR will be also blocked, so you can merge your PR first.

Miłosz [11:25]: And then, when the conflicts happen, you get an approve and you think it’s ready, and then you just resolve the conflicts. And two hours later, you are finished, but it takes time to do something more productive. So just spend time on maintaining this PR instead of writing code. And this merging can also introduce new bugs or challenges that no one reviews, so it’s another risk.

Miłosz [12:03]: There’s also a longer feedback loop, because communication is asynchronous in the comments, so someone gets sick or has days off, you need to wait for them, they come back, they lost some context, we need a separate discussion about all of this. And also that’s a paradox people might think that small PRs require context switching because you have to very often review code and review what you are doing but in reality big changes are even worse because of this because you need to every time you come back to it you need to refresh this huge context in your mind instead of something very small. So that’s another one. And also if you want to stack PRs, so you want to branch out of an existing branch because you need the changes in there, which is often useful. With small PRs, it’s quite easy to do, not risky, but with big PRs, you block everything. The big PR gets merged.

Robert [13:20]: And later you have PR that depend on PR that depend on another PR. And if the first PR in the chain will need to be changed largely, it’s just an horror to keep it up to date. I have PTSD if I think about such PRs.

Miłosz [13:40]: And then someone asks when this will be done and says, Yeah, we only need to wait for Robert’s PR to be merged. We’ve been waiting for two weeks now, so at the moment. Yeah, we’ve seen it too many times. And other ideas? Yeah, this lack of visibility is also risky because… If you don’t know when the feature will drop, because a PR takes us long to merge, you may be tempted to use estimates, which is also problematic, but this comes from the fact that no one knows when something will happen. You have this huge PR, which seems almost ready to merge, but you keep being added to it and discussed, and you just never know when it’s finished. And the worst of all, you can create a long-living branch. One that takes three months or so to merge. And it’s a completely different version of the software.

Robert [15:03]: It keeps living in parallel.

Miłosz [15:07]: If you ever work with this, you probably don’t want to do it again.

Robert [15:12]: So, it all sounds bad. I mean, there are multiple reasons. Why we should not do big PRs, but from the other side, for some magical reason, from my experience, having big PRs is very often the problem in multiple teams. And there are a couple of things that I believe are creating big PRs. And again, I think it will be mentioned multiple times in this episode that, in my opinion, we very often have good intentions, but despite that, the problem still occurs. And the same is with big PRs. I think that’s why it’s important to be conscious about what can cause big PRs. I think one of the things that is coming to my mind at the beginning is that we have some kind of complete feature mindset.

Robert [16:13]: We like to create a feature and have it work on production ASAP. So it makes sense that we’re working on something, and it’s just nice when you write that and it’s going directly to the production. And it sounds good, but unfortunately often features are not that simple that we can easily implement it within one PR. If we’ll try to do that, okay, maybe someone sometimes will estimate that, yeah, it’s just one day of work. But how often do you think that something is one day of work and it’s ending up with a week of work because there is some hidden complexity that you didn’t take into consideration at the beginning.

Miłosz [16:54]: And splitting it seems like additional work. So you might be tempted to avoid it. You wouldn’t want to go. It should be easier, right?

Robert [17:05]: Yeah. And I think it’s connected to, planning, but In many cases, when we’ll discuss about causes why we have big PRs, it’s connected to how we plan things, how we did some shortcuts maybe with planning, and I think often big PRs are connected to some surprises that we are creating. So what I mean by that is that…

Miłosz [17:37]: So-called missed requirements?

Robert [17:41]: Yes, yes.

Miłosz [17:51]: If we don’t know some edge cases before we start the work, then you’re in progress of working on a PR, you discover something new, you have to stop, maybe discuss with someone else,

Miłosz [18:05]: and then it starts to get tough to push through. Because you put some work, you started some writing some code already, it’s half done, but something blocks you.

Robert [18:19]: Yeah, and obviously, if you include something like that in the PR, if somebody will be reviewing that, it will be kind of surprising. Even if you have some planning process at the beginning, if some person is reviewing your PR, this person might be surprised. I mean, okay, we’re planning this, but why does PR have this, this, that, and this? And this is triggering discussions. and discussions is triggering a slowdown of PRs, basically.

Miłosz [18:50]: Yeah. Sometimes, if PRs get reviewed slowly in general in your team, you may think, like, no, it makes no sense to… What happened?

Robert [19:08]: Oh our camera died.

Miłosz [19:12]: Yeah, so let’s maybe take a minute break.

Robert [19:20]: It’s tough. Yeah. So, this episode is special in a couple dimensions, because the first fact is that our room that we usually are using is used, so we were not able to use that. And our camera is overheating, so…

Miłosz [20:01]: Great. Summer is there.

Robert [20:02]: Summer is there. Cool.

Miłosz [20:04]: So, as I said before…

Robert [20:06]: What do you see? It’s live. if it’s not pre-recorded.

Miłosz [20:10]: So if your PRs get reviewed slowly in general in your team, you might think like, okay, it makes no sense to quit additional PRs because I will wait for all of them for a long time to be reviewed. Maybe let’s just make a bigger run and review it once.

Miłosz [20:30]: But again, the huge PR gets reviewed even slower. So it’s not a good solution really.

Robert [20:40]: And I think it’s also one anti-pattern that I often see. It’s the teams that are not real teams. What I mean by that is that if you have a couple developers that are independently working on some features, they are actually not a team. They are a set of individuals working on something. And at the end, there is no shared responsibility to deliver some features. So, if you have no shared responsibility to deliver features, it’s hard to convince somebody to read your PRs, basically. Because nobody cares.

Miłosz [21:16]: It’s not their job.

Robert [21:18]: My job is to implement my features and save you.

Miłosz [21:22]: I don’t want to spend time on your work, because I’m responsible for mine and there are stakeholders waiting for my work to be done.

Robert [21:32]: Exactly, exactly. And I think it’s a big problem, because if you are not working as a team, but a set of individuals, well, there is no pressure or nobody feels obligated to review stuff.

Miłosz [21:50]: Yeah, so in this case, review is not the team’s responsibility. Basically, no one’s responsibility. Just beg people to do it for you. That sounds like an issue. You can see the setup when there’s a feature mentioned and some stakeholders know to which developer to talk to about it. Because they are seen as owners or masters of this area. This is one of the indicators, I think, that this happens.

Robert [22:33]: We already covered a couple reasons why having big PRs is a problem and how big PRs happen. I’m not sure if you agree with me, but I think one of the most important parts here and reasons for them is lack of planning, basically. Maybe I would say deep enough planning. I like the phrase that five days of coding can save one day of planning. And I think it’s often true and it’s often a problem in multiple teams. I mean, we’re sort of developers, engineers, we like to solve problems with the code. So if we hear about planning, meetings, no, no. Yeah, yeah. It’s always like, no, no. Yeah, yeah. Talking to people, especially non-developers. We are pretty allergic to that.

Miłosz [23:27]: Especially after you hear about the future, you already have a database schema in your mind. You already know how to solve it. All you need to do is just sit down and write.

Robert [23:40]: Yeah, and I kind of understand why people are allergic to deep learning, but from other sides, I think it’s also creating a lot of problems. And often it’s visible in CodeReview, because I think a lot of people are blaming the CodeReview process for many things like code review is such bad, it’s such a waste of time, it’s taking that much time. But I think in previous episodes, we also sometimes had this, let’s say, non-direct, visibility of some errors. So for example, some people are blaming DII for making complexity of the applications because dependency injection is so complicated, etc.

Miłosz [24:30]: So it’s like the tip of the iceberg. Yeah. Everything is the issue, but underneath there’s some more complex issues.

Robert [24:36]: And I think it’s a similar case with code review. A lot of people blame code review for a lot of bad things, but I think it’s just surfacing other problems that you have in the team. For example, if you have, I don’t know, 200 comments in the code PR that you submitted, Probably a good question to ask. Okay, why you have 200 questions there? Why you didn’t agree on that earlier before you started implementation? Because when it’s already implemented, reverting that, changing that, it will be super expensive. If you spent more time a bit earlier during planning and planning properly, probably you’ll waste much less time. And I think it’s not about always planning everything for 100% to not have an unexpected complexity, because it will always happen. And trying to plan that accurately, it will be super expensive in terms of time. But I think it’s, from our experience, we see that spending more time on planning, it can definitely…

Miłosz [25:47]: It’s easy to change the whiteboard. you erase it and rewrite it. It’s much harder to change a 10,000 long 10,000 lines long PR with 100 comments. The earlier you discuss things and agree on how to do it, then the easier you iterate this.

Robert [26:11]: I know that a lot of people are like, let’s skip planning, let’s just write that and ship that and it will be everything fine. And I know that I was also like that when I was younger and less experienced, but later you can kind of learn that, okay, it can give you some acceleration at the beginning, let’s say, but later it can be very problematic. Because, again, you lose many of the advantages that you will have from this code reveal. Knowledge sharing, or if you are doing a kind of more strict process of code reveal, it will be just blocking at some stage, because this stupid approach will be at some point, probably will go with some strange direction that you didn’t agree with somebody else, and it will be just surprising the PR, the surprise that we were mentioning a bit earlier. And again, PRs shouldn’t be surprising, so it should be implementation of what you agreed on high level earlier.

Miłosz [27:16]: Yeah, so it’s another paradox. You save time on planning, but in the end, you spend even more time on the reviews. Just like with the paradox of context switching, when you think smaller PRs cause context switching, but actually the big PRs cause even worse.

Robert [27:40]: And yeah, I think it’s interesting because on the first side, my thing that, yeah, splitting to multiple PRs, it will just add a lot of overhead and it sounds not logical to have multiple PRs instead of one big, but I think what I can recommend is trying to try using smaller PRs and see how different it will be. Obviously, it requires a team culture that supports that, because if you create 10 smaller PRs and nobody will review it kind of immediately, it will not help. And you cannot do much about that, and probably you will end up with big PRs at the end anyway, because it will just start to stack. So yeah, it wouldn’t make sense, but we’ll go into that, what you can do in the team to encourage teammates for a quick review.

Robert [28:40]: So, yeah, we started to mention that missing enough planning obviously can help to create bigger PRs. But again, it’s also important to not go into extreme, because I think it’s also easy to go into really extreme and try to plan for everything, to go really deep into the planning. But it will be very expensive to do. I think it’s probably not a possible event because without touching the code it’s really hard sometimes to find all edge cases and complexities that you have.

Miłosz [29:16]: The balance is in deciding what you want to agree on. For example, we skipped on most implementation details like exact database schema or naming of things. That was up to the implementer most of the time. High-level architecture, like what endpoints do you want, what services this impacts, and so on. Some very high-level picture of it, that’s super useful to have early.

Robert [29:47]: I think it’s all about this non-surprising PR at the end. To think what you should plan earlier, to not surprise the reviewer when you are reviewing that. And if you were not working like that earlier, obviously, it will take some time to learn how to do that. But I think it’s a skill that you are just getting with time. You’re implementing more, you’re learning more how, okay, this is surprising somebody in the PR. Okay, maybe we should raise it during that.

Miłosz [30:20]: Next time, let’s agree with what endpoints they will be exposed. This is a quick discussion, right? Of course, it’s better to have before any code is written. Like, do we want endpoint in this service, or do we want a message handler in this service? High-level, that can take 5 minutes. But after someone implements it, we can make one hour to revert and replace it. Or even worse, if the discussion is asynchronous.

Robert [30:52]: Another thing that it’s worth doing is also including in the planning discussion some product person, let’s say. So it depends on each company how it looks like, but often you have some, maybe not stakeholder, but maybe some product owner or product manager or whatever. And it often may be helpful. So obviously it’s required not to have some discussions that are not super technical because this person will be just bored and this person will not understand anything.

Miłosz [31:23]: So they will do it first. First understand the product.

Robert [31:28]: But I think it’s nice because it also can help to cut scope a lot. So I remember that And often when we are doing it like that, we included product people during the planning phase. So when we have some planning meetings and we have this product person in the meetings, it sometimes helped a lot to simplify scope. Because we started to discuss some features, but we had like, okay, it will be very complicated because this, this, and this. And the person in the same room said, okay, so if it’s complicated, we can just keep that because it’s not that important. And sometimes you can lose hours as a developer to try to figure out some super complex problem, but at the end, nobody will care that much.

Miłosz [32:11]: I think it’s one of the most important skills for product engineering, and I think it’s super rare as well. Because most developers, when they get a clear description of a problem from a stakeholder, will just say, okay, we can do it, because they want to be the hero and show, and we’ll be like this.

Robert [32:32]: And we are problem solvers, so if we see problem, okay, we need to solve this problem, it will be so cool.

Miłosz [32:38]: Especially if it’s well defined, if you have all the requirements and so on, you might be eager to just jump into it. You will tell them, okay, we can do it tomorrow, no problem. They will be like guys you are the best team in the company but then you get stuck on these complex cities and you need to talk to the stakeholders again and it takes a long time to read about everything So, I think it’s a super important skill to be able to pull them early to the discussion and see where you can cut the scope. Or, even if it’s not out of scope entirely, just find the V1 you want to deploy. And that’s super useful to have.

Robert [33:28]: And I think we’re often kind of missing this, that we are implementers of some

Robert [33:35]: external requirements and again, we’re trying to be heroes or.

Miłosz [33:39]: I am just a developer tell me what to do and I will do it so that’s a common approach and E-definance because you can just trust people know what they want to help them figure out what exactly it is they want so basically,

Robert [34:01]: One unexpected strategy to make your PR smaller, implement less things.

Miłosz [34:08]: Yeah, or start with less. That’s maybe more precise. Try to start with a small scope.

Robert [34:17]: But sometimes it’s not possible. I mean, sometimes we’ll ask product master or camera, right again. So, okay, can we simplify that? No, you can. So, what’s the strategy in this case?

Miłosz [34:34]: I guess, breaking down changes, right? So, our ideal scenario would be having PRs that are implemented within one day. So, as you said at the start, someone posts a PR in the morning, it is merged on production in the afternoon. Something like this. This cycle time would be ideal.

Miłosz [35:09]: And there are actually some strategies you can use.

Miłosz [35:17]: Probably the simplest one is to merge code that’s not ready yet. behind the feature flag. But it’s critical that you’re able to manage feature flags in your project one way or another, even if it’s just some configuration. But it allows you to merge the PR, so it’s already on production, even if it’s used. But you avoid these discussions and conflicts, and so on. We just work on the code that’s already on the main branch. Even if not exposed yet. And you can have the tests running, you can enable it on some pre-production environment, you can show it to stakeholders in some very early form, some MVP form, and get some early feedback from them. So that would be my number one for breaking down changes and for small PRs.

Robert [36:30]: Another thing that comes into my mind, it sounds pretty obvious, but basically to have some discussion to some RFC, ADR, some design document, and basically discuss it with the rest of your team and have some plan how we can lead it to some smaller parts. We already mentioned it multiple times. So, I think this magical value is one day that I know for so many people, it sounds like, no, it’s not possible to start implementation in the morning and merge something by the end of the day and not working until midnight. I mean, I know that for some people it sounds crazy, but it’s very possible.

Miłosz [37:14]: Yeah, I remember having five minutes cycle time. If it’s small, you can just ping someone and then take a look and in 10 seconds you get an approve.

Robert [37:26]: And the rest of the people is aligned because you spent some teamwork to discuss how it should be implemented. And it’s not surprising for people. So somebody can take a look quickly on that. See, ah, yeah, it’s what we discussed. Looks great. Merging.

Miłosz [37:43]: And because it’s merged, for the next PR that’s built on top of this, Those changes are already out of context, so you don’t spend time discussing them, but that’s your dream.

Robert [37:57]: Do you have maybe some tips from our experience, how you can split this work

Robert [38:01]: to make it working nicely?

Miłosz [38:05]: Yeah, so you can either split vertically or horizontally. And both kind of make sense. Probably vertically makes more sense at first, so instead of big chunks of the feature, you’ll deploy small parts of it. So for example, you can… Let’s say if you beat a dashboard or something like that, you can deploy just one chart or one button. And the endpoints behind it and so on. So it’s something you can deploy, for example, on staging environment or even in production. And show it to your stakeholders and ask them, is this what you want? It can be a very simple feature, part of the entire feature, but it’s already something you can ask them about.

Robert [39:03]: Yeah, and I think you can go even one step further because, okay, you can have some vertical slice to prepare a chart, but often though vertical slices of entire functionality, it can be still pretty complex. You can also slice this vertical slice horizontally. For example, have separate tasks for SQL repository. So it works pretty nicely with clean architecture that we’ve been discussing in the previous episode, so I recommended to check it out. What Longstar showed is that it helps pretty nicely with splitting these features layer by layer. So it can be implemented by one person or by multiple people, and you can implement… Again, it’s good for more complex features, because for some features it doesn’t make sense to split that because it will just use a solar head. But if it’s a more complex feature, let’s say that it’s one week work for one person. You can always split it by layer. And for example, every person in the team can take one layer. And instead of implementing something for an entire week, you can have five people that are just separate slices with a grid API layer, and it can be even implemented within a day. And it’s cool, because you can show it to some stakeholders after this day and ask, okay, this is something but yeah? No? Okay, so let’s iterate on that. And it’s much, much, much faster.

Miłosz [40:24]: Yeah, I think it’s another superpower that’s very rare in tech teams in general. It gave you to work in parallel on a single feature. More often you will find this approach of, okay, so this person is responsible for this feature, see you again in two weeks on the Spring demo.

Robert [40:41]: And usually it’s not two weeks because there are some challenges or block errors.

Miłosz [40:46]: So instead, if you can agree on the interfaces, then one person can implement the HTTP handler, the other one the message handler, the third one the database adapter, something like that. And it’s quite simple, because you agree on the interfaces first, and then you implement them. It’s really straightforward, and we know it works because we’ve seen it live.

Robert [41:11]: And we have some comparisons. It was a process that we worked over a time, and we have comparison how it worked earlier, when we were not doing splitting, and later, when we did the splitting, and we were able to iterate that fast, and it was cool.

Miłosz [41:27]: Except both vertical and horizontal, splitting is not that easy. Maybe the horizontal is easier because it’s a technical one, but the vertical split is related to product so you need some product sense to understand which features make sense at all and you need to discuss it with someone probably to understand better Oh no,

Robert [41:50]: Talking to people.

Miłosz [41:51]: I think it’s worth doing I’m just saying, I think it’s rare because it’s not trivial you need some effort from the engineers to do it

Robert [42:01]: But after you learn that, and the team will learn that, it’s…

Miłosz [42:05]: That is quite natural, yeah. For sure you can learn it. The entire teams can learn it. And yeah, it will be the best if, side note, there’s no single team lead doing this, but it’s encouraged so everyone can own this with time.

Robert [42:25]: And I think it’s another, we again see the same pattern, that it’s some kind of overhead myth here. You may think that, okay, cutting it by horizontally and vertically, it’s that much overhead for that. But it’s not logical that in theory it will require more time. But from the other side, if you will not cut that and one person will implement that, for some magical reason, it will take longer anyway at the end, even if there is some overhead by cutting that. But if only one person is implementing that, this person may be bored, may have other interruptions, context switching. So at the end, there is also some hidden cost of one person doing all entire work for two weeks. And if you add all the complexities that we mentioned with the non-splitting PRs, but having one big PR instead, it’s starting to be much, much, much harder and take much more time. And again, at the first sight, it may sound like it’s requiring much more effort, but I recommend to check that because after you see that, you can see that in practice, it’s much, much faster. the planned things.

Robert [43:51]: But you may think that, okay, it all sounds nice, but it’s not possible to split all functionalities that are maybe okay. Those functionalities are a bit easier. You can probably split most of them, but sometimes there may be some work that is not possible to split. And the first thing that I have in mind is some kind of big refactoring. So, if you’re doing some big refactoring in a project, sometimes it may be hard to split that. So, I like to compare it to Fatality from Mortal Kombat. So, you know, it’s like you’re getting backbone and, you know, it’s getting all the bones.

Miłosz [44:35]: So, there are lots of unknowns. You just don’t know what you will find.

Robert [44:39]: Yeah, but I also mean that sometimes touching one thing is creating some kind of domino effect. So you touch one thing, but to make it work, you need to adjust a lot of things in the project and need some kind of atomic change at the end. So if you touch that, you probably need to spend one week to just fix everything. And it’s, I think, also a pretty interesting case how to deal with that. Because, OK, this one person can do this big refactoring for a week or two weeks. Later, give a PR and, guys, can you review that? Because, you know, it’s important refactoring. If you are looking at, hmm, okay, two weeks of work, 2,000 files changed, 200,000 lines of code added, the same removed, okay, good luck reviewing that. So, one of the tactics here may be, okay, it just, okay, sounds good, let’s merge that. But I think it’s not the best, because usually you can end up with some brain changes in the code base, and everybody later will be, maybe not surprised in the PR, but surprised if somebody will be, because I’ll take something like that.

Robert [45:48]: So, do you have some examples from our teams? What we did in such scenarios to avoid this? I think probably it happened to us multiple times to do this big PR and close eyes and okay, let’s merge that, whatever. But I think it maybe wasn’t a disaster, but it wasn’t the best strategy.

Miłosz [46:11]: You can try mob programming.

Robert [46:13]: Sounds like an app.

Miłosz [46:14]: If you are brave.

Robert [46:17]: So, what is mob-programming? Because I think a lot of people may not know mob-programming. So maybe some of you know what spur-programming is, but mob-programming, I think it’s much more exotic.

Miłosz [46:30]: It’s spur-programming, but more than a pair. It’s basically a group of people coding together, and one person is the driver with the keyboard, and the rest comment and observe. It’s very similar to per-programming except there’s an entire team, if it’s small enough. Like people, I’d say. And yeah, and you take turns coding and implement the feature together.

Robert [47:02]: Sounds like a big waste of time, isn’t it?

Miłosz [47:06]: Yeah, like all meetings. I think we should all code alone,

Robert [47:12]: Not discuss it in the basement.

Miłosz [47:16]: Yeah, so for many, maybe not for many, but for some kinds of work, this can work well. Because of the knowledge sharing, primarily. So you don’t need to waste time reading PRs, especially if they are big, because we do some refactoring or scaffolding of the project and so on. Instead you do it together. So for sure it doesn’t make sense to do everything like this. If you implement an HTTP handler, there’s probably no need for four people to watch how you do it. But if it’s some critical domain code or some huge refactor, it’s probably quicker this way than if everyone reads the PR separately and comments.

Robert [48:06]: I think it’s probably good to consider alternatives, if you are thinking that it’s a waste of time. Because, okay, what are alternatives? So you can probably just close eyes and merge these big refactoring PRs. It’s some solution, but usually it doesn’t end up well. Sometimes, okay, the person that created this big PR can arrange some meeting and show some highlights of how this feature or refactoring was done, it’s probably a bit better, but still, it’s no way that you can go over, entire thinking process that you did in one or two weeks of development.

Miłosz [48:50]: So you need to consider if it makes sense.

Robert [48:53]: But do you have maybe some tips how to run more programming sessions to keep it efficient? Because you have an entire team in the room, potentially, maybe on the call. I think it’s harder. I mean, I think it’s easier to do it in person, but…

Miłosz [49:12]: Everything is harder on the call. So watch out. But yeah, if you do timebox that should help, so make sure there’s no one-person coding, but you switch seats.

Robert [49:24]: Yeah, I think it’s very important because it’s easy to end up with a situation when one person, sometimes some team lead or whatever, is coding everything and everybody is looking on their phones to update the code because it’s…

Miłosz [49:38]: So I need to keep everyone engaged. Don’t speak away. Watch out for CO2 levels in the room if you use a room make sure your team knows the process and everyone is aligned for example a big risk is if you let people drift into a topic and start discussing every line in the code that can end up terribly I think it works best if your team has been working together for a while. They know each other and I remember we had sessions with some new joiners to the team and that wasn’t going that well because they were not aware of some patterns we were using. They questioned everything.

Robert [50:37]: Yeah, so I think it’s good to have some facilitator.

Miłosz [50:41]: Exactly, you need some heavy facilitator. We can cut discussions.

Robert [50:46]: I remember that what often works for us was that, okay, if we’re parking some discussions, sometimes we left a task for that, and this task could be done asynchronously. Because, again, running a session for an entire team is just expensive in terms of time, basically. And you need to be sure that you are using this time very efficiently.

Miłosz [51:16]: So, sometimes you can also do it with a short session just to start a project. I remember when we started new services, we just use Mood Programming, where we just bootstrap the project or something like that.

Robert [51:30]: Yeah, and it was pretty cool because we had really nice boosts at the beginning, so everybody was aligned. how it was implemented, and it was speeding up work at the beginning very, very much. Because, again, it’s also the theme that is going over and over during this episode, so knowledge sharing. So this very deep knowledge sharing that actually accelerates the work a lot. Because you may have multiple tasks to implement, everybody knows how everything else in the project works, so everybody can pick up the task and tasks are not blocked because, for example, this task can be unimplemented by somebody. Everybody can take that, everybody can review that, and it’s accelerating work a lot. That’s kind of hidden functionality of that, let’s say. But yeah, it can help a lot. And yeah, this is also the same with mob programming. So basically you can mob program the feature at the beginning of the project and and later iterate really, really fast.

Miłosz [52:38]: Maybe we can talk about code review culture, going back to the topic.

Robert [52:43]: Yeah, because I think it’s pretty important because a lot of ideas that we have here, it requires some nice culture in the team. Because again, you can do small TRs, but if nobody will review them, it’s pointless because you end up with waiting for ages for the reviews, so it will be not helpful. So what do you have in mind from our experience? What are the important traits of the team that have high review culture?

Miłosz [53:19]: Maybe what you mentioned before, making the team responsibility, not individual responsibility. The team is responsible for delivering the entire feature product. Anyone should be able to review code for knowledge showing. Avoid the silos where people are closed in their own parts of the code and they know everything about it. No one else is able to suggest changes there.

Robert [53:52]: And by everybody we mean that, okay, if you have 15 people team, or maybe not entire team, but you can have some… Okay, it’s really problematic if your team is 15 people.

Miłosz [54:03]: I guess ideally we have two or three people working on a single feature at a time and they can review each underscore.

Robert [54:11]: I would even say that maybe five. Five is also good because you can have some extra capacity for that. And it’s a number out of an hour.

Miłosz [54:22]: Yeah, but I mean, it’s probably an anti-pattern to have one person responsible for all the reviews. So like the leader with the grey beard who knows everything and everyone just asks them to review their code. That’s probably not what you want, because this person becomes a bottleneck.

Robert [54:44]: It’s also not good when you have your buddy, for example, that is doing your reviews. because it’s a bit better, because it’s not in the hands of one person. By the way, I think many of you may not know the Phoenix Project book. I don’t remember the name, but it was covering a similar situation of one person that was the person that known everything, but it was also a bottleneck at the end. So it’s a very similar situation.

Miłosz [55:16]: I would usually just ask the person who has more experience in the area than I have. So, they can give you some insights on the way that or maybe

Robert [55:25]: Just have a team channel and.

Miłosz [55:28]: Oh yeah

Robert [55:30]: And pick up it and yeah.

Miłosz [55:32]: Yeah, that’s yeah, that’s the best thing times, I remember. What I mentioned about the 5-minute cycle time is when you just post a PR, post a link to some Slack channel and then you get an approval for comments 10 minutes later. That was my ideal working condition.

Robert [55:56]: Yeah, it’s possible. It requires times and good culture.

Miłosz [56:03]: I think the main concern is there is context switching. But once again, switching context to a 50-line long VR is something completely different than 500-line VR.

Robert [56:16]: Or even 200 lines. I mean, it’s also something that you can review pretty quickly.

Miłosz [56:21]: Yeah, I just mean the size in general. If it’s in tens of lines, hundreds of lines, probably still kind of fine, but if it’s thousands, then yeah, probably someone will open it and then they will go for a coffee break or back to their own code and we’ll just make a mental note to come back to it later. If it’s 50 lines or 100 lines, they can just read it in one minute and tell you what they think.

Robert [56:54]: I have actually one thing in mind when we’re discussing that, that one of the problem with the discussion that we have now, it’s that having this so having everything working requires multiple things. For example, if one thing doesn’t work, it can break the good PR flows. For example, if, for example, it takes ages for people to pick up PRs, so everything else can be problematic. So it’s important that to not pick some parts of this, but try to, improve it from multiple angles. Okay, we’re going back to the strong code review culture. So other thing that it’s maybe also not obvious is actually how we are doing reviews. So I have two things in mind.

Robert [57:46]: So the first is the feedback culture and feedback mindset. So you should not try this last.

Robert [57:59]: You should not take this code review feedback personally, let’s say, because I think it’s often that people are looking like, oh, somebody commented on my code, I’m so stupid, and this person is also stupid because he doesn’t like me or whatever. So, try to avoid that and try to separate that thought. This is the cult, this is you, and this is a separate thing. It’s also always good to assume good intentions, because it’s easy to go in a direction like, ah, somebody is commenting on that, would like to make my life harder, and why he’s doing that. And I think it’s very rarely true, so most often it’s that actually somebody cares on the other side, and this person would like to create code that is maintainable and easy to work with. So I think it’s important to start with mindset by default that the person on the other side have good intentions. So this person cares and this person would like to just have nice code. Obviously, sometimes it’s going into extremes. Sometimes the person can have also some bad intentions, but it’s more about the default approach so I think by default and it will be probably true in 80% or 90% cases that people have good intentions unless.

Miłosz [59:19]: They need to pick yeah

Robert [59:23]: And I think it’s another problem so if you don’t know what nitpicking is so you know it’s all those comments like okay it’s not important but maybe you should use spaces here or, change how it’s formatted to a different…

Miłosz [59:41]: Can you order the variables alphabetically?

Robert [59:46]: Yeah, yeah. And all those comments that are…

Miłosz [59:50]: Basically, I would like this code to look like this, but you do it differently.

Robert [59:55]: So you can resolve this comment. I don’t mind. But, yeah. So this is nitpicking, long story short. And in my opinion, If I do a code review and I think about adding a nitpick, I don’t do that. Because if it’s not that important that it’s not required for merging, it doesn’t make sense actually to add any comment, spend other person’s time on resolving that or maybe addressing that, because it’s just not important. In my opinion, all those nitpicks should be covered by the Linter, basically. So if you have some styling requirements, just put Linter for that and not spend human’s time.

Miłosz [1:00:46]: Because they are often cosmetics, so our Linter is good at catching that.

Robert [1:00:50]: Exactly. And Linter have also one nice feature that it has a pretty quick feedback loop. Because if you are leaving a comment in the PR, some person will read it with some delay, some person needs to address that. It’s just a lot of waste of time. And with Linter, you’re just running Linter. You can see, in the worst case, a couple of seconds later. In the best case, in less than one second, that everything is fine or what should be fixed. You can format that and it’s fine. So I think this is how it should be solved with NitPix. So if something is that important enough to check for that, fine. Let’s put linter for that and linter will ensure. If it’s not important, why to do that?

Miłosz [1:01:34]: Makes sense. Also, as the person submitting the PR, you can do some things to make it easier. Something I like to do is being my first reviewer. Just checking the trendset and often you will find some leftovers. Instead of someone commenting it for you, you can just do it yourself. Make their job a bit easier. and they can also include a short description, like why you did something. We sometimes comment on the onlines ourselves just like a heads up for the reviewer like, you know, I know this can look strange, but I did this because of something.

Robert [1:02:21]: Or even better, leave some comments in the code that explains why it’s done in some way. And I think it’s important that you mention that to describe why not how. Because how it’s written, what’s written there, is visible in the code. I mean, it is the source of truth. But why? It’s not in the code. You can only see how it’s done.

Miłosz [1:02:44]: You generate the how with AI. Summary, but it’s not really the point. The point is to give some extra context, not what is already there, but different words.

Robert [1:02:54]: Exactly.

Miłosz [1:02:56]: Maybe one other tactic is draft PRs. If you have some very rough idea, especially if it’s something experimental or something your team didn’t do before, I would create a draft PR and very quickly, in a sloppy way, share with everyone and ask them to give me some first impressions.

Robert [1:03:20]: Especially if you discovered something unexpected in a PR. So remember, don’t surprise the reviewer, and if you feel that, okay, this may surprise somebody, yeah, good place for a draw.

Miłosz [1:03:34]: This works only if you have this early feedback culture. You can get someone’s comments in 15 minutes. I think it’s way better to discuss it early, because if you go deeper into this and you have more tool you work, if you have some insights. And if you ask them early, it’s even better to have some quick meeting to align and agree and then implement.

Robert [1:04:02]: It’s also connected with this idea that you are working as a team, not as a set of individuals. So, you collectively care about this draft.

Miłosz [1:04:13]: And you have to see why we are in that. Yeah, but this needs to be in place because you will need to wait two days for our draft PR review. It doesn’t make zero sense.

Robert [1:04:24]: Exactly. And I think it’s actually something pretty important to touch on. So, what are the tactics to accelerate review cycles? Because I think it’s a requirement for most of the things that we mentioned. So, if your review cycle is slow, it doesn’t make sense to split your PRs to smaller PRs, because if you are creating 10 smaller PRs and you are waiting for them for one week, it’s not a big difference between making one big PR… You’re just stuck PRs needlessly. Yeah, probably in this case it’s better to not split PRs and have big PRs that nobody will really review or create 200 comments. Probably it’s better than having 10 PRs like that. Unfortunately, so I think this accelerating review cycles is key here and keeping this cycle time more is also critical. So one of the most important things and mindset to learn is that making reviews should be.

Robert [1:05:33]: We should prioritize making review, basically. So it’s pretty compatible with the lean Kanban approaches.

Robert [1:05:42]: So basically the idea is that you should prioritize work to be done. So if you have something that is close to be done, PR, this is work that is almost done, let’s say, you should prioritize that and try to get it out of your board, let’s say, as soon as possible. And, It’s a simple idea, but, again, very, very important.

Miłosz [1:06:06]: So, I guess, higher priority than full-in-progress, because it’s closer to production.

Robert [1:06:11]: Exactly, exactly. So, obviously, it requires some context switching. So, if you’re working on some feature and you see that there is some PR to be reviewed, obviously, you probably shouldn’t, if you’re working on something, jump straight into that and review that. But maybe there is some person that finishes the job and can jump in this PR.

Miłosz [1:06:37]: Yes, contact switching can be a problem, probably for some people more than for others, but you probably don’t work eight hours looking at the code anyway. I mean, writing.

Robert [1:06:49]: You are not. We’ll need to talk about it after.

Miłosz [1:06:52]: Maybe you’re supposed to.

Robert [1:06:54]: We’ll talk after recording about that. But jokes aside, yeah. And I think it’s going back to the knowledge sharing. So we covered this knowledge sharing multiple times, and I know that some people are, why we need knowledge sharing? And this is actually a great example why it’s important, because if you don’t have only one person that is able to review your QR, but entire team can do that, you don’t need to wait for this one person. You can probably have one person that finished his work or her work and can look into this PR, and it’s making this acceleration working, basically. If you have only one person that can review your PR, obviously it’s making the cycles much longer, because only this person, multiple stars in the sky need to go into a special…

Miłosz [1:07:48]: Into a line.

Robert [1:07:50]: Yeah, yeah, yeah.

Miłosz [1:07:54]: Yeah, and the end result is this cool flow of PRs going smooth from coach to production.

Robert [1:08:02]: Yeah, so basically you can work on something in the morning and by the end of the day, and you’re not working for 18 hours, and it can be merged within this day. But again, it’s important mindset shift because I know it’s natural to focus on your job to be done. But again, try to focus on finishing work and work as a team, not set of individuals.

Miłosz [1:08:35]: That’s a culture thing. Not easy to change, but helpful to have in place. There’s also this kind of controversial topic of metrics. There are many tools that you connect to your GitHub and that you track things like average PR size, cycle time, timing review and so on. And they can be useful. I think there’s a risk that someone uses them for measuring performance. That’s probably not a good idea, because people will start gaming them and that’s not what you want. Especially if you try to compare teams using them or something like that. I’m not recommended, but you can use it inside the team for insight or maybe to find some outliers. If you have one or two huge PRs in one iteration, you can ask yourself why this happened. Maybe it was okay, maybe we could improve something.

Robert [1:09:45]: But I think it’s also important to set expectations. So we said that, okay, in this ideal work, you are starting in the morning, merging by the end of the day. But also, to set expectations. In our teams, it also wasn’t like that for every PR. So it was probably for 80%, sometimes 70% of PRs. But I think it’s enough.

Miłosz [1:10:07]: So the target is not perfection, right? Yeah. Just that most work goes like this. Sometimes there will be some crazy refactor that you need to do at once. Maybe it’s fine.

Robert [1:10:19]: Sometimes you probably should reflect that, okay, why this took longer, what we can do to improve that, and try to do some kind of continuous improvement. As we were already saying about fancy Japanese words, Aizen.

Miłosz [1:10:34]: So it can be a good starting point for retrospectives in discussion.

Robert [1:10:40]: Yeah, I remember that when we were really into optimizing that, we on the retrospectives were looking on these outliers and discussing, okay, why this PR took, for example, three days, for example, to be merged. And, okay, is this something that we can improve here? Okay, this PR was like that because, I don’t know, it was some unexpected complexity there and, okay, it happens. Probably couldn’t do much about that. It’s fine. Or maybe we can see sometimes, okay, we didn’t spend enough time on planning here. Maybe we can, for example, next time, plan more about this, because this, and just improve. And again, this is a skill. So if you work in such environment for a longer time, you’ll also learn what things are worth to do deeper planning and work when it’s just not productive. Oh, we have some comments. Hi guys, I came late, I have concern. How could you create small PRs if you are working on tickets? And maybe the ticket has complex tasks. How do you recommend doing small PRs in those cases?

Miłosz [1:12:04]: Complex task yeah if you can follow up what you mean by complex but two things we mentioned before so one is splitting the ticket into smaller tasks but by vertically by the product features

Robert [1:12:26]: Maybe it’s the situation when you have tasks from somebody up and you are basically implementing the tasks that you are receiving from some product owner somewhere that is planning your work.

Miłosz [1:12:40]: Oh, yeah, right. I didn’t consider this. Because it’s going to be like this.

Robert [1:12:48]: I think the root cause, if I understand it properly, the root cause is probably that some person is planning your work, as long as I get it properly. But I think it’s the root cause. I mean, it shouldn’t be a process like somebody’s giving you tasks that you should implement and deciding on how to state that. It should be probably the opposite. So you should discuss with this person, okay, what we have to do. So you should actually receive the problem that is to be solved, but not tasks that will be implemented, basically. And it should be some collective process to agree how to do that. Unfortunately, I know it’s not possible in all companies, and it’s sometimes impossible to change. And in this case, change companies.

Miłosz [1:13:36]: If you can split it, try more programming.

Robert [1:13:41]: Or maybe just split it internally, so you can have one task from somewhere else, do process yourself to split it to multiple smaller tasks, and just have smaller PR. So we already mentioned it a bit earlier, that you can merge code that is not functional yet, So it can work in the background, or maybe not. You may have some functions that are implemented but are not included in the production process, let’s say. In the last PRs, for example, you can have multiple PRs, and in the last PR you can integrate everything. In the meantime, if something is much more complicated, you can also do the…

Miłosz [1:14:38]: It’s kind of what you mentioned before,

Robert [1:14:41]: Right?

Miłosz [1:14:44]: Just because a task is complex doesn’t mean you can’t split it into small PRs. You can have the scaffolding first and then use feature flags to deplace on that code to production before you start using it. I’m not sure if you’ve listened the entire episode, but we mentioned this before.

Robert [1:15:06]: But yeah, the downside is that, again, it requires the entire team to kind of come into this approach. because if you have a long review cycle and you have many smaller PRs, it may be just blocked too long. You will not see the benefit of that. I also wanted to share that if you are splitting such tasks in more smaller tasks, if it’s more complex, you can also run shadow mode before the last merge. So you can run new code along the old code and compare if this code is creating the same result without storing it to the database or something like that. So sometimes you may be doing some bigger refactoring or something like that. So maybe you have some tests for that, that you see that, okay, it’s passing, but sometimes you would like to also compare on the production if it’s working as previously, but without breaking anything. So you can run two functions like this alongside within one service and just compare result at the end. Run it for a week, month for a complex one, and if you see that the results are consistent and it’s working nice, you can just swap that. But again, it’s for much more complex tasks.

Robert [1:16:23]: Ah, okay, so I was right. The case is that tasks are assigned to me, and the tasks have goals. In some cases, I need to make changes in several repositories, and maybe I need to touch several files at once. So yeah, in this case, I’ll say that this is probably the pattern that tasks are assigned to you. In my opinion, it should not work like this.

Miłosz [1:16:45]: So yeah, but what can you do in this situation?

Robert [1:16:49]: Yeah, yeah. So, as I said, unfortunately, in many companies, you cannot re-change that. So, the thing that you can do is you can split these tasks internally. And, again, it depends on the team. I know that sometimes it may be hard to change the entire team culture to smaller PRs. Because, okay, maybe you can split your tasks into smaller tasks and try to have smaller PRs, but if nobody will look in your TRs and it will be no teamwork, because, okay, I assume if tasks are assigned to you, it may be problematic, because you are not collectively working on that. It’s your responsibility and it’s harder to actually integrate with that. Sometimes, unfortunately, it’s a bit hard to change the company culture.

Miłosz [1:17:35]: Yeah, but we can still do just this on the technical side.

Robert [1:17:40]: And I would say that, okay, sometimes it’s maybe hard, but not impossible to change that. I mean, we’ve seen sometimes in our life that the company culture was super hard to change, but if you push really hard, you can also do some positive changes.

Miłosz [1:17:54]: But if you can, you can do it also on the technical side of things. You can focus on the logic first and then implementation details or the other way around. The reviewers don’t need to have the full context in their minds at the same time. They can do it in parts. It’s much easier. So there’s some impact you can make.

Robert [1:18:20]: We recommend checking what we’ve mentioned a bit earlier. So we have multiple strategies and hopefully it will help. And again, try to change maybe something in your team. Maybe it will be possible. Good luck! But yeah, I think also don’t be demotivated that it’s so bad. Because from our experience also in the teams, when you need to do these changes and you need to push hard to do that, actually you can learn a lot. Because it’s not like just, oh, let’s do it like this. And everybody’s happy. Yeah, it’s too easy. When you are in the team that you need to re-spend a lot of time to convince your product owner, the rest of the team for that. It will be hard, but if you succeed, it will be much easier in the other teams and it’s a useful skill.

Miłosz [1:19:11]: If you want to succeed, it’s a change of the job.

Robert [1:19:15]: Yep, it’s the last resort, but it often works. And if you succeed in this team, it will be also something nice to get a better job earlier. I mean, if I heard such stories like, yeah, I worked in the team when tasks were assigned to me and I convinced product owner to spend time together to split tasks. I convinced my team to split work into smaller. It was always like, okay, nice. It’s so you know what to do and it’s also helping to join nicer teams later. It’s also a nice opportunity. You should try to see it in this way. All right.

Robert [1:20:02]: Other alternative that I have in mind that you can, Solve problem of big PRs is you know what. What do you mean? I see big PR, I close my eyes and yeah, legit to me, that’s marriage death.

Miłosz [1:20:18]: That is good, yeah.

Robert [1:20:19]: I know that many teams are people who use this tactic and it can also work in some places. I mean, let’s imagine that we’re in a team, in some component that is fighting for survival and the long-term in this case of company is six months. And if they will fail, they will just fail and it will be closed. So obviously, you probably don’t need to care that much about long-term maintainability because long-term may not exist, basically. But if you are in a more mature company or you are not trying to survive, and you know that maybe you are building these features and it’s very likely that it’s not the final, phase of this feature and you will need to change them. It’s pretty likely if you are working in startup because very rarely the idea that you have at the beginning survives multiple iterations. So basically you need to be able to very quickly. In those cases, probably YOLO mode is not the best load.

Miłosz [1:21:33]: It’s a bit like we discussed in the first episode, when to write low-quality code. Sometimes it makes sense, but you need to consider if you are in this place or not.

Robert [1:21:45]: I think it’s also connected to using some basic episodes and using some good practices. Basically, to skip some good practices, you need to understand what this good practice is giving you and what’s skipping this good practice who gets all the gear.

Miłosz [1:22:03]: If it’s worth the short term to drop some… I mean, sorry, if it’s long in the long term, drop some good practice in the short term.

Robert [1:22:14]: Yeah. Like code reviews. Yeah, yeah. Because, okay, maybe just one person can work on some functionality for one month, and, okay, just rush that, and because there is some tight deadline or whatever, okay, fine, but it’s not for free. You will need to pay it back if you will need to extend this functionality, because the rest of the team will have no idea how it’s implemented, and if only one person, implemented that, probably it’s also not the best application in the world and it’s very similar in the case when you have YOLO code reviews. So you’re closing your eyes and yeah, legit to me and so again, it’s not like you may notice that the name of this podcast is No Silver Bullet so it’s not like you should always do code review.

Miłosz [1:23:02]: If you work on a own, you can’t. Unless you use AI I think that’s a good closing point consider what makes sense in this particular scenario also what you mentioned before try to be aware of the perfection trap don’t go too far into don’t take the metric too seriously and don’t say something like we don’t accept PRs above 500 lines ever it’s like don’t make it a hard limit or something like this

Robert [1:23:46]: But i think it’s also important that if it’s happening too often it’s also good to have some pushback like sorry mate but yeah this is, and maybe you can spend some effort on splitting that because often when you have big pr it could be split and it’s not that hard really so especially.

Miłosz [1:24:10]: If you have something like you know have a feature and then you are a good scout, so you can just refactor this a bit.

Robert [1:24:18]: Rename one function that is used in 2000 places in the project.

Miłosz [1:24:22]: So this is a perfect scenario where you can split it to a separate PR. You don’t mix the context.

Robert [1:24:28]: It’s probably five minutes to make a separate PR.

Miłosz [1:24:32]: It will help a lot for the reviewers because they don’t mix the context of refactor with actual changes.

Robert [1:24:39]: I think it’s actually what you said. It’s very important. If you’re doing refactoring, don’t mix it with implementing new functionality, because it’s very, very hard to actually distinguish that. Because if you’re doing refactoring and keeping the functionality working as earlier, it’s much easier to read, because you’re just focusing on nothing. And if you’re changing functionality and doing refactoring, it’s much, much harder to spot actually what is part of refactoring and what is part of new functionality.

Miłosz [1:25:08]: It probably helps if you learn a bit of Git, so you are comfortable branching out of a branch, which in this case is helpful. It’s not super complex, but I know many people are intimidated by Git. Stacking PRs is sometimes a great help. Instead of one huge PR with many mixed changes sometimes it’s easier to create three separate PRs branching from each other which is a bit of work to do huge amount, but for the reviewers it will be much easier to review.

Robert [1:25:50]: But also try to avoid that, because if you’re branching out of the branch, it means that you probably have problem with some basics, like you’re waiting for too long for the review.

Miłosz [1:26:03]: Yeah, but sometimes even if you wait like, you know, four hours for review, so it’s the same day, it makes sense to branch out so you can work on something next.

Robert [1:26:11]: It’s not that big, yeah, yeah, yeah.

Miłosz [1:26:13]: But it’s also… Probably not the default for everything, but I think it’s… If you’re not comfortable doing it, then you will be trapped. Because you will wait for a review and you will think, okay, I can’t work.

Robert [1:26:28]: But there is also one trap. If you try to start to work on the next functionality that is branched out of that, that you’ll receive with you, it probably may make the reverse cycle slower because you’ll need to contact switch back to that. Sometimes it’s maybe better to do and make a coffee, two or three, four hours. But again, as we said earlier… But again, as we said earlier, there are multiple things that you need to join to get this good, fast review cycle, and it’s making a much better effect at the end, when you, just have multiple good practices that we mentioned, and it really flows.

Miłosz [1:27:20]: Yeah, this is a great place to be in when the world just goes straight to the production.

Robert [1:27:26]: And again, if somebody is telling you that, you know, implementing functionality and merging it within one day is just in the books on the conferences, it’s not, really. It’s possible to do. It requires effort. It requires high confidence. Team code review culture, but it’s possible, and it’s very worth, and I really hope that everybody who is listening for that will be able to work in such a team. We did that, and it was just super cool to feel this work flowing and see how fast we’re able to implement things.

Miłosz [1:28:06]: Yeah, I think it’s one of the top experiences you can have in a team.

Robert [1:28:10]: Now we are actually two in our teams, so it’s a bit also easier because we can just…

Miłosz [1:28:17]: Still applies, though.

Robert [1:28:18]: Yeah, yeah, yeah. So obviously it’s a bit easier to do when we’re two.

Miłosz [1:28:24]: But it’s more tempting to build big PR, so we need to watch out.

Robert [1:28:31]: Yep, we also do it from time to time. So don’t be perfectionists. I would say 80% so try to aim for 80% of PRs to be merged within the same day. It’s possible to do. But also, don’t probably 100%, it will be hard. But try to learn from those PRs that are not merged within this timebox. Alright, so I think we are out of the topics so it will be time for Q&A so if you have any questions please leave them on the chat and we’ll be more than happy to ask for them. Also, one small announcement. So, if you are longer with us, and if you are in our newsletter that we highly recommend, we mentioned that we’ll be dropping the… Live from our podcast. So in other words, we’ll pre-record episodes because, it’s a bit hard sometimes to organize everything.

Miłosz [1:29:30]: As we can see today, we have our overheating camera.

Robert [1:29:33]: Yeah, yeah, yeah, exactly. And as also you may notice one week ago, when we were not able to be live because of some sick leaves, etc. And also, it’s not super easy to actually record it live because we need to think really quickly. It’s probably too quick for us, and it’s not that easy to keep this slow. And for our podcasting platform, we are doing edits, and it’s easier to listen for that. But in the YouTube, the version that is staying there is the live version, basically. So there are some pauses, so it’s a bit…

Miłosz [1:30:10]: But it was a fun experiment. We did 10 episodes.

Robert [1:30:14]: Definitely.

Miłosz [1:30:15]: I think we want to come back with live format, but probably something more interesting than this talking. Maybe some coding or something like that.

Robert [1:30:26]: If we have time. Yeah. So we cannot promise, but we’ll also run some AMA sessions from time to time, so the next one in the next week. We should put soon to the description or maybe I will do it now. So I will put to the a link to the form for AMA questions. So I will put it to this episode’s description now. We’ll also cover some questions that we have in the comments on YouTube. Yes. I will just put it to the description now because I may forget that. Can I edit it? What else?

Miłosz [1:31:25]: Yeah, that’s it. Thank you for joining the last live episode, for now at least. And thank you for the comments.

Robert [1:31:34]: Yeah, we’ll see each other next week in the AMA episode. And later we are doing some short summer break because it was actually pretty intensive now or a bit longer. And yeah, we should be back after summer with some other updated podcast. So we’ll have some more surprises, but we’ll see after vacations.

Miłosz [1:32:13]: Cool.

Robert [1:32:14]: All right, cool. So I think we are done for today. So remember to leave some comments if you have under the video, leave thumbs up so the Deadly Algorithm will like us more. Subscribe to our podcast on Spotify, Apple Podcasts, and every application that you are using. Yeah.

Miłosz [1:32:42]: Thank you, everyone.

Robert [1:32:43]: Thank you. And see you next time in a week. Bye.

Event-Driven Architecture: The Hard Parts

Wed, 04 Jun 2025 16:00:00 +0000

Quick takeaways

Event-driven architecture (EDA) is powerful but tricky – it’s great for scaling and decoupling, but has many hidden traps.
Observability is essential – debugging async systems without tracing, logs, and correlation IDs is almost impossible.
Use the outbox pattern – it’s the safest way to publish events without losing data.
Design events carefully – large, generic events can lead to tight coupling and painful refactors.
Avoid over-engineering – sometimes synchronous systems or simple monoliths are just better.
Start with sync if unsure – it’s easier to migrate from a well-structured synchronous system to async later than the other way around.

Introduction

In this episode of No Silver Bullet, we dive deep into the real-world challenges of working with event-driven architecture.

We share hard-learned lessons from building distributed systems with events, covering pitfalls like dropped messages, debugging, eventual consistency, and designing events.

We talk about the trade-offs, share practical advice, and help you decide when EDA is actually worth the complexity.

Show Notes

Quotes

If you have a big event that 10 teams depend on, you can’t just say, okay guys, we will replace this event with another one or with 10 other ones.

Miłosz

If you feel like you need distributed transactions between three services, maybe you should just merge them into one.

Robert

The good news here is that message brokers are usually designed so that you don’t lose the message, because it’s quite critical for you if you use EDA.

Miłosz

I heard horror stories about sending 10 millions of SMSs to one person within 10 minutes.

Robert

The point here is that we don’t accept inconsistency, we just accept that it happens a bit later. And it can be also difficult to explain to your product peers or your boss or whoever.

Miłosz

Unfortunately, you cannot test every edge case that you can have in your application. I mean, you can try, but in reality, it’s never enough time, as long as you’re not building really critical services.

Robert

Timestamps

Transcript

Miłosz [0:00]: Event-driven architecture sounds great. You get better scaling, loose coupling, resilient systems.

Miłosz [0:06]: But it’s difficult to get right and can turn into over-engineering. Today we talk about the tough parts that make you wonder if going async was actually a good idea. I am Ior.

Robert [0:18]: And I’m Robert, and this is No Silver Bullet Live Podcast, where we discuss mindful backend engineering. We spent almost 20 years working together across different projects and teams. And we learned that but following advice like always do X or never do Y doesn’t work and can limit your growth. In this show, we share multiple perspectives that will help you to make smart choices and grow into principal engineer level.

Miłosz [0:45]: If you have any questions or comments, you can leave them on the chat. We’ll pick them up during the discussion or at the end we will also have a Q&A

Miłosz [0:55]: session where we can discuss all of it.

Robert [0:57]: Exactly. And today’s episode is kind of a continuation of the previous episode, so if you didn’t have a chance to listen to that, it’s highly recommended, but it’s not required. So, in the previous episode, we discussed the differences between async and sync architecture. Today, we’ll focus on one implementation of architecture that is async, so event-driven architecture. So this is some form of event-driven architecture. And you will take a look on challenges of applying event-driven architectures to your applications. Because we had a chance to work with multiple event-driven applications. So we know when it works nicely, when it doesn’t. So it will be helpful for you to decide if event-driven architecture is for you.

Miłosz [1:45]: In the previous episode, we assumed synchronous architecture is the default approach in many projects. Then we focused on some tips when moving to async is the idea. And today we focus on the challenges, because there are many with EDA. So maybe I’ll start with a very quick TLDR of what event-driven architecture is, which won’t be a complete definition, but just so we are on the same page. So, EventDriven architecture is based on events, obviously, and an event is some state of change in your system. And the idea is that your system or systems publish events after something happens, something changes in the system, and then other services or systems react to it. And can publish their own events, and this is how they communicate, in contrast to calling each other by some more direct ways. And this architecture is often built on top of messages, and that’s what we talk about today.

Robert [3:00]: It’s probably also worth mentioning that you have a message, and message is some kind of way how you are transporting events. So basically, in most cases, event is payload inside of the event. So you can think if you are, for example, in writing Go, so message is something like htRequest and event is part of the body, let’s say. So it’s the analogy here.

Miłosz [3:26]: So event is a kind of a message, very specific message that says that something already happened, some facts happened and then whoever listens is free to react as they see fit exactly,

Robert [3:42]: Okay, but let’s go to the challenges because I think every nice technique has some challenges so yeah, this is the NoCyverBluet podcast and.

Miłosz [3:56]: There are definitely many challenges when it comes to even driven architecture The first one can be debugging issues and the fear of losing events. And I actually have a quote that comes from Reddit comment we received from under the previous episode. And the commenter said, Stuff comes in, doesn’t come out. Things appear that you cannot explain where they came from. The amount of telemetry you have to add on your own to make those things sane is absolutely ridiculous in 2025.

Robert [4:31]: Yeah, probably it sounds like a person that maybe has seen some people misusing this kind of architecture.

Miłosz [4:37]: So you clearly see someone that struggled with this kind of approach before. But I would also say this critique is kind of fair. This can definitely happen. So it’s good to be ready for it and prepare.

Robert [4:55]: And I think it’s a common theme if you were discussing multiple different techniques in previous episodes. It was a common theme that people were always complaining about some techniques. Also in places when this was misused. Because using some technique for some approaches may be just too sophisticated, let’s say. And if you are not solving a real problem with any technique, you will just have more problems at the end.

Miłosz [5:27]: But even if you pick the right tool for the job, still can have this issue. Like, if you just call an HTTP endpoint, it will end up with some status code. If the other end doesn’t support the method, you will get 404. You will right away know what’s going on. Events, this is the hard part. If something goes wrong and you publish an event, and then the other end just doesn’t react to it. That’s the hard part, because you don’t know what happened.

Robert [6:01]: It’s kind of similar to using CI and tests. I mean, if you’re using CI and tests, you may complain that you have a problem that you need to write tests, they’re flaky, and you maintain them. And yeah, it’s true. But maybe the question is, if you have right tests, maybe you should remove some of them in this case. But it’s all about cost versus return of investment, basically.

Miłosz [6:25]: Yeah. The good news here is that message brokers are usually designed so that you don’t lose the message, because it’s quite critical for you if you use EDA. So I guess you could compare it to databases as well, but you don’t see people worried about SQL database losing the data, because it was designed to keep it, right?

Robert [6:55]: And usually, if somebody is complaining that a message broker is losing messages, it’s probably not a problem with the message broker. Let’s say that it’s similar to many people complaining like, oh, I think it’s a kernel bug or a programming language bug. in 99% of cases it’s not. Sometimes it is, but I would not start with assuming that it’s the case.

Miłosz [7:19]: This reminds me of this chapter in the Pragmatic Programmer about select just works. It’s exactly about this issue. So yeah, if you use the message broker the right way you shouldn’t have this issue. Of course it’s not that trivial sometimes to configure everything. But the good news is It is designed to help you not to lose any messages. But still, some weird stuff happens. I have quite recent example of this, where we have been running services on staging. And we got some events missing, just randomly. And it was super frustrating because we had proper telemetry and tracing and event log. And we just couldn’t figure out why this single event doesn’t arrive at the destination. And yeah, the intuition here is to blame the software, right? We were usingbiu RabitMQ at the time and we started wondering how can it be that it just randomly rejects some messages.

Miłosz [8:34]: And of course, this wasn’t the reason. The reason was our frontend team were running our services locally. And they just used the staging RabitMQ address to run it. So unknowingly, they were consuming our messages and the handlers were doing changes in the local database instead of the staging database.

Robert [8:57]: So let’s learn having access to non-development environment from local environment is always a great idea.

Miłosz [9:05]: Yeah, that’s one lesson learned for sure. And the other is, RabbitMQ probably works. So if you see some very weird issue with some event arriving. Probably look into your code, or in this case, who uses your code. Still, it doesn’t need to be super complex to have this observability in place. If you have a tracing setup, probably the best, because you can see the entire flow of the system. But I would say even a simple correlation ID in logs can help a lot. You can just trace which services receive the event you published.

Robert [9:53]: And I assume that if you are thinking about using event-driven architecture, you probably should have some kind of distributed architecture under a hood. And I assume that if you have, you should have already some observability for that in place. Some tracing or at least supporting correlation ID properly. If you are supporting it already for other transports like HTTP or gRPC, adding it for messages, it shouldn’t be hard.

Miłosz [10:28]: It’s also often skipped. Observability is not a super interesting topic. Sometimes if you build a product, it’s often an afterthought. And then you already have trouble if you have no tooling in place and you start using events and then you need to debug something. That would be a terrible idea. So, definitely start with this.

Robert [10:57]: There is actually a question. How you build on the chat? So just a reminder. So if you have any questions, please drop them on the chat. If not now, we’ll answer all of them at the end. But I think that this one is relevant. How you build your testing to catch dropped events like this?

Robert [11:15]: Basically, in this case, I would say that in most cases, what is helpful for us is using component tests. So there are a bit like end-to-end tests, but just on the level of your service. And there pretty useful for that. So in other words, you are calling some endpoint, you are assuming that something is processed, and at the end, it should be some output going from the system that is showing that something was processed. The bad thing is that usually when you are losing some events, it’s some small chance that it will happen. So it’s because of some misconfiguration or some edge case or some lost connection and it’s a bit harder to do that. So in this case, what’s useful is using some library that already have tests for that and is ensuring that no events are missed. Of course, you still can have some misconfiguration, but in most cases, unfortunately, it’s like you might notice something on the production and you need to fix it afterwards. But, for example, in Watermill, so the library that we’re developing, we have a lot of tests that are checking managed cases like disconnecting or doing some strange negative acknowledgement to ensure that no message is lost. And so far it works because, We just have a lot of tests for that.

Miłosz [12:42]: Regarding component tests, it’s important to use the same configuration you use in production. The exact same setup. Because it may be tempting to have something much simpler in tests, so it’s easier to just start. But this way you don’t test this entire topology, which can be sometimes quite difficult. Especially if you have something like a single event topic that routes events to some data lake or event log, something like this. It can be useful to do it for all services, but you should do it the same way in the tests, otherwise you’re not sure if you are testing the right thing.

Robert [13:30]: But in general, if you are using, for example, some libraries like Watermill, it should abstract it from you. And it’s hard to lose this message, basically, as long as you will not do things like something ever happened, and you will think, okay, everything is all right, I’m acknowledging the messaging going forward. But I would say that it’s just about being sane where you acknowledge some message that you are processing.

Miłosz [13:51]: Yeah, but testing is also not everything. You need this tooling to debug these production issues, because this issue of another team consuming your events by mistake, there’s no way you can do a test for it locally or in the CI.

Robert [14:08]: I think it’s probably not only the case for an asynchronous architecture, because it can also happen for anything basically. If someone is interacting with your staging, there’s no way that you can test that, and this is the reality.

Miłosz [14:21]: Yeah, but I think it’s easier to see the HTTP requests, right? Because you usually have logs on both sides. And in this case, you see the event is published, but you don’t see the other end at all. So that might be a bit more difficult to test.

Robert [14:41]: That’s true, that’s true. But yeah, TLDR, unfortunately, you cannot test every edge case that you can have in your application. I mean, you can try, but in reality, it’s never had enough time, as long as you’re not building really critical services. So, in this case, you can spend insane amount of time on ensuring that. But, yeah, it’s…

Miłosz [15:05]: Yeah, so to reiterate use observability from the start. Add some simple correlation ID in logs, tracing if you can. That will help a lot in debugging. And also don’t reinvent the wheel when it comes to using brokers. So for sure don’t implement your own queue from scratch. Probably not a good idea. Similar to not developing your own database. Use proven patterns, brokers and libraries that are tested and they work on production.

Robert [15:47]: And if you were mentioning patterns, we’ll today also mention the outbox pattern. And I think that in this situation, this is probably the most important one because this is a super common issue that people are not using the outbox pattern. And why it’s probably one of the simplest ways to miss events. But we’ll go into that.

Miłosz [16:06]: A good design of events can also help here. The more complex the workflow you have, the more difficult it will be to debug if anything wrong happens. So if you can keep your design simple and easy to grasp, it would be also much easier to debug any issues. So let’s say, if you have a single event that’s being consumed by tens of subscribers, instead of smaller events maybe that have fewer consumers, it might be more difficult to understand why some issues are happening.

Robert [16:55]: And you also mentioned that some people are kind of afraid of this losing message. And I think it’s also important to, you know, if you are building an asynchronous architecture, it doesn’t need to be event-driven, it can be also message-driven. You should just trust it very much. I mean, you should trust that if you are publishing some event, this event is later consumed. Because I heard multiple times that people were kind of not trusting in their infrastructure that they have for that. They feel that, okay, I’m metting some event. I’m not really sure if it will be later constant or whatever. So I would say that this is a red flag. I mean, it should be something that it should be as stable as your database. So when you are doing SQL query, you’re assuming that this database basically works. And this should be also the case for your event driven or message event applications.

Miłosz [17:50]: I wouldn’t like to work with a system that no one trusts the job done

Robert [17:57]: So, the next thing that I would say is also a pretty big challenge is maybe changing the mindset of how you are building your system. Because if you are building your system in a synchronous way, as somebody mentioned on the chat, it’s also kind of more straightforward. So you are doing some operation, you are storing something in the database, some external service is called, and you know that it’s done. And when you’re building an event-driven application, for example, there is some trade.

Robert [18:32]: There’s a use case that we discussed in the previous slide. In this use case, for example, we are reducing user and we are sending some data to some external system. But user registration is usually the critical part of the application. And if we are sending some data to external system, we cannot always trust that the system will be up, especially if you have out of traffic and we have some promotions. And let’s imagine that we have 1,000 users registering in one minute, for example, or even more, and it doesn’t work because some third-party system cannot handle that. It can happen, even if it’s not super big scale. But from the other side, it may be also important to send data to this external system. So, a nice solution, maybe using Event-Driven architecture here, because you can send this data asynchronously. But the downside is that the data that will go to the system may be not always consistent. So, it may happen sometimes that there is some bug, and instead of storing this data synchronously to the system, you’ll need to wait for a while before it will be stored there.

Miłosz [19:39]: And at first it seems like a big issue because how can I live without transactions? What to do now.

Robert [19:46]: But I would say that it’s important to think about the trade-offs here, because what would be the trade-off here? So instead of doing it within the request, if it’s delayed, it probably means that something is wrong.

Robert [19:57]: And if something is wrong, so this request will probably just not work. So it’s much better to have this request working, so your customers will not experience any degradation of your service, everybody will be happy and data will be just delayed this is a trade-off and this is also something that we often, heard when we’re discussing it with some business people that if you ask them that okay does this data need to be consistent and i’m sure that in 90 of cases the person will say yeah yes data need to be always consistent because why it shouldn’t be consistent what will happen if it will be delayed yeah.

Miłosz [20:35]: Yeah it’s like asking if it should be high quality

Robert [20:38]: Of course Yes.

Miłosz [20:39]: It needs to be.

Robert [20:40]: Yeah, boss, can we make bucks?

Miłosz [20:42]: What am I paying for?

Robert [20:44]: And, yeah, obviously, like you said, with quality, we can always write highest quality code. But is this worth? I mean, writing this highest quality code, it will cost more time. Maybe we don’t have this time. Maybe it’s fine to cut some corners and they vary it faster. And this is the same with consistency. So sometimes it’s better to be delayed if you compare it with something not happening at all.

Miłosz [21:11]: So this is the eventual consistency concept from EventDriven architecture. The point here is that we don’t accept inconsistency, we just accept that it happens a bit later.

Robert [21:25]: Eventually.

Miłosz [21:26]: Eventually, yeah. So it’s an important point and it can be also difficult to explain to your product peers or your boss or whoever. Because we normally this inconsistency we know is super small smaller than a second but sometimes it can be longer longer What’s surprising is, very often, it’s more acceptable than you would think. You could have inconsistent systems in reality, sometimes for hours even, and it’s completely fine. So this is more a product decision than purely a technical decision. Something you have to discuss with your stakeholders, basically.

Robert [22:15]: Yeah and i think it’s again always good to show trade-offs so okay it can be delayed but what if for example the system is down what we prefer maybe it’s fine that it will be a bit delayed but it will not stop entire entire users journey you can also for example say how it’s work on bigger social media platform for example so if you are posting a post to x facebook instagram whatever so you see that some post was posted but probably it’s taking a couple seconds before it will be shown in different people’s wall and it’s just because of adding it everywhere synchronously will be just super slow because probably it needs to happen in multiple regions and doing it synchronously it will just sometimes fail and probably it’s super rare that you are adding some posts somewhere and you see the error maybe it’s happening under the hood so for example maybe in some region, it’s added after five minutes, maybe after some hours even, you’d never know. But you don’t see it in the UI, because it’s just eventually consistent.

Miłosz [23:21]: It’s similar when changing your avatar or something like this. You can often see that many components on the website will be updated, others don’t. But also not in the technical sense, you can see it in real life, in how some systems work. Even bank transfers which seem like something that’s always consistent, right? Because you have the ledger. The traditional transfers are not immediate. So it’s a very similar concept.

Robert [24:00]: Yeah, but it’s important to keep in mind. So if you are discussing it with some product owner, some product person, it’s important to show that, okay, this is the trade-off and the UX implication of that. Because if you are adding something to your system, it may be not visible immediately for users. So it’s important to keep it in mind. Sometimes it’s worth asking a question if it’s really needed. If it’s needed to have it eventually consistent. We already mentioned it in the previous episode, but we are big fans of server sent events. So this is really cool technology for handling this kind of updates. Also, I think a link in the episode materials, our blog post showing how to implement it in production-grade environments.

Miłosz [24:52]: So it’s maybe another challenge here that you have to adjust your UI for this kind of errors. Something we also mentioned in the previous episode.

Robert [25:03]: The good thing is that there is also some midway here, because let’s imagine that we are creating some system that is adding some data to your system, and maybe it needs to synchronize some data with external systems. But there is fortunately a mid-air ground that often people are missing, I think. I have an idea why, because to do it properly you need to know a couple of some techniques, but from the other side, it’s not rocket science, especially if you do it already once. So, because if you are doing some changes, you can still store some changes to your database locally in the transaction and also emit an event. It has one challenge compared to that, but it’s nice from the UX perspective, because you can write your data to your source of truth database, that is, for example, database of your service, and you can also emit an event. And it’s nice because if operations exceeded, you can show it to the user immediately and you don’t need to do any magic with servers and events. It will be just there.

Robert [26:05]: But there is one trouble with this approach because, as I said, it assumes that we are storing something in the database and we are emitting an event. But what if something will happen in the middle? So, for example, we stored our data in the database, and our service died before sending the event. And before somebody saying that, oh no, the chance for that is super small, trust me, even for low-scale systems, it’s happening more often than you can think. It’s enough. And you may also say, oh, we have graceful shutdowns. Yes, as long as, for example, your service will not go out of memory, or you have some bug that will not make it shut down gracefully.

Miłosz [26:49]: The earnest network dice, which is super common.

Robert [26:53]: And you cannot do much about that. So it’s actually an interesting challenge to think. Okay, so in this case, we should store the data first to the database and emit our event to Message Broker, or vice versa. Maybe we should emit our event first and store data to the database.

Miłosz [27:10]: Or call the publisher inside the transaction.

Robert [27:14]: Or maybe in Thread and try to synchronize that.

Miłosz [27:18]: Yeah, so basically there is no good Almost no good way out here.

Robert [27:23]: In every situation, we’ll end up with inconsistency, basically.

Miłosz [27:28]: Yeah, but there’s an outbox pattern that you mentioned before, which is a nice way out of this.

Robert [27:35]: Yeah, and it’s also pretty cool in the simplicity. So basically, the idea of an outbox pattern is that the event that we would like to emit, you should store it in the same database as you are storing your data. For example, let’s imagine that your source of truth database is Postgres, So within one transaction, you are storing your event to Postgres, and you are also storing the data that you would like to send to your Postgres, and you are storing the event that you would like to event to this database within one transaction. And later, you are just streaming those events from this table with events to your method broker. And that’s it, basically.

Miłosz [28:18]: And the streaming happens in the background. So it’s essentially the main transaction.

Robert [28:22]: So it’s still eventually consistent, but well, like everything with event driven architecture, everything is eventually consistent. So it’s obviously a bit slower because there is one operation in the middle. So you need to stream it from your Postgres to your message broker. But well, trade-offs.

Miłosz [28:43]: The difficult part is streaming from databases is not the easiest thing to do. And there are some patterns how to do it, but still, it’s easy to make a mistake and, for example, Watermill SQL supports it, but took us three major versions to do it right and avoid some bugs. So it’s also another thing you probably don’t want to write from scratch. Probably better to use something that’s proven.

Robert [29:17]: If you are using Go, use Watermill for that, because Watermill supports Outbook’s implementation out of the box.

Miłosz [29:23]: There’s also a tool called Debezium, we used some time ago. It’s also quite difficult to configure, so there’s no easy answers here.

Robert [29:34]: You put it in a nice words. It was a nightmare.

Miłosz [29:41]: So once again, it’s something you don’t want to reinvent, probably, unless you have a lot of time and can write lots of tests and test it on production properly.

Robert [29:53]: And there are two alternative approaches here that I have in mind. The first one is that in some cases, actually, you can store data within transactions and later emit an event, but it’s only in a situation when it’s not a problem to lose some data in the event. For example, we are storing something in, let’s imagine, some telemetry or some continuous data.

Miłosz [30:18]: Yeah, why you don’t care about losing the message?

Robert [30:21]: Yeah, so in worst case, you may lose one data point, and it’s fine because a couple of seconds later, you’ll have another one. So this is fine. And maybe in this case, if you care about performance, it’s totally fine. The second interesting approach may be also if you’d like to avoid the streaming from PubSub to from your database to PubSub, you can just use your database as PubSub. So for example, in Watermill we have PubSub implementations for SQL, MySQL, Postgres, Firestore. So it can be also done, but it’s also a kind of specific implementation. I would say that by default you should use Outbox and stream it to your message broker. And I think we can probably link the source code of Watermill SQL in the episode notes. If you’re not writing in Go, you can take an inspiration. Ask AI, please rewrite this code in a way that it will not be copyrighted.

Miłosz [31:29]: If you don’t get scared of these SQL queries.

Robert [31:33]: Yeah, but don’t do it yourself. So there are a couple of queries there.

Miłosz [31:38]: Okay, let’s move to the next challenge, which is designing events.

Miłosz [31:44]: This is something that looks quite easy, or seems easy. Because it’s more soft than implementing some technical stuff. But it’s also quite difficult to get right. And especially because there’s no good universal answers here. For example, we often get asked if what’s better, should I have one big event or several or smaller events. Like the difference between user updated and having something like user password changed, user email changed and so on. And there is no good answer to this, because it has many trade-offs either way you go. So small events versus big events or generic versus specific and this kind of discussion there’s just no We’ve got one solution here.

Robert [32:45]: But I think it’s probably one case when it’s a pretty bad idea. So having an event when you’re just putting an entire entity from the database there. We’ve seen it in one system. It was actually pretty interesting archaeology. Because when we joined the component, the event was already like 50 fields. It was crazy big, and it was super hard to mock, and it was super hard to change that. But actually, a couple years earlier, when I checked in Git history, it was a pretty small event. It would have probably four fields or something like that. But it was the same model as it was stored in the database. It was growing, growing, growing, growing, growing. Like a bowling frog, basically.

Miłosz [33:25]: So you can imagine something like user updated with 50 fields in the event with all the details of the user.

Robert [33:33]: And even if you feel that at the beginning it’s not a big issue, maybe. I mean, also you should watch out too, you know, not overcomplicate days from day one.

Miłosz [33:41]: Exactly, that’s the hard part, right? Because, you know, say this is an anti-pattern, but let’s say someone has a small project they do, they write themselves. So then having even like this, maybe it’s good enough.

Robert [33:56]: Yeah, but later, let’s imagine that the company is still there and you have like 10 teams, 50 people, and you have no idea who’s using that, or you would like to remove some part of these fields, almost impossible. And again, it just started with four fields and putting database model directly to event. So again, try to find some balance, but watch out for some extremes that it can…

Miłosz [34:25]: So this is what I mentioned before regarding debugging and production. So we have this one huge event that everyone consumes. Maybe it’s more difficult to trace it and debug it, in contrast to smaller events. But this is just one heuristic. So there’s a question on the chat, what do you prefer? Granular or more composed? I’m not sure I can give one answer here.

Robert [34:57]: Yeah, but I think probably we can risk assessment, but that usually we have more problems with the bigger one, less granular. I think it’s probably easier to join events later than split them, basically.

Miłosz [35:18]: Yeah, because it’s easier to get tight coupling with bigger events, because many systems start depending on it, and then you can just… Yeah, you talk about refactor, and it’s not always that easy. Exactly because of this reason. If you have a big event that 10 teams depend on, You can’t just say, okay guys, we will replace this event with another one or with 10 other ones.

Robert [35:51]: And it’s even worse if one of the teams that depends on that is data science team. If one of those teams is data science, it’s impossible. Sorry.

Miłosz [36:02]: Yeah, refactor is easy if this is an internal event your service uses. That’s cool. But if it becomes a contract with other teams, yeah, that’s much more difficult. So, yeah, probably if you have more teams, it makes sense to start smaller, so at least you have this small contract. Maybe similar to API design of HTTP, probably you want to have a smaller method than one huge update method. It also depends on the API, but in terms of contract, it’s probably easier to track who is using each method rather than how people use this one golden method that does everything.

Robert [36:52]: And I think a good heuristic to see if your events are too granular are cases when you need to do some kind of aggregation. So, basically, to do something, you need to listen to two events and do some magic to, OK, I finally received two events and I can do operations. So, it may be the sign that, OK, maybe it’s too granular. So that’s the challenge.

Miłosz [37:19]: I would say, this lack of good universal answers here. Also, it’s not something very technical. You can just get better at it by reading a book. Most often you want to have discussions or event-storming sessions with other people in the team or in the company, with your stakeholders, to understand how to model it. So that also can be challenging.

Robert [37:48]: Also, watch out for not discuss it for a couple weeks, because I think at some point, probably it would be good to record an episode about some kind of trust to engineering things. But it’s another extreme that doesn’t help to gain trust in engineering things when you’re spending weeks or discussing something. Such stuff.

Miłosz [38:10]: That’s also a good point. So it’s easy to become a purist here as well and just decide we are not moving forward until we decide how exactly this event should be made it’s probably also an extreme you want to avoid

Robert [38:29]: To summarize I think one of the biggest challenges here is decoupling so decoupling is downside here because everything is that decoupled with event-driven architecture that sometimes it’s hard to really change something downstream.

Miłosz [38:46]: Or check how it’s used.

Robert [38:48]: Yeah.

Miłosz [38:49]: Especially if it’s… This is what you mentioned before about this, in contrast with HTTP, where you see the logs and, you know, who called it? If you just use events, it might be more difficult if you don’t have the tooling in place.

Robert [39:06]: Agreed. Okay so the next very common challenge that you can have when you are building event systems especially when you didn’t work with them is at least once deliver and this is, connected again to how most of the message brokers are built but tldr is that, most of the message brokers are offering you at least once deliver semantics so it means that you can receive every message at least once. It’s not possible that you will not receive this message, but you can basically receive it multiple times. This is connected to the problem that we were discussing a bit earlier with Outbox. What will happen if your consumer will lose network connection in the middle of processing the message? They are processing the message, you lose the network connection, somebody cut the fiber cable between data Or DNS.

Miłosz [40:13]: It’s always a DNS.

Robert [40:16]: And what message broker should do? He doesn’t know if you reprocess that, so you stored the data and committed the transaction already, or maybe not yet. And the assumption here is, in most cases, that message broker is assuming that you didn’t process that, and it’s red-delivered.

Miłosz [40:35]: Remember, message brokers are designed so you don’t lose the message, which is actually a good thing.

Robert [40:39]: Yeah, the downside is that you need to keep it in mind. So basically, if messages can be redelivered, you need to handle the situation when it’s delivered. The good thing is that it’s rocket science. So for example, if you’re a creating user, you’re basically checking if the user that you are creating already exists. It’s good to have some identity that you can match and deduplicate by that. Or sometimes you can have some duplication ID, but you need to keep it in mind.

Miłosz [41:10]: Maybe just reiterate why this is an issue. The event that create the user gets duplicated, let’s say it’s executed twice, right? So the first message will create the user, and the second one, or the second delivery of the same message, tries to insert the user into the database and it ends up with some kind of duplicate ID error on the SQL side. And this error will cause the message to go back to the queue, and it effectively blocks your queue.

Robert [41:47]: And it’s probably the best situation, because you can notice this error and fix that just maybe by adding to your queries like on WK do nothing and it’s fine.

Miłosz [41:58]: Yeah, sometimes it can have worse consequences. If, for example, this email will be delivered over and over again, and it’s usually within milliseconds. So if the handler does something like sends a message to the user via notification or text message or something like this, and for some reason it’s not stopped. You can do it thousands of times.

Robert [42:26]: Yeah, I heard horror stories about sending 10 millions of SMSs to one person within 10 minutes.

Miłosz [42:35]: Yeah. So that can be tricky. But more often you have an error. You can detect and fix.

Robert [42:44]: I think there is also one case when it’s worse and it’s hard to detect. So, for example, when you have some counters or something. So, when you have a counter and a message is re-delivered, it’s a pretty subtle error. So, it’s hard to detect because, for example, as we had the question on the chat earlier, how to detect some things and when. When, for example, this counter is not implemented properly, it may be hard to detect that because probably, maybe at the end, someone will notice that I’m, for example, charged much more than I should have.

Miłosz [43:20]: So this is a worse scenario where you don’t know that an issue occurred. You’ll discover it much later when it’s too late.

Robert [43:29]: But for that, actually, we have a pretty good tactic on how to avoid that. So when you have systems who are prone for this kind of issues, you can use, for example, in Watermill, duplication middleware. So basically, this is the middleware that is delivering every message twice. So basically, every message that you receive in your local environment will be delivered twice. And it’s fine when your system is not super heavy and you are able to run tests with that, because you are just 100% sure that everything is processed properly, because this is basically how it should work. You should be able to receive every message twice. Your system should work exactly the same as this message will be delivered.

Miłosz [44:10]: It’s a bit like chaos engineering, where you introduce some issues and purpose to the system.

Robert [44:18]: The downside is that if your operations are heavier, you probably need to have more runners in your CI or more powerful laptops.

Miłosz [44:27]: Or you could just do it in the tests.

Robert [44:31]: Or just for some handlers, this is also the case. But it’s not worth doing it for everything. I mean, it depends on the system, of course, but doing it for everything, it may be probably an overkill. For example, if you’re working with some financial domain, like we worked, for example, we just have this middleware enabled and we’re just handling every message twice. If you’re crazy, you can even do it with your production.

Miłosz [44:57]: Yeah, it should be safe to do it, actually. So the solution are idempotent handlers. So the handlers that you can safely execute twice or more times with the same input, and they won’t fail. In this case, in this example of the user being added, you just add onConflict, do nothing. Or you just somehow ignore the error. That’s a good application. And I know many people when they hear about it the first time are try very hard to figure out another solution. Because the idea of writing handlers like this is… I don’t know, annoying, maybe? They don’t like to do it. But the truth is, it’s not that complicated most of the time, right?

Robert [45:51]: Yeah, so if you’re inserting stuff just like that, it’s a bit harder with counters, because sometimes you need to have some table with the application ID, so you’re storing some message ID or operation ID. they are deduplicating it by that. One scenario when it’s harder is when you are calling some external systems. So for example, when you are charging customers, but in most cases, when you are using some same systems, they should have some idempotency key or something like that. And basically, in this case, you are just sending this idempotency key and it’s deduplicated on the other side. I think it’s showing nicely like, okay, you’re calling some external system that it’s not really even driven, but often this kind of system offers some kind of application because it’s not really event-driven, but it’s a very similar situation.

Miłosz [46:39]: Yeah, payment systems try to work like this most often.

Robert [46:43]: Yeah, because this is the same situation as you have with message brokers. So you have HTTP connection, it was interrupted before the HTTP status code was sent to you, and the system on the other side doesn’t know what to do. Especially when it’s important in cases when you, for example, are charging people. Nobody would like to be charged twice. As long as you are not the person that is charging.

Miłosz [47:08]: You can also see this idea in API design sometimes. If you have an HTTP handler that returns 200 or 201. So you can differentiate either by the response if the entity has been created or just everything is fine, nothing to do. It can also be helpful to handle this.

Robert [47:33]: That’s true.

Miłosz [47:38]: Okay. A bit related challenge here is messages that can’t be processed, so similar to this error. Because of the replicated message, you can have some message that has been published by mistake or can’t be unmartialed because of some issue. Basically it’s broken and whatever you do, it won’t be processed. And most often it will block your queue or this part of it.

Robert [48:15]: And I think it’s showing nicely the trade-off that we are making here with asynchronous or even Jumann architecture. So in normal HTTP endpoints, it would be not an issue. So it will just return 500, and it’s not a problem. But you are losing the resilience. But in this case, well, resilience is important, but you need to do something about this message.

Miłosz [48:37]: Yeah. And most often you want to move it out of the queue, so acknowledge it. But you probably also don’t want to lose it entirely, because you might have some important details or you don’t want to lose this information. So then you can move it to this separate queue, sometimes called the data queue or poison queue, or it just sits and waits for investigation. It sounds simple, but it’s also not trivial sometimes. I mean publishing to a separate queue is very easy, but you need to have some tooling to review those messages and move them back to the main queue if needed. And I think there’s no universal tooling here, depends on the PubSub. Some PubSUps have some admin interface, like RabbitMQ lets you pick messages. That’s one way to do it. But it can be also tricky. You can also provide your own tooling for it. We recently added a CLI like this for Watermill, for the SQL queue. So it can be useful to have some even simple tooling that lets you just review what’s there and has actions like delete or move to the main queue.

Robert [50:04]: It’s also worth mentioning, because we’ve been mentioning Watermill a couple of times, but if you are with us for the first time, you might not know, it’s not our product, it’s an open source library that we are just sharing for free and there is no enterprise additions. Everything that we mentioned is just free in open source. There was one episode, I think two episodes ago, we’ve been sharing more details about Watermill. So worth checking definitely if you are thinking about creating your own open source library, because Watermill is pretty successful in the Go community. So to give you some number, it has more than 8000 stars on GitHub and a lot of projects using that. So again, if you’re thinking about creating your own over-the-source project that is useful for other people, definitely worth checking.

Miłosz [50:55]: Yeah, and it can help you not reinvent the wheel. Like this CLI, for example. It’s much easier to use something that’s ready to use tend to figure out your own… I mean, maybe you can wipe code one, but I wouldn’t like to wipe code anything message-related. It’s kind of a risky area. If you lose a message, it might be permanent.

Robert [51:23]: Fortunately, we have a lot of tests, so it may be a pretty good case for that. So maybe some agent that is running our tests in the background, but it will have a lot of logs to process, so it will be expensive.

Miłosz [51:38]: Gabriel mentioned that it’s important to regularly check deadletter queue and analyze the reason of the issues. One more challenge here to have some alerting in place. You might have the deadletter queue in place, but if you don’t know the messages arrive there.

Robert [51:59]: I would actually say that I don’t fully agree, because I think you shouldn’t be checking your metrics. This metrics should let you know, because it’s just hard to check hundreds of metrics at the end.

Miłosz [52:13]: I mean, alerting, right?

Robert [52:16]: No, I mean, you should have alerting, but it’s making you free from checking the metrics.

Miłosz [52:26]: So, what you don’t agree with?

Robert [52:29]: I would say that you shouldn’t check metrics. I mean, alerting should notify you about… Oh.

Miłosz [52:35]: Yeah, yeah. Yeah, sure. Basically, you need a way to know that there’s something in the letter Q.

Robert [52:42]: Yeah, so it’s more like you shouldn’t be looking at your metrics like, oh, everything looks… Right, because at some point you just may have too much metrics.

Miłosz [52:52]: Ah, okay. I see now. You meant the regular check part. Yeah, yeah, okay. I see.

Robert [52:59]: Because probably someday it may come that, okay, not look on those metrics and something may happen. So it’s better to have something that will notify. But I assume that, yeah, Gabriel had this in mind. But I think it’s important to distinguish because I’ve seen some components that had metrics, but didn’t have alerting. And it’s better than nothing, but still, it’s At some point, you just may miss something.

Miłosz [53:25]: Unless you have someone looking at dashboards.

Robert [53:28]: You’re paying somebody just to, you know, his head is fixed and he’s looking all the day on the metrics.

Miłosz [53:34]: I know one company like that.

Robert [53:37]: What’s this company? Tell me later. All right. But I think that there is one thing that can help us with this deadletter queue and poison queues. Because, for example, I’m personally not that big fan of poison queues, that letter queues, because you need to move these messages back, and it’s a bit problematic sometimes. And one solution here may be proper message ordering. So basically, if you’re doing proper segregation of the messages, So in other words, a bug for one entity will not block all other entities. You don’t need to put this message to Poison Queue because it’s isolated and it will just spin for maybe one person, for one customer, for some small part of the system. But if you’re ordering all the messages in one big queue, actually one spinning event can block the entire system. And this is a pretty bad situation. In this case, you should have some kind of poison queue. But the problem with ordering is that it’s also the science of having good bias. So you shouldn’t have one queue for everything, because throughput will be small.

Robert [54:59]: Because one message will block everything. But from the other side, the good thing is that if you just have one big queue, you don’t need to care about proper order of events. So you just processing each event one by one and you don’t care about ordering. If you have everything unordered, it would be cool because it will not block anything else. But from other side, if you have events like addUser, removeUser and, this will come out of order, your read models that you have may be not right.

Miłosz [55:30]: Or in some special design to make it work.

Robert [55:33]: Yeah.

Miłosz [55:33]: Like versioning or whatever.

Robert [55:37]: Yeah. Again, we also were covering it more in the previous episode, but in my opinion, the most important here is to balance and trying to have as narrow ordering as you can. So not having super big queue, but also not have everything unordered. And it’s hard to give one hint how to do that, but probably you need to find the smallest unit that should be ordered and just have ordering with that. I think it’s also nicely compatible with the aggregate idea from Domain Driven Design. I think probably at some point we’ll also do an episode about that, but basically aggregate pattern from domain-driven design is all about that. How to find the smallest boundary of transactional boundary, but it’s kind of connected with that.

Miłosz [56:33]: One challenge here is also to realize you need ordering. Because if you’re just starting out, you might not think about it at all. And you might just assume the order in which your publish messages is the same as you receive, which can sometimes be the case. Okay, the next topic here is transactions again, about distributed transactions.

Robert [57:03]: Our favorite ones.

Miłosz [57:05]: So, yeah, this thing is like, you know, many people when they start with distributed systems or even different architecture and they split the operations into a few services, maybe they initially use events, but then realize they actually need this to be in a transaction. And the solution is obvious. We need the transaction to span multiple services. So you just have to orchestrate those handlers to work together, and if one fails, because you have to roll back the rest of the pipeline.

Robert [57:49]: Sounds like a great smart plan. What can go wrong?

Miłosz [57:52]: Exactly. If you are a smart engineer, you should be eager to implement this right away. It’s a sign you are a good programmer. But most of the time you should probably reconsider because the complexity will be tragic in the long run. I don’t even know what to start with, but maybe the easiest case to consider is, let’s say you have three services that communicate through events and you create this kind of distributed transaction or saga on top of them. And then the third one fails, so you roll back everything that happened before but then the rollback fails and what do you do now?

Robert [58:43]: I have PTSD already.

Miłosz [58:44]: So you’re stuck with this inconsistency in the system that It can be fixed. You have to go manually to the database or whatever storage you have and fix it somehow.

Robert [58:59]: And later implement compensating transactions for everything. You spend an enormous amount of time on it.

Miłosz [59:07]: So this is something that can be useful in some very specific cases, but most of the time you probably don’t want to deal with it.

Robert [59:15]: Very specific, probably it means super, super complicated system and many, many big teams.

Miłosz [59:22]: Probably it would be easier to just merge your three services into one and run a database transaction on top of it, rather than this thing. Or just embrace eventual consistency, like we said before, and just accept that the transaction doesn’t need to be here. Yeah.

Robert [59:42]: And again, it’s a different case when, for example, it’s some very complicated business project that is orchestrating between, for example, three teams where every team is 20 people. In this case, okay, it totally makes sense, but it’s a complicated process that probably each service in this case is very complex and you have some orchestration at a very high level. But if your team is 5 people, 10 people, and you are doing such a complicated transaction that is distributed, you can maybe ask yourself a question. Maybe it can be just one service. And I know that it’s not sexy to have less microservices, but at least it will save a lot of time on complexity.

Robert [1:00:27]: And, for example, we have a pretty nice exercise about that in our training, Go Event Driven. So, basically, when we’re introducing sagas, we’re not starting with implementing sagas. We’re actually giving exercise with services with saga implementation, and task is to remove saga, just to show how much it can simplify everything when you are removing saga, and how much complexity is added there just because somebody decided to do it as a distributed transaction. So I think it’s important when you’re using this kind of tool like Saga that can create a lot of complexity to actually know how you can hurt yourself with that and after that implementing that. Not the opposite way because I know it’s very easy to start to use that but if you would like to implement it properly it takes a lot of time and it needs to pay back.

Miłosz [1:01:25]: Yeah, so watch out for too complex architecture.

Robert [1:01:33]: We’ve been already covering a bit the topic of testing, so obviously it’s a bit different than testing synchronous endpoints, so we already mentioned that you need to take in, consideration, eventual consistency, you need to keep in mind that messages might be delivered more than once. The good thing is that you already know that you can have, for example, duplicate middleware that is duplicating everything, but it’s also making it a bit harder to test that. So it’s probably more similar to problems that you have on frontend. So you are clicking something and you need to check in some intervals if something happened. In this case, it’s similar. So you’re calling some endpoint, you see some that, for example, accepted, and you need to poll for some data that is happening under the hood. Maybe sometimes you can listen for events, but it’s also listening for some events and waiting after it will happen. Obviously, it’s adding some complexity. Rather, it’s not rocket science. I mean, if you did it once, you can…

Miłosz [1:02:39]: Yeah, experience helps. And also, if you create some helper, test framework or test library that makes it easy to publish and subscribe to events. It’s much more fun to work with.

Robert [1:02:54]: From our experience, component tests are nicely compatible with this approach. We’ll link how you can do component testing. We have an article about that,

Robert [1:03:04]: so we’ll link it in the materials of this episode. Because we know that a lot of people don’t know that something like component tests exists. DLDR, it’s nothing between end-to-end and integration tests, and it’s fitting this case pretty nicely.

Miłosz [1:03:19]: And lets you test the broker in the same way you run it on production, which is important? Sometimes you also don’t have access to the same exact broker you run. That can be challenging. Like if you use cloud-based Pub-Sub, you just can’t run it locally, right? You can use emulators, which often are good enough, but for some edge cases, they might be just not supported. And that’s the hard part here.

Robert [1:03:52]: But maybe you don’t need these cases from the other side.

Miłosz [1:03:55]: Yeah, if not, then it’s fine. But in general, it’s one more moving part into the local and CI environment, so it can be a bit more tricky. On the other hand, message brokers are usually not super heavyweight, maybe except for Kafka, which is often fun to set up in Docker.

Robert [1:04:20]: And you need a lot of memory for Heap or JVM.

Miłosz [1:04:24]: Yeah, but if you use something more lightweight, Postgres maybe, or Gravity Split Lightweight too, Redis is super lightweight. So it doesn’t have to be painful.

Robert [1:04:39]: Also, when you are creating this kind of tests, unfortunately, it’s easier to create a flaky test, because something is not happening immediately. And if you’re polling for something, there is more, bigger chance of flaky tests, basically. And in this case, it’s important to have good observability in place. So the thing that we were discussing at the beginning about tracing, logging is super important, and ensuring that correlation IDs are propagated properly. It’s also worth logging correlation ID in tests. So it’s easier to correlate the test failure to the logs that you have from Docker containers.

Miłosz [1:05:24]: Oh, yeah. Debugging those tests can be fun as well.

Robert [1:05:28]: But if you have logs, and there are good logs and correlation IDs are propagated properly, it’s much, much easier. LiveTip sometimes can also work to just get all those logs, put it to some AI and ask. Sometimes it can really help, basically, when you have a super big wall of text.

Miłosz [1:05:47]: Yeah, we need to integrate it as an agent.

Robert [1:05:51]: Enter Text…

Miłosz [1:05:54]: You don’t see the logs on the output, but just a single summary. It will cost you much.

Robert [1:06:00]: I run me off.

Miłosz [1:06:04]: Coming back to observability, maybe one more thing is we talk about alerting a bit of the data queue, but also a super important thing to monitor and have alerts on is the number of unacknowledged messages or the queue length. Because sometimes you might just quietly keep spinning on one handler and you don’t see the impact anyhow in the system. And then later you look at the logs and, oh my god, there’s 200,000 messages waiting for to be processed.

Robert [1:06:42]: And also Gabriel put the follow-up comment that he hates systems which are like a fridge.

Miłosz [1:06:49]: I need this on a t-shirt. For context, this is about system that you need to open to check it. So it makes a lot of sense. But I like the end result. I hate systems which are like a fridge. I hate systems which are cold inside. Cool. Should we move on to the summary?

Robert [1:07:27]: Yes, but maybe one more thing also.

Miłosz [1:07:31]: There are a

Robert [1:07:34]: Couple of challenges that we’ve covered there. Like every technique, like we mentioned, most of the techniques require some effort and extra cost to implement them. Yeah, and it’s not different for event-driven architecture. It requires some special techniques, and it’s important to use the right tool for the right problem. If you use event-driven architecture for systems that doesn’t require that, it will be just pure over-engineering. You’ll pay all the challenges that we just mentioned for nothing. You’ll just have one more problem in your Sometimes.

Miłosz [1:08:21]: Your system is just synchronous by nature, and that’s fine.

Robert [1:08:25]: Yeah, so your team is not super big, like for example mentioned with sagas. You don’t have super big scalability requirements. You are not integrated with an external system. You don’t have super big traffic spikes. In this case, maybe doing systems synchronously, it will be just enough. And if you’re in doubt, you can always start with Synchronous and migrate to a Synchronous one later. Like we mentioned in the previous episode, if you’re using some kind of abstraction over interactions to your system like clean architecture, it’s much easier because you have some layer that is allowing you to interact with your system and it doesn’t matter if it’s from HTTP request, when it’s a message. It’s making the migration much, much easier. And it’s also a common thing to ask your questions if you are doing some decisions. How you can reverse this decision? If it’s hard, ask yourself, okay, maybe there is some way how I can make this decision easier to be reversed. It’s worth, of course, and it can be helpful with tough decisions.

Miłosz [1:09:36]: Yeah, but if you are not sure, you can just start with sync approach and later migrate.

Robert [1:09:46]: Many traps that we mentioned today. The good thing is that it’s not that many traps for other sites. So I think it’s important to know about those traps and not reinventing the wheel. Because all the challenges that you may have with event-driven or asynchronous architectures, they are solved problems. And obviously you can try to solve them yourself. But I’m quite sure that you will not do it better than how programmers figured it out within the last 50 years.

Miłosz [1:10:21]: So it’s the hardest if you just start out. You don’t know what you don’t know, so you might try some naive approaches or just doing it yourself. You read about event-driven architecture and then you decide, okay, I will use events and I will send them via HTTP or I will have a long-living an HTTP connection between services or something like this. So, sure, this could work, but maybe just use a message broker, which was designed for exactly this thing. And thousands of companies used it on production for 10 years. So it’s very likely it will work out for you, if you will.

Robert [1:11:03]: And if you think you can do it better, I would recommend you to start learning what’s already there. And maybe you can build something better, but I think to do that, you need to actually know the state of art that is currently there. And I think it’s possible to still improve things, but again, I think it’s important to have a very strong base. But it’s only in case when you have some idea how to improve some stuff. When you’re normally working with some product, probably don’t have time for renovating the oil and improving stuff. What’s already there should be more than enough.

Miłosz [1:11:41]: It’s the cool part. That’s the difficult thing to do. It’s exactly like with sagas. You can convince yourself that it’s a good idea to create a distributed transaction framework for your systems. But maybe what you need to do is improve your product a bit to create something people want to buy. Which might be not as interesting in a technical sense. It’s probably the right thing to do for your company.

Robert [1:12:12]: But from another side, I think it’s also kind of the trap that you might think that doing some complex stuff like SAG or whatever may be interesting. Maybe at the beginning, but later you’ll need to maintain that. And I’m pretty sure, don’t ask me from where, from experience, that later you’ll need to maintain that. And it will be not interesting.

Miłosz [1:12:32]: Yeah, you know what, you need to build a framework and then leave the company.

Robert [1:12:37]: And it will be no longer a problem.

Miłosz [1:12:39]: Problem solved.

Robert [1:12:40]: Unfortunately, it’s often the case. But trust me, it’s much more fun to work with a system that is fairly simple, but you can iterate quickly and add features quickly there. And again, from our experience, using some boring, already well-known techniques and tools, gives a lot of fun at the end, because maintenance of that is much, much easier, and you can also do a lot of stuff.

Miłosz [1:13:08]: Moving fast is fun.

Robert [1:13:11]: For sure. Yeah, seeing how people are using your product and are happy about that, it’s also pretty cool. And I know that it’s not super technical from one side, but trust me, try this and it’s much.

Miłosz [1:13:22]: Much high-stay.

Robert [1:13:23]: And I think it’s also nice later when you’re looking for a job and you’re going to interview, and the interviewer will see that, okay, you’re a pragmatic person because you were not doing CV-driven development, but instead you were using the right tool for the right problem. So it’s, I think, often a really nice skill, and when we’re interviewing people, it was a big bonus point to see that. Okay, even if a person was just using very simple tech, but in proper context, and have big knowledge of the simple tools, it was always great to see it.

Miłosz [1:13:59]: There’s no silver bullet.

Robert [1:14:02]: Yeah. Sure. So, probably at the end we should say, is it worth using Evangelion architecture? And, well, unfortunately, it’s not that easy. So, as I said, there is no silver bullet, and there are many parameters for that. So, we believe that it’s most important to give you a lot of data points so you can decide on your own when using Evangelion architecture is good for you or not, to be prepared, the price that you need to pay, and to see if it’s fitting properly and if it will pay back, basically.

Miłosz [1:14:42]: Yeah, there are probably no shortcuts here, no easy answers, since you consider what you work with. You should have no all those challenges before you start, so you are prepared for things like we mentioned about at least once delivery or eventual consistency. It’s just a bit different mindsets than working with Synchronous approach. So if you have the basic theory in mind, and you spend some time working with the projects in a Synchronous way, you will be ready for handling it.

Robert [1:15:24]: I think the only silver bullet here is the fact that it’s not like it never works or it always works. I think it’s important to know what tools you have So not be a worker that only knows how to use Hammer, because this is not the best worker. So you should just know what tools are available and be prepared to use them in a proper context. Because using Synchronous Architecture Everywhere, it will not work. Using Asynchronous Architecture Everywhere, it will also not work. It’s just about finding the right tools for the right problem, because Opposite is not the best.

Miłosz [1:16:07]: Yeah, so if you want to learn more about Event Driven architecture in general, or in Go specifically, we have our training of Event Driven on sale for one more week. So this is a place where you learn hands-on with exercises, and we do it for the first time in a cohort-based way, so everyone learns together. You can still join us for one more week and then we will start an intensive month of learning together.

Robert [1:16:40]: Next one probably in half year or so. So, if you’re just listening to that and you’re listening for this now, we are super lucky. If you’re listening for this episode later, sorry. You can sign up to the waitlist.

Miłosz [1:16:58]: Okay. Before we move to the Q&A, it’s probably time you hit that subscribe button and leave a comment or a review, if you listen to the Minna Podcast application. And as Robert mentioned, you can subscribe to our newsletter on our blog, and we will let you know about the next episodes. Actually, we don’t have a topic for the next episode this time.

Robert [1:17:21]: It’s a secret.

Miłosz [1:17:22]: It’s a surprise.

Robert [1:17:24]: Yeah, but the one thing that we already know is that we’ll be changing the form of this live podcast a bit. So basically, we’ll do two more live episodes and later we’ll switch to sometimes doing live episodes. Most of the time, it will be pre-recorded because of a couple of technical challenges of doing it live. Like some workers that are super loud. Fortunately, our mics are pretty good and noise-canceling, but it’s super loud here.

Miłosz [1:17:57]: It’s 15 minutes before the start. They started drilling, so it’s super fun. We also want to make the sessions a bit more interactive. If you do live sessions, we will try to figure out something more interesting for you to follow.

Robert [1:18:09]: I think we’ll be running some Q&A sessions, definitely, between pre-recorded sessions. And we’ve also seen that most of you are listening to this after recording, so we’ll try to balance it a bit. And I think at the end also the content will be a bit better, because trust us, it’s not easiest to go over the scenario of the episode live, because we need to just keep flowing. But again, we’ll still have live sessions, so no worries, it will be just a slight change. But again, we’ll have two more live episodes, so don’t forget to join us.

Miłosz [1:18:44]: And the short summer break and then we’re back.

Robert [1:18:50]: Correct. Alright, so time for Q&A, I think.

Miłosz [1:18:58]: I think we discussed most of the comments before because they were relevant to what we discussed at the moment. There’s a recent question from Pedro. It’s about starting a monolith and breaking down into smaller surfaces and then adding events.

Robert [1:19:23]: AKA monolith first.

Miłosz [1:19:25]: And this is something we did once, more than once. Once specifically. So we had this project when we started with a quite small team, but then there was a perspective that it will be much bigger in the future. didn’t happen.

Robert [1:19:46]: At least we didn’t lose time for doing microservices.

Miłosz [1:19:49]: We wanted to be ready to scale but we also didn’t want to create distributed transactions at the beginning. So we started with our monoleaf and quite simple setup where we have modules that are completely separated and connected with code, basically, with interfaces between each other. So you can think like, you know, imagine microservices that expose HTTP API. So it’s the same idea, but instead you have modules that expose interfaces in code. And that’s, as you can see, that’s pretty much the same thing, just different transport. In this case, there is no transport, because we don’t need it. But we try to do it in way that, if needed, we could take any module out and it would be quite simple to deploy separately.

Robert [1:20:47]: But from the other side also, if your team is not that big, so like 10-15 peoples, maybe 20, it shouldn’t be that big problem, so you should be able to split it from anything. Okay, it depends on time frame basically.

Miłosz [1:21:04]: But we like to think of Monolith versus Microservices question as deployment, not about project structure.

Robert [1:21:15]: And also about some kind of social concept or concept of team. So basically, when you need microservices, you should probably need them because of how big is your team. So basically, you have one project and you have 50 people, for example, committing to one repository and deploying it.

Miłosz [1:21:34]: Yeah, maybe they block each other, maybe the CA becomes complicated.

Robert [1:21:39]: Somebody is introducing some flaky tests and 50 people are not able to deploy for an entire day. So yeah, this is a place where you need to split. But if your team is five, it’s not an issue. But anyway, if you’d like to see the implementation of the project that Miłosz mentioned, how we did it, we have blog posts about that. So we’ll include it in the episode materials tomorrow. And also, maybe one interesting thing. We also used to do the opposite one. We had microservices, and we did the microservices. We actually connected multiple microservices into one monolith, because it was just too granular, and it was too complicated to communicate between them. And having one service with everything, so similar situation to what we had with Saga, it would just make everything much easier.

Miłosz [1:22:34]: And Gabriel mentioned that onion architecture can help when you use the DD from the one. So onion or clean or hexagonal or whatever, layered architecture in general can help. For bigger projects especially. Also not silver bullets. A separate episode about this. So in this setup I mentioned this is exactly this idea that following hexagonal terminology let’s say the modules communicate via adapters and that’s it it’s very similar to communication with any RPC just with non-network connection and

Robert [1:23:18]: Also worth mentioning you don’t need to use DDD for on your architecture, you don’t need to always use DomainLayer. And it’s also totally fine. Because also often people join this and sometimes it’s creating some monster and people say, oh, architecture is overcomplicating. But often it’s because of that, because you don’t need to always introduce DomainLayer.

Miłosz [1:23:42]: Yeah, and each module can have a separate structure. One can use CleanArch, one can use DD and some can be one file.

Robert [1:23:51]: Or about that in the episode about clean architecture.

Miłosz [1:23:56]: Yeah, so Gabriel mentioned you can move faster with M&Oleaf. If there’s a small team,

Robert [1:24:02]: Then definitely.

Miłosz [1:24:03]: If there’s a small team, a single developer especially, if you’re a single developer and you have microservices, something is wrong.

Robert [1:24:12]: It’s not your fun project after hours.

Miłosz [1:24:16]: Yeah. We love, but we also worked with teams where there were more microservices than people on the team. And it’s not fun at all.

Robert [1:24:29]: Probably it was fun at the beginning, but later it was the opposite.

Miłosz [1:24:35]: At the end, DDD is the best way to go, says Pedro. It depends, as always.

Robert [1:24:43]: I’d say that probably depends which part of Domain-Driven Design, because many people are actually kind of not splitting Domain-Driven Design in two important parts. Domain-Driven Design are strategic and tactical patterns. Most people are talking about the tactical one that are in the code, But there is also a very important part of Domain Driven Design named strategic patterns. And actually, it’s totally not connected to the code. So it’s more about gathering requirements, talking with business, ensuring that the problem that you are solving is the right problem. And probably you should do it always, even if you are not naming it Domain Driven Design. Domain Driven Design is just giving you some tools for that. So if you are talking about strategic patterns, totally, you should do that, even if you are not naming it Domain Driven Design. What tactical patterns? Maybe. Sometimes.

Miłosz [1:25:36]: It all depends on the service or the module. What does it do? There’s a service that lets you upload your avatar. You probably don’t need to involve your stakeholders to discuss what file formats are supported or something like that.

Robert [1:25:53]: And have a separate service for that. We’ve seen it once.

Miłosz [1:25:57]: So, yeah. Once again, balance is important. And if you have this modular monolith with separate modules that are independent, it helps, because you can have one module that is super technical and you don’t care about the design here, and another that’s pure business rules and you spend a lot of time on design. So, modularization is the way to go. Probably not in the form of microservices, but some kind of separation is great to have.

Robert [1:26:39]: It’s also, in general, starting with microservices, there are cases, but probably there are cases when you are starting a pretty big development team, or for some reason you know that your development team will be big. And it’s not like yeah, yeah, totally, our, theme will be big and product will be super scalable every startup thinks this and probably five percent is really ending up there so and.

Miłosz [1:27:07]: It’s also not that easy to design a system that would be

Robert [1:27:11]: And be profitable 90 of them are spending time on setting up kubernetes and burning all the money and they are failing later five other five percent is failing because product is bad, and other 5% is lucky. And didn’t spend most of the time on setting up Kubernetes and Istio.

Miłosz [1:27:33]: You mentioned this in the first episode as well. It’s very hard to design a system that will be ready for all the future challenges. It’s better to evolve over time. Because you just can’t foresee how things will change.

Robert [1:27:48]: Easier to say than do, but again, check first episode. We have a couple nice ideas how to do there.

Miłosz [1:27:56]: I think there are no more questions. Pedro says we are doing a great job. Thank you very much.

Robert [1:28:03]: We are doing our best.

Miłosz [1:28:04]: Thanks for joining us. Thank you for all the questions. We will see you in the next episode about a secret topic.

Robert [1:28:14]: Exactly. In two weeks. So we’ll let you know in our newsletter. Just a reminder so go event driven you’ll be just able to join for another week next time or I’ll be healthier or so.

Miłosz [1:28:28]: If you are listening later you can join the waitlist

Robert [1:28:33]: Or go in an evening if you don’t go don’t know go yet because maybe a lot of you doesn’t know go and yeah we’ll.

Miłosz [1:28:42]: Leave links in the description yep oh right Thank you everyone for joining us for the questions. See you in two weeks.

Robert [1:28:52]: Thank you, Milosz. Thank you, everyone.

Miłosz [1:28:54]: Bye.

Synchronous vs Asynchronous Architecture

Wed, 28 May 2025 16:00:00 +0000

Quick takeaways

Start with synchronous architecture by default - it’s simpler to understand, debug, and maintain for most use cases
Async architecture improves scalability and resilience - message queues and events help handle traffic spikes and failures
Design matters more than the technology choice - tight coupling creates the same problems in both sync and async approaches
Consider team experience - async architecture require more experienced teams and better tooling to handle new challenges
Adjust as your system grows - external APIs, heavy operations, or the need to handle failures gracefully are good use cases
Hybrid approach - use both sync and async where they fit best, rather than forcing one over the other

Introduction

In this episode, we discuss when to choose synchronous versus asynchronous architecture for backend systems.

We talk about the trade-offs between simple, predictable sync communication and the complexity but resilience of async approaches using message queues and event-driven architecture.

Instead of picking one approach over another, we focus on understanding when each makes sense and how to avoid common pitfalls like distributed monoliths and over-engineering.

Show Notes

Go Event-Driven training
Watermill - our open-source Go library for working with message streams
Event Storming - a design technique with a great unfinished ebook by Alberto Brandolini
CQRS (Command Query Responsibility Segregation) - a pattern that works well with both sync and async approaches
- Killing the legacy and other CQRS stories - Miłosz’s talk on this topic
Our CQRS article and Server-Sent Events post
Message brokers mentioned: RabbitMQ, Kafka, NATS, Google Cloud Pub/Sub
Event schemas: Protobuf, Avro, CloudEvents
Event Sourcing - a pattern mentioned in context of recreating state from events
Clean Architecture/Hexagonal Architecture - architectural patterns mentioned for making sync/async migration easier

Quotes

What’s the key difference here is the fact that when you’re publishing some message, you basically don’t care how it will be processed, by whom it will be processed. And it’s nice because it’s decoupling you from the consumers.

Robert

The bigger the system, the worse it becomes. For example, if many services depend on this one, which is a bottleneck, it can kill your entire platform.

Miłosz

Decoupling is not always good. It’s similar like with Don’t Repeat Yourself. Often, don’t repeat yourself may be a good idea, but not always. If you go into extreme, it may be problematic.

Robert

The worst case here is when you ignore the issue. You assume the database commit never fails. And it might be true most of the time, but usually if something can happen, it will happen on production sooner or later.

Miłosz

Event is information about the fact that happened. So it’s not something that will happen, may happen. No, it’s information about something that happened and it cannot fail.

Robert

It’s just counterintuitive, because you think that splitting things makes things decoupled. But it’s not really the case. It depends on how you communicate, really.

Miłosz

Timestamps

Transcript

Miłosz [0:00]: Should we make this async? It’s one of the questions that seems simple,

Miłosz [0:04]: but can make a huge difference in the long run. If you get it wrong, you’ll spend months fixing the mess. How to decide? You should not just guess. I am Iwos.

Robert [0:13]: And I’m Robert. And this is No Silver Bullet Live podcast, where we discuss mindful backend engineering. We spent almost 20 years working together across different projects and teams. Learn that following advice like always do X or never do Y doesn’t work and can limit your growth. In this show, we share multiple perspectives that will help you to make smart choices and grow you into principal engineer level.

Miłosz [0:38]: And if you have any follow-up questions, you can leave them in the chat. We can pick them up during the discussion if they are relevant. Otherwise, we will have a Q&A session at the end. So we will go through all the questions.

Robert [0:54]: And to be on the same page, today we’ll discuss async in context of architecture, not async in context of your code. So we’ll not discuss threads, we’ll not discuss goroutines, we’re not discussing code level, async, we’ll discuss the higher level architecture, so related to pubs, queues, and similar systems.

Miłosz [1:21]: Yeah, maybe the context would be we have two services, microservices or programs in general, two or more, and we want them to talk to each other over the network in some way. That’s what we want to focus on today. And usually, it works like one of the services asks for something, makes a request, and then waits for the result. And you have to handle the error in some way, if it happens, or basically check the status code, which is the main difference between sync and async.

Miłosz [2:08]: And yeah maybe one specific case is where you fire and forget so you make the request and then you don’t care about what happens later in the other service but I think we can agree not to talk about this case today because it’s pretty rare but you don’t care about what happened on the other end if you lose the status code, the response, you don’t know if we can continue, or should you let the user know what happened? Let’s skip this one for now.

Robert [2:55]: If you are talking about synchronous architecture, as contrast, in most cases we are talking about our gRPC, HTTP architecture, so in other words, you’re calling from one place, you’re calling one service, and within the request context, some operation is done. So maybe some database, something is written to the database, maybe some transaction is done, maybe some money is moved, or something like that, so basically…

Miłosz [3:27]: Yeah, transferred to another account. Something happens in the other service.

Robert [3:35]: Yeah, and basically you know that if this request returned you the some 200-like HTTP code or gRPC status code that it succeeded, you know that it succeeded. And it’s nice because it’s the typical architecture that we are building. So if you are learning how to code, usually you are using synchronous architecture. You are not using event-driven architecture or as a broker, because it’s probably something more complex for somebody who’s learning. I guess, Milos, when you were learning how to code, you started with building some synchronous APIs, probably.

Miłosz [4:16]: Yeah, I started with no APIs at the very beginning.

Robert [4:20]: When you were learning programming, there were no APIs yet.

Miłosz [4:24]: Well, I didn’t learn microservices first, but it’s for sure the first method of communication you can see while learning. It’s probably the most natural. That’s one of the upsides of using synchronous architecture, that it’s quite predictable in the flow. It’s kind of like the method call goes over the network. and when you read the code, you don’t really need to know what happens underneath. I think there are some frameworks that try to do it for you, even that you can distinguish between… Like the RPC calls basically look like normal function calls. That’s a separate topic. But the point is, that’s one good thing about SYNC. That it’s pretty easy to understand what’s going on in your function.

Robert [5:25]: And it’s natural for you, so the entry point for that is much smaller.

Miłosz [5:30]: If an error happens, or an exception is thrown, or whatever, you know it will exit the function, just like any other function call. That’s one of the pros.

Robert [5:44]: If you were working as a software engineer for longer, As usual, you may prefer simple approaches, because simple approach doesn’t require any additional complexity. But, well, as usual, it doesn’t cover all the possible use cases. So, unfortunately, there are some cases when the simple approach of synchronous communication is not enough. So, for example, you are calling some external service, it can be other team, it can be other company, And for some reason this is working slowly or even doesn’t work. It’s not the best situation, but this is where we are often. So we need to call some other team services or some external services. Doing it synchronously sometimes may be hard. I remember when we were working with some, for example, invoicing software that was outsourced and it was sometimes working, sometimes doesn’t work, but we were not able to change that because it was some contract and we were being obligated to use that.

Miłosz [6:47]: It might be even an external API completely out of your company. There’s some third-party service you use, and you need to use it, but you have to deal with the API, whatever it takes.

Robert [7:02]: Yeah, and sometimes, for example, also some real-world examples, sometimes you may be integrated with some external marketing systems. And for some unknown reason that I don’t know why, but often those systems are not that stable. And sometimes it’s part of customer registration flow, but from some other side, you need to send some data when you’re registering customer. And, well, you cannot risk that basically it will break customer registration

Robert [7:30]: flow because it’s part of the critical flow. So it’s also pretty problematic and in synchronous way, it will be hard to handle.

Miłosz [7:41]: Yeah, it’s one of the limits. I think it’s often a question people ask us about designing the architecture, so They recognize that they have a database transaction that does something like saves the user, let’s say, in the database. And does a few other things. And they also need to call this, let’s say, marketing software via API. And they are confused how to model it, because the API call, should it be inside the transaction, should it be outside the transaction, how to do it. And basically, there’s no good way to handle this in the synchronous approach way.

Robert [8:24]: Yeah, you can choose basically what’s less important. For example, okay, we’ve failed to send some marketing information, integration to some marketing system. Well, or maybe we should, it’s something critical, and maybe we should interrupt our registration function flow. It depends, unfortunately. But, well, when you are doing it in a synchronous way, you need to do those trade-offs.

Miłosz [8:48]: The worst case here is when you ignore the issue. Like, you assume, okay, but database commit never fails.

Robert [8:56]: Or the system will always work.

Miłosz [8:58]: Yeah, and it might be true most of the time, but usually if something can happen, it will happen on production sooner or later. And what I really don’t like about ignoring this challenge is that you end up with inconsistency in the system. And fixing this is super hard. First of all, you need to figure out if you have inconsistency. And if you have no proper monitoring in place, you might never know. You can see it weeks later or months later.

Robert [9:37]: Or even worse, let’s imagine that something on the other side, we assume that it’s stable, it’s always working, this external system of other company, and you have some big promotion, big marketing, and you have hundreds of people trying to register, and the system is down. And I guess that a lot of people may be pretty upset because somebody spent some horrible amount of money on marketing, preparing that. And because of this assumption, it’s not working. And often, you know, it’s important integration that if you disable it, you’ll find out that, okay, it’s hard to track what’s happening with customers and it’s a super big problem for everybody.

Miłosz [10:17]: Yeah, so it can easily become a bottleneck.

Robert [10:20]: Yeah, and if you are talking about bottlenecks, the other problem that often you have with synchronous architecture is problem with scalability. So often you have some operations that are heavy and they are working fine when you don’t have super big traffic, but hopefully your product that you’re working on at some point may have some bigger, spikes of traffic, and you may have more customers. And, well, if it’s not designed in a proper way, and it’s just done everything asynchronously, at some point, people will just start to see errors, because the operations that they are trying to do cannot be done, because you just don’t have enough resources. And if you’re doing it synchronously, you cannot do much about that, if it’s a critical part of the functionality.

Miłosz [11:08]: And the bigger the system, the worse it becomes. For example, if many services depend on this one, which is a bottleneck, it can kill your entire platform.

Robert [11:24]: I’ve seen that many systems are trying to do some workarounds for that, like runjobs or or even worse, database triggers. It’s some kind of workaround for that, but from our experience, it doesn’t scale at the end. Not only in terms of resources, but also in terms of maintenance. It’s a mess at some point of complexity, basically.

Miłosz [11:56]: So, the other approach is asynchronous architecture, and we don’t talk about the code level. So, stuff like callbacks, or promises, or director model, whatever you use in your code, it’s out of scope for now. Coming back to the context of two services that talk to each other, I would say that the goal of a synchronous architecture in this context is that we want to execute this in the background, this request, whatever it is, but also we don’t want to lose the information about the result. And even if the result is empty, maybe there is no payload. But the status message or the fact if an error happened or not is an important fact. So we want to handle it sometime in the future, but we don’t want to wait for it to complete in the current function.

Robert [13:00]: And again, some poor man’s approach may be putting it to the table and doing some batch jobs, but while it can work, it’s not a perfect approach, because Because at some point, if you have more jobs to process, you may end up with just taking more time than the entire day have. So paralyzing of that may be a challenge. And it’s often also delaying when it’s processed, basically. Yeah.

Miłosz [13:33]: So Gabriel mentioned in the chat that it’s easier to debug. I think this was about a synchronous approach. Yeah, that’s a super important point. If you have a classical function that goes one after the other of the steps, it’s quite easy to understand what’s wrong and debug. With icing it’s a bit more complicated when you get into that. So, what solutions do we have in terms of the architecture? I think maybe what many people may be familiar with is task queues, which is usually a simple FIFO first in first out queue of tasks that some worker processes pick up from. And it’s a pretty easy solution and can work well for simple use cases when you just want something to be done in the background and forget about it. The downside might be that if you have a more complex setup, it’s probably not enough to handle it well, but it can work.

Robert [14:50]: The two cases that I would have in mind when queues may be not helpful, compared to some event-driven architecture, is a case when you have more teams, so it doesn’t help you with organizational scalability. So basically, you have one queue, and it’s kind of similar to sharing database. So if multiple teams are sending messages to queue, and one message is spinning, it’s blocking other teams. So it’s not best experience. And the second thing is that if it’s order queue, even if you have multiple queues and you have some high priority or low priority queues, it’s also pretty limited in terms of performance.

Miłosz [15:36]: Yeah, but for a simpler architecture, it might be a pretty good solution. For sure, better than this, hoping it won’t break in a synchronous way. But a step forward in this are messages in general, and often using a message broker in the middle, like or a PubSub. So this is a similar approach, but instead of having workers, pulling tasks and doing one thing. We connect the services using messages. And there’s usually many ways you can build this topology, like who receives the message, and it depends also on the infrastructure you use. You might be familiar with message brokers like RobidMQ or Kafka, or the cloud-based ones, and they all have a bit different configuration. But the point is, it allows you to communicate between processes using those messages.

Robert [16:45]: Yeah, and what’s the key difference here is the fact that when you’re publishing some message, you basically don’t care how it will be processed, by whom it will be processed. Basically, emit some thing, usually event, and this event can be processed, maybe not processed, it may be spinning for a while, you don’t care. And it’s nice because it’s decoupling you from the consumers.

Robert [17:13]: And it’s adding the thing that I mentioned in terms of queues that it’s not handled by queues. So it’s creating some kind of way that helps with scaling multiple teams. So, if you are decoupled from customers of your teams, basically tens of, hundreds of teams can listen to your events and you don’t care if it will fail or not.

Miłosz [17:40]: So, this is if you use events, right? So, an event is kind of a message and it states a fact that happened already. This is a big, I think, more like a mindset shift rather than a technical thing. Because you can easily use messages, but if you send, for example, requests like in RPC as other messages, doesn’t give you the same decoupling benefits.

Robert [18:16]: Yeah, I think it’s nicely how it works out of the box with events. I mean, how it fits this model nicely. Events is information about the fact that happened. So it’s not something that will happen, maybe happen. No, it’s information about something happened and it cannot fail because it already happened. So you can’t say, oh no no it didn’t happen it happened and it’s nice because it’s in some way immutable so everybody can listen to that they cannot say it didn’t happen yeah and.

Miłosz [18:48]: They can’t not agree yeah it i think it simplifies the design often right because let’s say you have this process after i don’t know someone placing an order and instead of this one process figuring out tell other things that need to happen in the entire system, like send a receipt, send an email, notify the sales team, whatever.

Miłosz [19:16]: Can we manage your guess like that? You can just send a single order place event from this system, and then other systems react to this.

Robert [19:27]: We’ll go to anti-patterns a bit later, but I think one of very often seen anti-patterns is passive aggressive events on the pattern. So usually when you are creating an event, it should be verb in past tense. So, order placed, but it shouldn’t be event like order should be placed. Because in this case, it’s more like a command, and commands are more fitting the queue system. So you’re sending this queue, and one consumer is subscribing to that, and maybe order will be placed. But maybe it will be not placed, because you don’t have enough items in an inventory, for example.

Miłosz [20:11]: It’s a funny thing, because it’s mostly about naming things. There’s nothing much different about an event than a regular message. The difference is just the payload and how you call it. But if you pick the right names, the design becomes much easier to follow.

Robert [20:29]: I think it’s interesting because many people are looking at that like, ah, naming doesn’t matter, whatever, name it XYZ and it will be fine. But I will not agree after seeing how it can help, or it can not help if it’s not properly designed.

Miłosz [20:46]: Yeah, it’s about mental load, pretty much. How easy it is to understand the design. And then it’s also easier to debug issues or extend it.

Robert [20:57]: And I remember that some people asked about that in Watermill, so the library that we’re maintaining in Go. And what’s the difference between command and event? because basically API is very similar, but there are some really small reasons. And yeah, basically you can send command as event, but again, it’s just practical to follow this because it will be easier later.

Miłosz [21:25]: Maybe let’s come back a bit to messages in general, not just events. And I think one thing that’s pretty different in how you approach asynchronous way is error handling.

Robert [21:40]: Yeah, and I think it’s a cool thing that you get out of the box, we can say, with event-driven or many asynchronous approaches that is basic or un-message broker. Because if you are doing operations in the background, for example, instead of emitting an event or sending a message, when you are just doing it within a process, you never know if this process will not die. Because, for example, Kubernetes will say, you should be that. You never know. It can happen. And I know that in some smaller systems that are not mission critical, it doesn’t matter. You lose some operation, maybe somebody from operations team will be upset, but that’s it. But if you’re building some systems when this consistency really matters, and I would say that in most cases it is, if you are not storing things that you would like to process in Message Broker, in some cases it will just happen that your process will die and some operation will be not processed. And when you’re sending this event or message to your message broker, it’s up to the message broker to ensure that it’s processed. And only if you will confirm that it was processed, the message will be deleted.

Robert [23:07]: And it’s especially useful when you have some temporal outage. So imagine that your some external API was down for a couple minutes, your database was down for a couple minutes. In this case, when you are subscribing to the messages in MessageBroker, it’s not an issue. So messages will accumulate over the time, and it will just auto-resolve after the time. And I think it’s pretty cool when we work with some systems, when we’ve seen that, okay, some outages of external systems was happening, but it was not a big deal. So we sometimes receive non-critical alert like, oh, we have some messages that are stuck, but it was just stuck for a couple of minutes and later you can see how the chart is going down. Always pretty possible to see that.

Miłosz [23:52]: It can also be a bug in your code. You can mess something up and then the direct requests fail. So in synchronous approach all services calling your service this broken service don’t have much choice what to do like best they can do is probably return the error up the stack to the previous caller and then up to the user show on the UI right but if you use messages for this then yeah this this thing won’t be done in the moment, but then you can deploy a fix for the code issue, and as I said, all the messages can be processed again. Auto-healing is pretty cool to have. I think it’s also another factor that simplifies the mental model, because you don’t need to care about error handling in each scenario, how to do it, how to make this nice for the user to report or… Get over with. Instead, you know that you can just retry, and most of the time it will work fine.

Robert [25:14]: And another similar situation is when you are receiving bigger traffic because of many reasons. So many people went to your website or you have some attacks sometimes. It’s also kind of similar situation because normally you would not be able to process all those requests, but if you just receive a lot of messages, you can accumulate them and process them in order that you can have. And it’s also helping you to scale it with that, because if you have some way of autoscaling your application, you can basically configure it in multiple ways. But when this autoscaler will detect that, I know you are using a lot of CPUs or there are more messages to be processed, you can just autoscale more, process those messages and downscale your application back. And it’s kind of almost out of the box, we can say.

Miłosz [26:08]: Yeah, not always ideal. It depends if you need this thing done right away or not. But if you can wait a while, it’s much simpler than scaling up to handle all incoming requests.

Robert [26:22]: Yeah, and again, I think many people are mentioning event-driven or in-general asynchronous architectures in terms of technical and performance scalability, but we have also the perspective of organizational scalability. So you can basically integrate more teams over events, and thanks to the decoupling, you can add more teams and they can just work more independently.

Miłosz [26:53]: If you design them well that’s a prerequisite obviously because there are also lots of downsides of AC Cloud architecture maybe not downsides but challenges you need to be aware of right We mentioned a couple of times already that the design is important, so that’s one thing. It’s more complex in general and you need to understand the rules well. Ideally, the entire team would be familiar with what the stuff works. So the more experienced team, the better for this kind of architecture.

Robert [27:35]: Yeah, but as usual, so you have more complex problems to solve, it’s better to have more experienced team to handle that.

Miłosz [27:44]: Yeah, because for some platforms you can just use only a synchronous approach and be done with it. So at some point you just need to move to at least parts of async. And there’s lots of new issues you need to deal with, like eventual consistency or phrase conditions or debugging, for example. Local development can be a bit more complex. You need new infrastructure, the message broker, which also is something that you need to be familiar with and you need to maintain.

Robert [28:25]: Yeah, also testing is a bit more challenging because it’s no longer working like you’re doing HTTP requests and after request is done, you know that operation is done. No longer. You need to do some way of polling. So if you work with frontend a bit, it will be familiar for you because like in frontend, things are not that synchronous and sometimes you need to just poll for some operation that you expect that will happen and it will, eventually happen, but maybe it will not.

Miłosz [28:59]: Yes, but speaking of front-end, the UI also needs to be adjusted sometimes.

Robert [29:03]: Yeah, I think even Gabriel mentioned that it can be more difficult to handle errors in UI in async system. So you need to either use WebSocket or ServerSendEvents to push async errors, like in case of failed sync operation. Yeah, we totally agree that this is the challenging part.

Miłosz [29:25]: Exactly, yeah. This is it. So with synchronous approach, you can just return the error of the stack until the UI and then just show a big 500 error and something like try again or contact network support. With async approaches, the error is there somewhere in the system and the user doesn’t know if something happened or not, if it was successful or not. Let’s say this extra setup to show the status as recurrence.

Robert [30:04]: In other words, trade-offs. It really depends on basically what the system is doing under the hood. Because again, if you’re storing stuff to the database and it’s your database and it’s stable, So it probably wouldn’t be a problem. But if you’re doing some stuff in the background, it’s starting to be tricky. And it’s also easy to create a poor UX design that will be just misleading for

Robert [30:34]: people that are using that.

Miłosz [30:36]: Which is also counterintuitive because it seems like something specific to backend. And here you need to consider it in the entire flow, even in the UI.

Robert [30:47]: Yeah, and again, the thing that Gabriel mentioned, we are also big advocates of servers and events. If you are interested how to implement it properly, go. So we have a pretty nice article about that, that Miros wrote on our blog with super nice example. So we’ll link it in the podcast materials if you are interested.

Miłosz [31:13]: Any other downsides? what about observability?

Robert [31:18]: Yeah, it’s also a bit tough topic because when you are building your REST API, you can basically monitor your HTTP load balancer.

Miłosz [31:31]: You can open the console in the browser and most of the time you will see some error when it happened.

Robert [31:39]: Yeah, or just check your load balancer and you have 500s on your load balancer. You’ll see that something is wrong, we need to do about that. With an asynchronous approach, it’s not that easy because probably you should monitor how long is the queue, how long, what is the age of oldest message. It’s also not simple, because it depends on queue or topic or subscription. Sometimes having event dot is there or queue for five minutes is fine. But sometimes, if it’s delay 30 seconds, it’s maybe a problem. Sometimes having 10 messages spinning is fine, sometimes it may be a critical issue. So I’m not saying that it’s not possible to implement, but it requires more thinking and it requires more of what you said at the beginning, that you need to spend a lot of time on designing and understanding. Okay, maybe not a lot of time, but basically you need to have it in the back of your head.

Miłosz [32:49]: Have some tooling in place, like tracing at least, to see what’s going on, to understand where the issue is.

Robert [32:57]: Yeah, and some metrics that I know that not every team has, but in this kind of architecture, you should have that. And I know that also not all message brokers are exposing the metrics about, for example, oldest message or how many of the messages are spinning. So sometimes, unfortunately, you need to write it by hand and export it by hand. So, obviously, it’s some cost. For the other side, it’s also not an unsolved problem. I mean, it’s a solved problem if you find a well-known solution for that. It’s not rocket science at the end of the day. But it’s just important to keep it in mind, and keep in mind that every operation is also different.

Robert [33:45]: And there’s also one challenge that I think not a lot of people are mentioning, because it may be a bit counter-intuitive, but it’s a problem of decoupling. And why it’s interesting? Because usually we’re talking about decoupling in context of good things, but also decoupling may be challenging when it’s, let’s say, extreme decoupling. Like with events. So let’s imagine that you’re emitting some event, like order was placed, And you may have no idea, actually, who is listening to this event. So when you have your API, you basically see that somebody is calling this API. You probably can have some tracing and by that correlate from where it comes. But if somebody is listening to your event and the service doesn’t have good observability, you have no way to understand who is doing that. In worst case, even you have no idea if anybody is doing that. And it’s starting to be problematic when you need to change something. We’ll get into that, but it may be challenging.

Robert [34:46]: Especially if you have a bigger system with more teams, it may take months to deprecate some events.

Miłosz [34:53]: Yeah, and you want to get rid of some code that produces some old events, but you don’t know if a team is using it. But it can be in larger organizations that can be challenging.

Robert [35:05]: So yeah, in other words, decoupling is not always good. But I think it’s also similar like with Don’t Repeat Yourself. Often, don’t repeat yourself may be a good idea, but not always. And it’s a similar situation, because don’t repeat yourself. It’s trying to kill the coupling, but also, if you go into extreme, it may be problematic. It’s, I think, interesting how you can apply some patterns and anti-patterns from level of code to level of architecture in some way.

Miłosz [35:41]: Yeah, because it may promote teams that are very focused on what they do. They just publish some events and then they don’t care about whoever is doing with those events. Yeah, it’s harder to see than with API calls. I think that there are some tools for this. I’m not sure if we used any of them, but probably also depends on the on the approach you take, because there’s no one standard for how to define events. I mean, there are some, but probably depends on the company and message broker and technology use.

Robert [36:24]: Yeah, but I think you reminded me about this case and it’s pretty interesting that I remember in that in one place we had an event that we’re integrating, we have integration between two teams, but indeed, we have some data quality issues there. And what we ended at the end was getting rid of this event and instead having a synchronous endpoint that the other team needed to call. And we were able to basically do validation.

Miłosz [36:52]: Oh, that’s interesting.

Robert [36:53]: Yeah. And in this case, I would say that this extreme decoupling was bad because, again, there was problem of responsibility because somebody could emit an event that was totally not valid, and it was hard to enforce, because if this event was emitted, we’ve been notifying it after a while in our systems that we are starting to receive some events. It was like, oh, we don’t care.

Miłosz [37:23]: We did our job, we published the event. Yeah, that’s an interesting thing. It’s like on the boundaries of a team, when you integrate two teams, two services that belong to different teams. And the API becomes super important. And yeah, maybe sometimes synchronous is better in this case. Yep, that’s a good take.

Miłosz [37:50]: Let’s move to common anti-patterns.

Robert [37:54]: So you say that it’s not perfect and it has some pitfalls and something can go wrong with asynchronous architectures.

Miłosz [38:03]: Looks like there’s no super bullet. So I would start with two naive takes on background processing. So I’ve heard of systems, or seen some, where people use something like a cron job to process the background tasks. It’s similar to task queue, but more like in-house. And again, similar to task queue, it can probably work well for some cases, except you reinvent the wheel in this case so it’s a bit worse I would say and the main issue is if you have any more complex scenario then simple please do this later then it breaks it won’t help you much you don’t get all those guarantees of pub subs and so on and

Robert [39:05]: It’s also often processed at some magic hour so it’s creating delay basically, if you are doing bad processing with your current job.

Miłosz [39:15]: Yeah.

Robert [39:17]: But I would even say that it’s still better than database triggers because, I mean, it’s choosing between two evils, but at least I would say that it’s probably a bit easier to obtain.

Miłosz [39:32]: Yeah, logic in the database is a controversial topic. I wouldn’t want to work with a system that does it. And of course there is probably this one specific case where it makes sense. I’m sure there is, but in general it’s tough to update and manage this.

Robert [39:52]: And test and debug.

Miłosz [39:55]: Yeah. If I had to use a cron job like this, I would just do it in the code. And for example, in Go it’s trivial to spawn a new Go routine. So that’s a perfect scenario. Have anything in the loop, pull the sleep and do something every, I don’t know, minute or something.

Robert [40:14]: It can be some good starting points that you can migrate a bit later to a more proper solution.

Miłosz [40:21]: My main issue with this is that you reinvent something that has been solved. And many people put years of work into figuring out edge cases and everything. So maybe it’s better to do something that exists.

Robert [40:38]: Especially that, for example, with Watermill in Go, you can pretty easily use Postgres or MySQL as PubSub. So basically, you don’t need to add an extra architecture there. And you can start with something simple, and without many changes, change it to some real PubSub later. So I would say, I know that it’s not available in all languages, but in general, it may be also some nice starting point. and it’s just less to migrate later when it will start to be problematic.

Miłosz [41:10]: There’s one worse kind of cron I’ve heard about, is that you write a cron to check for inconsistencies in the system.

Robert [41:22]: Oh, no.

Miłosz [41:24]: So let’s take this example of calling this marketing API after a user signs up. So you realize this can sometimes fail and you write a cron job that does a check between the database and the marketing API and see if anything is missing and

Robert [41:46]: Basically you need to implement everything twice.

Miłosz [41:49]: Yeah and my my worst worry here is that you kind of promote ignoring issues And whatever happens, whatever issue like this you have in the future, a challenge like that, you will just write another consistency check. And this is another code you have to maintain. And basically you just sweep the issue under the rug instead of fixing the root cause. So yeah, I think that’s too naive to take to work in a large-scale system.

Robert [42:30]: I can remind one case when it makes sense, but I would say it’s rather an exception. When you are working with some, let’s say, critical systems, you are quite sure that everything is fine, but you would like to do some double-check.

Miłosz [42:51]: Checks in general, consistency checks in general, I wouldn’t say that are a bad thing, but if you are doing them because you can design a better API, that’s the issue. You anticipate them to happen. I would create a consistency check in places where I’m sure they won’t happen, but I want to be extra careful anyway because it’s so critical.

Robert [43:19]: I think the big difference is if you are implementing some compensating operation in this checking. For example, if you are working in some financial system, it’s typical to do some reconciliation reports in some intervals, but it’s making reports. It’s not doing the compensating operations. I mean, your logic should be valid and you should just do validation at the end to ensure. But again, It’s just for critical systems that you need to be sure. For example, if your company or whatever, $100 came in, $100 should come at the end. But again, it’s not about doing compensation. It’s all about double-checking the logic that you have.

Miłosz [44:06]: Right. Especially if you have a few of those in the most critical areas, that should be fine. But if in every API call you need to do this check and compressing logic then it won’t be maintainable after a while you will go crazy just trying to cover all these cases and then any API change or whatever will also break this so you have more code to maintain again I would fix the root cause if you know it can happen.

Robert [44:42]: Agreed Thank you.

Miłosz [44:44]: And one more regarding this naive approach is, you know, when you use messaging and use kind of events, like you said, those passive-aggressive events, or just something that’s more like technical, not in the domain or business sense.

Miłosz [45:03]: It can seem like smart design, but not always the case. It doesn’t automatically give you good results. I worked once with a system where there were two applications, like an old one and a new one, and they shared some entities. And there was a system that, for every change in the entity, published an event that it changed, the other system consumed that and updated the entity, and the other way around as well. So, you know, if a naive setup like this, you ignore many edge cases that can happen. And there are some variants on the entities, they have some rules, but if you just do a naive update like this, even if it’s asynchronous, it doesn’t give you anything and can be even worse because there can be some inconsistencies and you won’t even know. Because maybe some message arrives out of order, or is overwritten by the other one.

Miłosz [46:16]: So this comes back again to the event design topic, right? This needs to be a proper design. You need to spend some time and understand what you are doing.

Robert [46:30]: And I think it’s probably time to give a big shout to event storming. That was usually pretty useful for designing some event-driven systems. But not only, because it may be also suggesting that event storming is for designing event-driven architectures or async architectures, but it’s not. I mean, event storming is good for basically designing any kind of system, but it’s pretty useful that you have notation that can be mapped to your software really directly, so I recommend it to check event storming. If you’re looking for some materials how to do that, there is an e-book that is probably unfinished for the last 5 or 10 years. But no worries.

Miłosz [47:14]: It’s great anyway.

Robert [47:15]: I think 5 or 10 years earlier it was already very good. So Alberto, really good job.

Robert [47:29]: Another downside or trap related to asynchronous architectures is creating distributed monolids. Because, like many useful techniques, they are helping with many things, but often they are also opening gate of hell.

Miłosz [47:44]: It’s the new normal. Instead of monoliths, we have distributed monoliths.

Robert [47:51]: Let’s wait for distributed monorepos.

Miłosz [47:54]: We’ve seen it too many times.

Robert [47:57]: Yeah, but the usual scenario is like, okay, microservices are great, we need to be scalable, etc. And the team is starting to build microservices, and you start to have more microservices than people, and at some point you’ll notice that communication between them is hard because if one microservice in the middle will not work for a while because our Kubernetes cluster is not that stable yet, all our requests in the platform are failing because of cascading failures, we need to do something about that. And unfortunately, often a reversing decision of going into microservices is not on the plate, because come on.

Miłosz [48:41]: Too late for that.

Robert [48:42]: Yeah, we invested that much in many stuff that we don’t need. So, often the solution is, well, let’s use event-driven architecture. And, well, it’s making all the problems that the team had earlier, it’s making it deeper and deeper, because debugging everything is harder, understanding everything is harder, everything works slower. Or, okay, the one upside may be that, okay, at least it’s kind of more stable, with caveat that usually there is the thing that Gabriel mentioned about UX, that, well, if something is processing in the background, it’s a bit harder to show it in the UI. So compared to showing error directly to user like earlier, now something is happening and user have no idea what is happening. And yeah, it’s a pretty problematic thing that we’ve seen many times. So it’s important that asynchronous architecture can solve, in this case, some problems. But again, don’t be afraid to do the thing that we like to recommend, and we’ve did multiple times and it’s the microservicization. So basically getting your services and making less services. Again, having more microservices than people in a team, it’s terrible.

Miłosz [50:04]: But the interesting part about this pattern or anti-pattern is that it comes from good intentions.

Miłosz [50:12]: It starts with this idea that we have a small team now but we will grow and we need to to be able to separate some code easily. And then you have those separate services and you figure out, okay, let’s now just connect them like we did with function calls before. And then it breaks, but it comes with this good intention of separating the constants.

Robert [50:44]: Oh, I think it’s problematic with good intentions, because I think all the bad systems that we worked with was built on the good intentions. Because, you know, it’s rarely the case that somebody had a bad intention. Like, let’s build the worst systems that we can. Yay, hooray, it will be so great idea. Usually, the intention is good. But I also understand that sometimes it’s also hard to reverse this decision. But from the other side, it’s also sometimes leading to systems that are totally unmaintainable. And, well, even I can say that rewriting is the only way to redo something.

Miłosz [51:27]: It’s just counterintuitive, because you think that splitting things makes things decoupled. But it’s not really the case. It depends on how you communicate, really.

Robert [51:38]: Yeah, so basically it’s all about splitting things that are coherent. So if you are splitting things that are coherent, you are just ending up with a lot of accidental complexity. And it’s pretty bad. I think it’s also partially the reason from where some people see asynchronous or even given architecture as anti-pattern and over-engineering. Because if you’ve seen many or a couple projects using some patterns and you’ve always seen that is overcomplicated. Well, it’s logical that next time when somebody will suggest you let’s do it asynchronously or let’s use evangelical architecture will be pessimistic because you’ve seen that, okay, it was always an over-engineered problem. But again, it’s probably, it wasn’t a problem of evangelical architecture. It was probably a problem of applying it in the wrong project.

Robert [52:30]: It’s important to keep it in mind. So if you’ve seen some technique or tool in many places and it was always overcomplicated, it’s maybe not a problem of this tool. Maybe it’s a problem of a person using this tool, or this person maybe didn’t understand this properly. It’s a thing that we’ve seen in many places, and sometimes it’s requiring a lot of effort to convince people that no, no, it’s not because of this tool, it’s because of applying it in the hard place. But again, if you are not convincing somebody, it’s not our issue at the end.

Robert [53:09]: I mean…

Miłosz [53:12]: There’s also not really a strict definition of all those techniques. And depending on the language you use and other technology, it will be a bit different. That’s why it’s hard to judge. Wrapping up distributed monoliths, the coupling is the issue here and… Synchronous is not really decoupled by default, because events are also a form of coupling, in what we mentioned before, right, so… On the one hand, you have this, you public an event and you don’t care what happens later, On the other hand, it’s also a kind of contract between you and the consumers. Let’s say you have this distributed monolith and you now replace the RPC calls with messages. Is it decoupled? No, not really. It’s the same issue, just with different transport. So it’s more about the design of communication, not really the async or sync here.

Robert [54:27]: Because, yeah, basically you can implement RPC over queue and it will be basically the same. I mean, it doesn’t change much if you are coding something directly over HTTP, if you are expecting to have a response from your operation.

Miłosz [54:44]: Yeah, I think Gabriel mentioned this before, that there is one interesting pattern is used by NATS. The call seems to be synchronous, but it’s asynchronous under the hood. I’m not sure about this exact one in NATS.

Robert [55:01]: But yeah, I guess that it may be about something that Celery supports. So basically, it’s basically doing RPC over queue.

Miłosz [55:11]: Yeah, this is a weird hybrid approach when you can do synchronous over message broker. It kind of works.

Robert [55:20]: It has some use cases, obviously, but it’s not really asynchronous. We also have each in Watermill, so we have the Request-Reply component. But again, it’s not really fully asynchronous. So it’s nice because you can have some response to know if something was successful, and if you are no longer listening, it will be processed anyway. Not like with Request, because if your Request data is originating, let’s say originating request, and it’s calling everything later, and this request will be cancelled. So if you are handling context cancellation properly, everything else should be killed, basically. So it should be kind of rolled back in all requests. But if you are doing RPC over cube, it will be done everywhere anyway. There are some cases when it makes sense, but…

Miłosz [56:07]: Maybe one promise here is that you use one platform for all communication. So it sounds good on paper. We don’t need RPC because you can do everything over messages. Maybe it’s a good idea. I don’t know.

Robert [56:22]: But it’s tricky. If you have a request and you would like to, for example, insert something in the database, your request was cancelled. And this cancellation is not propagating. because the translation is not propagated to the queue, so it can lead to some strange inconsistencies. Because you killed the request, so expect that it should be not written, but it’s somewhere later in going through the queue, so it will be done anyway. And it can lead to some strange state where you don’t know what really happened.

Miłosz [56:54]: Yeah, I would summarize it as messages behave differently from requests most of the time. So, if you use one as the other, it can work, but you can have some weird edge cases down the line, because it’s just a different method of communication. So I will be careful with this. Okay, and any more anti-patterns?

Robert [57:24]: So I have two connected anti-patterns here. So the first one is using no message ordering. So it’s maybe not anti-pattern, but it’s something to watch out. Because in many systems, you need to have some way of ordering your messages. So let’s imagine that you have event for subscriber subscribed and subscriber unsubscribed. And the problem is, if you not order them properly and build some read model based on them and they will be received out of the order, you may unsubscribe a person that was not subscribed yet, if the unsubscribe event will arrive first, because it can if it’s not ordered, and later you will have a subscribed event that should be…

Miłosz [58:13]: Yeah, that’s one of the things you learn when you first start using messages. Those wish behaviors that seem simple at first.

Robert [58:24]: Yeah, and the simple solution may be, let’s just order every message and just have everything ordered. And yeah, it sounds like a pretty good idea, but sometimes it may work. But it has also one big downside, at least. I would say two big downsides. So the first one is performance. So, basically, performance of processing those messages will be limited to how fast you are often to process them. And basically, they will be processed one by one. And if operations will be slower, it will just start to pile and you will be not able to process them.

Miłosz [59:01]: Or block all processing of all messages.

Robert [59:05]: And the second thing that I believe is worse is that if any of messages will spin, it will block all other messages. And it’s pretty bad because it’s killing all the advantages of asynchronous architecture in terms of stability.

Miłosz [59:26]: Yeah, on one hand it can help you not care about ordering, which seems like a nice idea. Yeah, but if you make it too wide, then it can be also an issue. There’s a lot of things to consider here, so it’s hard to give someone a solution, right? So much depends on the domain and the system design. Sometimes you can get away with no ordering, but you can use something like a version. I remember one system where we didn’t have ordering at all, because all events updated different parts of a read module.

Robert [1:00:12]: And I think it was also the problem where we’re using Google Cloud PubSub and it didn’t support ordering at this point also.

Miłosz [1:00:20]: Yeah, we can kind of get away with it, but I remember we actually used times for solving some conflicts, so it’s also not ideal. Depending on timestamps is controversial.

Robert [1:00:33]: It’s hard to synchronize. I think in some places where it was possible, we’ve been using versions. So basically every event was that within aggregate, let’s say that, within the, let’s say, transactional boundary, so where we need to keep ordering, had an incremental version. So if we received an event that was out of order, we’ve been rejecting that or in some cases we’ve seen that, okay, we received newer price, for example. Sorry, we received older price and we already stored the newer, so we just can ignore older. But the downside is that you need to support it in custom way for everything.

Miłosz [1:01:17]: Not super complex, but something to consider.

Robert [1:01:20]: But nice to have it out of the box. For example, Google and PubSub have ordering keys. Kafka has partitioning. So there are method workers that just have it out of the box. And you don’t need to think about that later, basically.

Miłosz [1:01:33]: Remember, you can talk a bit more about this in the next episode.

Robert [1:01:39]: Yeah, I think ordering is probably something that can be just an entire episode about that. because this is pretty… Maybe not complex, but it has many different use cases that it’s good to know about. Because in many cases, you should approach it differently. Often, you don’t need to care that much. You can choose something simple and it’s fine. But it depends a lot on scalability requirements or and, let’s say, resilience.

Miłosz [1:02:10]: The balance is difficult. It’s easy to do. No ordering or ordering by every message something in the middle you have to consider manufacturers

Robert [1:02:20]: Yeah but tldr i would say it’s try to do as narrow ordering as you can so for example try to do some subset of events for one customer maybe or you need to basically analyze where this ordering matters and based on that, do really narrow ordering and it will work, basically.

Miłosz [1:02:48]: There’s one thing that’s similar between async and sync. I think it’s API design. So with Synchron’s approach, you have some kind of contract in how your API works. If it’s a REST API, you can have some open API document with gRPC, probably protobuf schema.

Robert [1:03:14]: Yes, I see that Gabriel asked about that. What we are usually using is protobuf, because you can marshal it later to at least two formats. Usually we are using JSON as a transport format, because it’s easier to debug. But if you care about performance and the size of message, you can use protobuf, so it will be a bit smaller and faster. And you can also convert it to multiple programming languages, and also it’s nicely supported the versioning of fields.

Miłosz [1:03:51]: There’s the cloud events schema, I think. We didn’t use that yet. Protobov usually does the job. The point is you need to follow similar ideas for both sync and isync here. So your event schema or your API schema is your contract. So you have to make sure there are no breaking changes. And yeah, the API is stable. And probably do some versioning if you need to do breaking changes. We sometimes have events with version delay, like v1, just to anticipate it. Not always needed, but sometimes useful.

Robert [1:04:44]: Yeah, and it’s also worth mentioning that many message brokers are supporting some kind of schema registry out of the box. So those message brokers can validate the schema for you. So, for example, Kafka has it. I think Google and PubSub have it. I think it’s also worth mentioning that some companies are using Avro for that. I think we didn’t use that, but I know that it’s often used in some companies, but I think I guess that it’s more in Java world. I think in Go world, more gRPC is used, at least from our… Protobuf. Protobuf, yeah. Protobuf is more used in our bubble.

Miłosz [1:05:29]: It’s pretty nice. And as a plus, generated code is It’s a quite popular choice and it works pretty well with Protobuf as well.

Miłosz [1:05:49]: Shall we move to the final question? What to choose?

Robert [1:05:54]: Unfortunately, it’s as you heard. There are really many factors to consider. Yes.

Miłosz [1:06:05]: And the balance is the hardest part, as always. There is no silver bullet, so you have to figure out what balance means for you. It’s a bit like the CAP theorem in databases, right? So you have consistency versus availability, and you have to pick one.

Robert [1:06:26]: Yeah, so in async system you basically are sacrificing consistency, but you have better availability.

Miłosz [1:06:36]: You have eventual consistency, but that’s it.

Robert [1:06:40]: But it’s eventual. Cool.

Miłosz [1:06:43]: So what should we choose, do you think?

Robert [1:06:48]: I think to give you some simple algorithm, maybe. If you’re in-depth, you can always start with synchronous. I would say that in many cases, it will be a good starting point. In many cases, using async, by default, maybe, Maybe not over-engineering, but maybe not needed at the beginning. And if you have infrastructure already in place, switching something from async to sync, may be not that big difference, as long as you didn’t do any strange thing. And it’s especially simple if you are using clean architecture, because you should have some application layer entry point to your application. So basically calling you from RPC or event handler or something, it can be a pretty simple change. Of course, it depends a lot on the use case, but again, it’s if you’re in that. In some cases, you may be just sure, like you’re calling some external system and you know that it’s not stable, do it synchronously. You have some critical flow that you know that it needs to be working, you need to send some external data, do it asynchronously. Again i’m saying more about cases that you are not sure.

Miłosz [1:08:10]: Sometimes it’s a product decision as well if something can happen in the background and sometimes it’s tricky because in the background can mean a second like later or you know 10 seconds later so it so it’s also it takes some skill to to be able to talk to you know whoever your stakeholder is whoever is responsible for the product to understand if it’s fine to do it in an asynchronous way. Which means most often it will show up in the UI right after, but sometimes it will not after a few minutes maybe. That’s also counter-intuitive. Why would you talk with your product owner about sync versus async architecture, sometimes it makes sense.

Miłosz [1:09:07]: Probably it’s good to watch out for the extremes, as always, right? So, not everything can be handled by synchronous architecture, as you mentioned. There are some hard limits, where more complex systems just won’t be able to do it. But as you said, sometimes doing everything over messages can be also over-engineering. So if you ever consider doing everything one way, probably you need some balance. It’s usually a clear sign.

Robert [1:09:45]: And if you maybe didn’t work in any system that required building asynchronous architecture, it’s fine. But anyway, I would recommend to check out how you can do that, because it’s just know what tools you have available. So if you have a problem that will require asynchronous architecture approach, you’ll just know, okay, I have tool for that, it fits here, and I can use that. So you will not need to learn it when you have this problem, because it might be just too late.

Miłosz [1:10:17]: Sometimes a hybrid approach may be fine. One is what we mentioned before, the RPC of our messages. A bit unusual, maybe if it fits some use cases. Something else that comes to my mind is CQRS here. So, TLDR is a pattern where you split your controller functions, let’s say, into commands that change the system and queries that fetch the data. Super short definition. Not to dive too deep into it, but the interesting thing about it is that it gives you this extra structure in your code. Commands and queries are separate. And then it can make mixing them and how you call them a bit more intuitive. Because queries are usually synchronous. So someone is waiting for the data, that makes sense. and commands can be either. So it can give you this mental framework to decide how to mix sync and async in one project.

Robert [1:11:33]: I think it’s interesting, the CQRS example, that it’s a good example of technique that people really misunderstood because for many people if they hear CQRS it’s immediately asynchronous but no. So it’s totally misunderstanding of this pattern. Again, we’ll add to materials our article about SQLRS, because we’ve been covering that. Milos also did a really nice presentation about that, so I think we can also put it to the materials. It’s important to mention it all in our SQLRS. It’s not about doing things asynchronously. It’s some implementation. it’s worked nicely, but you can do skewers with just doing everything synchronously. And it’s nice because if you are using skewers in a proper array, so when commands are not really returning stuff, you can convert it to asynchronous architecture really easily, because it should not really depend that much on having the feedback from the command.

Miłosz [1:12:40]: It helps if you don’t have these mixed functions that change the state and also return data. It can be more tricky to decide what to do with them.

Robert [1:12:49]: And it also doesn’t require some super big overhead because this is also something that we heard from many people, like, oh, security requires so much effort. But no, it’s just a matter of, instead of having some super big application services with everything, just have multiple little services with commands instead of arguments and commands that doesn’t return stuff. And that’s it. And you have the basic secure implementation in your product. You can put it into your CV.

Miłosz [1:13:24]: Um…

Robert [1:13:27]: So yeah, I already mentioned also the migration. So SecureOS helps with that. But you can also get some hybrid approach, because often at the beginning of the project some endpoint can be synchronous, but with time, more and more stuff is added to that.

Robert [1:13:45]: We have actually a good example in our trainings platform. So we have webhooks from our payment provider about information that somebody bought our training. And at the beginning it was pretty simple, just adding, ah, somebody has training. But now it has a lot of stuff and we need to actually convert it to asynchronous to using events. But now it’s probably doing five things. And sometimes some of those things are failing because it’s just doing a lot of that. But, again, at the beginning, it didn’t make sense to do it asynchronously because it was pretty lightweight, but with time, you can convert that. And you can even start with still having a synchronous endpoint and with emitting some internal event or command and just cutting it part by part. If it’s some internal logic, you can, at the end, maybe just remove this endpoint and say the team that is calling this endpoint, you can now just emit an event or the opposite. it. So, for example, if you have a problem with data quality and other team is not taking responsibility of that, introduce endpoint instead of event and say, if you will not respond that this is valid, sorry, you need to provide this valid event and just agree what data should be in this endpoint to make everybody happy.

Miłosz [1:15:04]: And speaking of code structure, maybe one more thing that can help is layered architecture or cleanarchitecture or hexagonal or whatever you want to call it. If you place the logic in some isolated logic, place in your code, and then have two entry points that are either messages or RPC or HTTP. It’s super easy to switch between how you call the handler, because you have just two entry points that do nothing else, just just map the request to your internal structure. And it makes it super easy to migrate as well.

Robert [1:15:53]: And it’s happening, I would say, because it’s often argument in clean architecture, like, okay, in clean architecture you’re separating database, you can switch it, or whatever. It doesn’t happen often. But changing things from synchronous and asynchronous actually can happen quite often.

Miłosz [1:16:11]: Or just both at the same time. You can have some logic that can be executed synchronously or asynchronously by message. And if you use the layered approach, it’s super easy to do.

Robert [1:16:22]: Yeah, and it happens pretty often because database, again, you are not changing it very often, but changing things from synchronous to asynchronous or vice versa, the first I think it’s happening more often, but it’s happening often. Because, again, just some operations are starting to be bigger and bigger. You start to have more and more teams. Your product is growing. This is, hopefully, what’s happening. And in this case, it’s just making your job easier and saving all of your time. So it’s, I think, a great investment that is not mentioned that often.

Miłosz [1:16:56]: Yeah. Okay, maybe as a finishing point, we can talk about reducing this accidental complexity that comes with starting with messaging. First, once you start, but there’s a lot of topics to grasp. We talked about some today, but it’s not really an episode about event-driven architecture, only the comparison of Async and Sync. So one idea might be to start with some PubSub you already have, which is probably SQL database. I probably have some Postgres or MySQL running around, and you can actually have a pretty simple PubSub running on it. That should be good enough for most use cases. Or you can use a cloud-based PubSub. It means you don’t need to maintain the architecture or infrastructure, which can also be a complex task sometimes with high availability in place and all this.

Robert [1:18:10]: I think Postgres is a great starting point and it can handle much more than you think. I actually have some idea about one article that maybe we can show how far away we can push Postgres-based paths up, because I’m pretty sure that we can push it pretty far away, but let’s see.

Miłosz [1:18:29]: Yeah, unless you want some complex topology. For starting out, it’s a great way to start. Another way, maybe using a high-level library that abstracts away some of the concepts for you. So for example, Robert mentioned earlier Watermill, which we maintain in Go. I’m not sure about other languages. I think it’s something there’s something similar in Python. Probably in every other language there should be something like that. Because you probably don’t want to deal with all the protocol, the infrastructure you use, all the low-level details, so you can focus on the high-level publishing and subscribing to messages.

Robert [1:19:21]: It’s pretty easy to mess something with that. So lose message or make some sub-toolbox. We’re creating WaterMeal already for a couple years, and you’ll be surprised how many little quirks you can have in every PubSub. It’s very hard to debug at the end, because you may just be losing one message for one million messages, and it’s very problematic in systems. It’s good if somebody already did it for you. If, for example, you don’t use Go, We’ll recommend to use Go, but if you can use Go, you can always even look on our SQL implementation and see and inspire by our queries.

Miłosz [1:20:06]: Just use Go.

Robert [1:20:07]: It’s a best idea.

Miłosz [1:20:11]: It’s not that complex in the end. But there are also some quirks, so it’s better if you don’t reinvent the wheel. Because it took us some time and I think four major versions to arrive at where the SQL implementation is right now. So I’m going to manage cases.

Miłosz [1:20:33]: So, if you’d like to learn more about async architecture and event-driven architecture in particular, we have an event-driven training that’s sales open for the next two weeks, starting from today. So you can learn those concepts in a more hands-on way, solving exercises.

Robert [1:21:00]: If you are interested in learning, Doug, you are actually pretty lucky, because we’re running this training by Sayer, so if you are listening to it later, it may be already not open, but again, it will be open next year.

Miłosz [1:21:17]: If you listen to it later, you can join the waiting list and wait for the next sale. We run it as a cohort, so everyone learns together. And yeah, you can just Google Go event-driven or we will put a link in the notes.

Robert [1:21:32]: We have pretty good SEO, so it should be on the top of the list when you will Google Go event-driven.

Miłosz [1:21:40]: Okay. Before we move to the Q&A.

Robert [1:21:45]: Yeah, definitely. Okay, so Dima mentioned that naming always matters. Yeah, so definitely it matters more often than you will think, but…

Miłosz [1:22:07]: Okay, so Gabriel shared more about that events should be facts that already happened in the system, what we talked about. Provides many advantages as states can be recreated from events. Yeah, that’s one of the things we didn’t mention. It’s super useful so you can have this event log somewhere in your system and then reply those events to do whatever you want.

Robert [1:22:31]: And in other words, you have event sourcing, but event sourcing is not asynchronous. I mean, event sourcing can be asynchronous, but it can be synchronous as well. So I would say that it’s synchronous by default. And then it’s probably a nice topic for other episodes. So we mentioned already financial systems in a couple places, and event sourcing is actually great for this kind of system. When you need to have audit by the law or you need to have audit because you would like to understand how you ended up with some state.

Miłosz [1:23:10]: Gabriel also mentioned this dependency matrix of events based on types could be helping an organization. I think I said something like, I wonder if there is a ready software for this. I think event catalog or something like that, but we didn’t use anything that we could use across teams and I mean place where you can just go and see which team uses what because it is to integrate with your code I guess ideally if you don’t want to keep it up to date over time which can be a manual effort so it sounds like a challenge to do it for any language you use and if we find something like that we’ll let you know

Robert [1:24:15]: So Gabriel mentioned that there is a nice chapter about event storming and learning domain driven design from Orly. I didn’t read this one, I’m not sure if you had a chance.

Miłosz [1:24:25]: I have it on my shelf. Still waiting.

Robert [1:24:29]: TBD. Maybe this summer.

Miłosz [1:24:32]: But everything about event storming, I think you can recommend right away.

Robert [1:24:38]: Yep, definitely. I think event storming was one of the biggest game changers for working with some more complex projects and collaborating with some multiple stakeholders.

Miłosz [1:24:51]: Also not super complex to learn, but you have to be willing to talk to people. Can be controversial in tech teams, sadly. Ordering is probably not needed if you have a good state machine, which guarantees the state of aggregates. It sounds to me connected to event sourcing again.

Robert [1:25:17]: Or the thing that we have with versioning. That’s true. You can implement that. But from other side, if you also do it properly from a message broker perspective, it can simplify also a lot. Because you can just listen to event and don’t care about that. We’ve been able to compare migrating to that and ordering, at least in our case really help to simplify it a bit. Because I think it’s good because if it’s done properly, it works out of the box. So kind of out of the box, let’s say. And it’s harder to make mistake when you need to remember about something. But you can also implement universal state machine that can handle that. But universal things that are implemented by you often may have some chances.

Miłosz [1:26:07]: But you can see how many approaches there are to ordering. It’s a pretty long topic. You have to consider many approaches. So thanks for mentioning this one.

Robert [1:26:21]: So Gabriel will have Monopoly on our chat today.

Miłosz [1:26:25]: Thank you for contributing. You are the third author today.

Robert [1:26:31]: I think at some point we should try to also share the link to join with us. That would be cool. Risky but cool. Gabriel also recommended Reactive Manifesto, which is a nice definition of Reactive Systems. I remember reading that probably a long, long time ago.

Miłosz [1:26:52]: Yeah, same.

Robert [1:26:53]: Recommend checking, yeah?

Miłosz [1:26:55]: I feel like an ignorantat the moment, sorry.

Robert [1:27:00]: In Scala, there is a Lagon framework, which was nice, Akka is under the hood. Maybe. We are not Scala guys.

Miłosz [1:27:12]: Yeah, so it’s good that there is similar software, so you don’t need to understand everything, what’s going on, which can also be tricky because sometimes you may lose some you may not understand something correctly, the configuration of some PubSub. But to start out, it’s super important to have this A-level concepts in place. You just publish a message, subscribe to subscribe to a message and don’t care about what the bytes do underneath.

Robert [1:27:41]: And also Gabriel mentioned that it’s under the hood, so it would be actually interesting to see how actor model is going up. I mean, if it’s visible or not. Personally, I’m not a big fan of actor model. I mean, I don’t like the mental model of actor model. So even driven fits in my head much better because you can get to even storming and it maps one-to-one in some way and actor model for me is a bit more like magic mental mapping that you need to do and, I don’t know what’s your opinion on that but.

Miłosz [1:28:13]: Yeah I didn’t use it much except for playing a bit with elixir so yeah I can’t say much maybe you can do another episode sometimes about comparison

Robert [1:28:29]: Fortunately Gabriel said that but I love Go much more.

Miłosz [1:28:34]: Way to go!

Robert [1:28:43]: Okay. Cool, so a couple more seconds for more questions.

Miłosz [1:28:54]: While we wait, it’s the time you should hit the subscribe button and rate our show

Robert [1:29:02]: Let’s wait Please click this button You didn’t click yet I see your screen Okay, thank you So, also now 5 star review, Let’s write something nice. We’ll wait here. We have time.

Miłosz [1:29:23]: Yeah, and instead of the YouTube subscribe, there’s an even better subscribe button on our website, where you can join our newsletter, so then you won’t miss the updates from us.

Robert [1:29:34]: Plus, you will also receive notifications about the episode notes. There is an entire transcript. We’re linking to all sources that we’re mentioning here, like WaterMule, like ebook about event storming, and about a couple more things that we mentioned and I don’t remember now, but it will be on all-in materials for this episode. Plus, also if you release any new articles on the blog, you’ll be notified. So I think it’s pretty cool, because I mentioned, for example, article about SSC that we have on our blog. So we’re trying to share a lot of quality stuff that will be useful for you. Something more, Miroz.

Miłosz [1:30:19]: I think we have no more questions. Thank you, everyone.

Robert [1:30:22]: Just a reminder. So, Go Event Driven, available for next two weeks. Later, you need to wait for half year. So, don’t wait for the last hour, because…

Miłosz [1:30:34]: Better to join now.

Robert [1:30:35]: We are starting, and you will need to wait. Cool.

Miłosz [1:30:41]: Thank you, everyone, for joining us. Thank you, Robert, for joining me.

Robert [1:30:46]: Thank you, everyone. Thank you, Miłosz. And see you in two weeks. And we’ll talk about…

Miłosz [1:30:52]: Event-driven architecture more in-depth than today.

Robert [1:30:55]: Sounds like a good plan. So, see you in two weeks. Thank you very much.

Miłosz [1:30:58]: Bye-bye.

Robert [1:30:59]: Bye.

Watermill: from a hobby project to 8k stars on GitHub

Wed, 14 May 2025 16:00:00 +0000

Quick takeaways

Solve real problems first - successful open source projects start by addressing actual needs, not by looking for problems to fit a solution
Keep breaking changes minimal - Watermill stayed on v1 for 6 years with no breaking changes in the core library, building trust with users
Examples and documentation are crucial - provide real-world examples with automated tests, not just simple “hello world” demos
Promotion matters - creating a great library isn’t enough; you need to actively share it through conferences, blog posts, and communities
Be patient with growth - Watermill took 7 years to reach 8,000 stars; overnight success in open source is rare

Introduction

In this episode, we share the story of how Watermill, our event-driven library for Go, grew from a side project to a popular open source library with over 8,000 GitHub stars and 100+ contributors.

We discuss the key decisions and strategies that helped make Watermill successful, from focusing on solving real problems to maintaining backward compatibility and building a community around the project.

Show notes

Watermill documentation — our open source event-driven architecture library for Go
Watermill repository
Go Event-Driven training that complements Watermill
Real World Examples

Quotes

My idea was that it should be possible to make building event-driven applications as easy as HTTP handlers. And if you think about building HTTP APIs, actually you don’t need to care about many things when you are building it. So you don’t care about connection pooling, handling TLS, HTTP protocol, TCP protocol, DNS resolution, partial reads, network failures.

Robert

It’s not as easy as upload anything to your GitHub so some recruiter can tell you know how to code. But rather, if you spend time and maintain the library over time and people find it useful, then it can work like this.

Miłosz

If you would like to create a library that people like to use, create something that people need. I know that it sounds obvious, but I’ve seen a lot of libraries that, well, the use case may be clear for the author, but if the author is posting it somewhere, people are like, yeah, it’s cool, but we don’t really know how to use that and how it can be useful for us.

Robert

If someone sees a project that has not been updated since a few months, they might be afraid to use it again. Like, is it deprecated? Is it maintained? Because I’ve seen there is no updates. So regular updates are often a good idea.

Miłosz

If you are building your library and you have pretty complex examples, it’s a really good idea to start with some end-to-end tests for them from day zero.

Robert

Simple examples are fine to start with, but then when you try to apply this in your own projects, you will see it’s not that simple. I guess it’s better to have some more complex examples and see how it works in real life.

Miłosz

Timestamps

Transcript

Robert [0:00]: Did you ever wonder what’s the real key to build a successful open-source project? Today, we are going to share what helped us to grow Watermill from a small sideproject to a popular open-source library with over 8,000 stars on GitHub. What have we learned along the way? I’m Robert.

Miłosz [0:19]: And I’m Miłosz, and this is No Silver Bullet Live podcast, where we discuss mindful backend engineering. We have spent almost 20 years working together in different projects and teams. We learned that following advice like always do X or never do Y doesn’t work and can limit your growth. In this show, we share multiple perspectives that will help you make smart choices and grow into the principal engineer level.

Robert [0:46]: So if you have any questions, leave them on the chat, so we’ll surely address them at the end of the live. or maybe during, if it will be also relevant to any place that we’ll cover during the discussion.

Miłosz [1:01]: Right, so let’s start with what is Watermill for anyone who is not aware. Watermill is a code library we maintain for building event-driven and message-oriented applications, and building it the easy way. And it has currently over 8,000 stars on GitHub, more than 100 contributors.

Robert [1:25]: Probably somebody’s using that. So yeah, obviously GitHub stars is some kind of proxy, probably to understand how popular the library is. But, well, we’re not buying them, so probably it means something.

Miłosz [1:37]: Yeah, and GitHub also shows, I think, above 1,000 repressories that use it. Some of them are also quite popular. So this is kind of a social proof, I guess, the best we can have that it is in fact used.

Robert [1:51]: The outside is that we don’t really know what companies are using that. So we know some, but probably most we don’t know because we have no way of backtracking who is using Watermill. So if you are using Watermill in your company, let us know, because it will be cool to know.

Miłosz [2:07]: Sometimes we hear from people on conferences or just randomly on the internet that But yeah, we use Otremio, we love it. But it happens after someone uses it for a few years, let’s say. So it’s not always the best way to know.

Robert [2:21]: So it’s always cool to hear from you that it’s useful. So we always appreciate that.

Miłosz [2:27]: Yeah, so Watermill supports 12 officially supported Pub/Subs like Kafka, RabbitMQ, NATS, but also Postgres or MySQL. And we have SQLite support coming soon.

Robert [2:40]: Yep, exactly.

Miłosz [2:42]: So whatever infrastructure you use, you might find it useful.

Miłosz [2:49]: Before we go further, maybe let’s focus on what Watermill solves. Why did we create it?

Robert [2:56]: Yes, probably it will be easiest to start with actually how Watermill started. Because I think that this is interesting and it may be also useful for you to try to create some library. And, oh, it was like probably many libraries that are there. So we’re working on some project where we had one challenge. So in this project, Event-Driven Architecture would be a perfect fit because of resilience requirements and some tasks that we could do in the background. Upside was that I was already familiar with Event-Driven Architecture, but most of the team wasn’t at this point.

Miłosz [3:35]: Yeah, and I remember the meeting when we sat in the room and you draw the Event-Driven Architecture on the whiteboard and you were very confident it would work. And the rest of us were like, I don’t know, it seems complicated. And we never used those pops-ups, Kafka is scary, what if we mess something up? Exactly. What would be useful is to have something easy to use, to abstract away those complicated details.

Robert [4:06]: Yeah, and I would say that the bar was set pretty high because my idea was that, it should be possible to make building event-driven applications as easy as HTTP handlers. And if you think about building HTTP handlers, HTTP APIs, actually you don’t need to care about many things when you are building it. So you don’t care about connection pooling, handling TLS, HTTP protocol, TCP protocol, DNS resolution, partial leads, network failures.

Miłosz [4:37]: Reconnect.

Robert [4:38]: Yeah, it’s a lot of things that can happen in the background or it’s happening all the time, But you are not thinking about this normally. You are defining, I would like to have handler at this URL with this method, and that’s all. And it’s really high level, and you don’t need to think about all the details. And, well, Watermill is a similar idea. So you should be able to create your event-driven services, but with really high-level API. And adding support, for example, for one event should be super simple. You don’t need to really think about topics, Kafka, or any message broker that you are using under the hood. It’s not an only thing, because there are a couple more things that Watermill is solving. The next one is handing for you a marshalling and un-marshalling of messages, because usually you are using some transport.

Robert [5:36]: So JSON, protobuf, whatever, and just doing it over and over, it’s just boring and a waste of time. Another thing that also is cool and Watermill can support is allowing you to swap PubSub implementation pretty quickly. So if you have seen some presentation that I was doing about Watermill over the time, I often show an example when I was using, for example, Kafka, and I wanted to switch it to Google Cloud PubSub or vice versa. And I was able to do it during live coding, and it took like one minute or less to do that. And it’s cool about Watermill because it’s putting high-level abstraction on this PubSub. And thanks to that, you can switch it pretty easily, like with ORM, basically, with databases. basis.

Miłosz [6:27]: Yeah, so those are the basics we started with. And then we built more features on top of them. Some of them are quite complex, but with a high-level API, so you can use them easily, like CQRS pattern with commands and queries. Sorry, not queries, but commands and events. Or Poison Queue or the transactional outbox pattern, which is super useful and very common. are commonly needed, not everyone uses them, but probably more people should. And the important part of those features are that they should be easy to use for everyone in the team, even if they don’t know how it exactly works under the hood, which is something we would need in that project back then, because we, not all of us were experienced with even the architecture, but it would be a nice approach for the project. So if we had Watermill back then, it would be much easier to start.

Robert [7:30]: Yeah, and probably today we also don’t want to go that much into details about some Watermill specifics and things like what’s the difference between PubSub and Q, because I know that for many people it’s not clear what’s the difference, what are use cases for PubSub, what are for queue. It’s something that we’ll cover more in the episode that will be in two weeks. Today, I would like to more focus on what we believe that made Watermill successful. Because, again, it’s probably not the most popular library of all times in Go, but it’s probably the most popular library for building event-driven applications in Go. So I think it’s a pretty big achievement. And there are a couple of things that we believe that helped us with that. And I think it’s also worth mentioning that it wasn’t overnight success. And as we mentioned, so it was after some project that we started after work. And we started to spend more and more time on that. And it took us time basically to make it.

Miłosz [8:37]: And the first comment was seven years ago?

Robert [8:39]: Yeah, I think it was seven years ago and seven days, something like that. So yeah, super long time ago. and it took us half year to issue version 0.1 so not 1.0 0.1 so it was some kind of alpha that we were able to raise to people so and i think i didn’t really write down how many comments it was but it was a lot of work to at least prepare this 0.1 version i remember i’m actually not sure but I think before version 1.0, we spent 500 days to release that. And what also helped us during that time is that we didn’t have a daily job, because at this point we, for the first time, tried to go full-time on our own. That time it didn’t succeed, but at least we released Watermill 1.0. But it took a lot of effort to do that.

Miłosz [9:41]: It was just going over time, steadily, instead of this huge success right away.

Robert [9:49]: Like in previous episodes, we’re not giving you any magical solution that will make your library super popular in one month. Because probably it’s often about a lot of luck because you can create a great library, that for some reason didn’t take out because there are, for example, 10 other libraries that are doing the same. So there is just a lot of competition. And if you are not lucky, well…

Miłosz [10:23]: It’s difficult with regular products and open source is more specific than that. It’s probably worth asking why someone would create a open source project it’s probably not to make money you can see many companies and projects try to monetize on open source, it usually doesn’t end well all the issues with licensing and so on

Robert [10:50]: I think it’s not fully fair when you’re starting an open source project and making it closed source after time after more people are using that but unfortunately it’s often happening. I guess that often it’s some pressure from investors that invested in some company and it kind of makes sense. You can expect that if you are using VC-baked open source that someday investors will come and ask where is my money back.

Miłosz [11:21]: I can count on sponsors as such but it’s probably not the idea to do it just for this. We found a way to connect it with our commercial projects, but we will touch on this later. But so far, I think we can summarize that we learned a lot doing it. That’s probably the primary reason I would try again to build an open source library. Just to learn something new and publish it. And also gives you some credentials if it succeeds. Like if you’re looking for a new job or just want to have some proof of your competence. That’s a good way to do it.

Robert [12:06]: I think it’s interesting sometimes to hear on the interview, on both sides actually, because sometimes when we’ve been interviewed, some people know Watermill, but I also remember interviewing some people and they were like oh, okay, you’re a creator of Watermill and you work here, nice.

Miłosz [12:22]: But that’s the difference between a project you just upload to GitHub and no one uses, and some project that some company actually found useful. So it’s not as easy as upload anything to your GitHub so some recruiter can tell you know how to code. But rather, if you spend time and maintain the library over time and people find it useful, then it can work like this.

Robert [12:48]: And I think it’s also nice when you’re recruiting somebody and this person can send you his GitHub or her GitHub and you can see that the person is working on some open source project after hours. And you can have some understanding on how this person is working and you can also you know feel that this person is able to, spend some time after job to learn something maybe or maybe create something cool that other people can also benefit from yeah it’s.

Miłosz [13:19]: Also a nice way to give back to the community because we’ve been using many open source software before and this way other people can use our ideas which also It also feels nice, you can do it for the satisfaction you get from it.

Robert [13:37]: All right but it all sounds cool but many libraries are like that and they’re not that successful so i think it would be good to discuss some things that really makes some projects successful and, some of my biggest observation from other projects and i know it may sound obvious but, if you would like to create a library that people like to use create something that people need because I know that it sounds obvious, but from other side, I’ve seen a lot of libraries that, well, the use case may be clear for the author, but if the author is posting it somewhere, people are like, yeah, it’s cool, but we don’t really know how to use that and how it can be useful for us. And, yeah, it also doesn’t mean that it needs to be some solution for a problem.

Miłosz [14:36]: A very generic problem, right?

Robert [14:38]: Yeah, because solving generic problems is more harder because it requires more effort. So probably it’s good to find something in the middle, like, for example, water mill. So, creating event-driven applications, it’s not something that every company does, but a lot of companies does. So, it’s big enough, let’s say, market, if we can say it this way. But, okay, we can say that if you’re building open source library, it’s pretty close to creating a product. So, you can say that it’s some kind of market for that. And it’s big enough that there are many people that can use that and benefit from that. but it’s also kind of niche so there is no competition around there so it helps a lot.

Miłosz [15:25]: Yeah, and sometimes if you start a project and you have a problem you want to solve and then you look for a solution and you find an open source library so you might feel discouraged to try and build your own I think it still makes sense to try because you may have a fresh approach. You can provide a better API or better design or less dependencies. So there’s definitely a place for more libraries that solve the same thing.

Robert [16:01]: Yeah, definitely. So yeah, I think it’s a good…

Miłosz [16:04]: In Go ecosystem, it was clearly visible because in the very beginning there were only a few libraries to do everything. So everyone sticked to one, let’s say, HTTP router. Because there were no other options. But as other libraries appeared, many people moved on. Aside from routers, maybe CLI libraries are also a good example. So there’s still a place for better APIs or a different approach that will help people to do something in a better way.

Robert [16:47]: Again, try to focus on the problem that you are solving and avoid trying to find a problem for your solution. So I’ve seen always that people found some cool technology or some cool idea, but they didn’t have a real use case for it. I think it’s probably easy to do AI now. So you can try to put AI in many places, and it will be helpful in many places. But in many places, it will be more looking for problem, for solution that you have. So you should also really watch out for that. I think it’s one interesting observation about this focus at the beginning on solving the problem. I think at the beginning, we didn’t know what word terminal is. So we had, as we said earlier, we had a problem that we had people that didn’t build any event-driven application earlier. And we didn’t have an idea, okay, we need to build a library for abstraction.

Miłosz [17:51]: All use cases everywhere.

Robert [17:52]: Yeah, yeah. Fun fact is that actually Watermill at the beginning, when it was alpha, it was named GoDDD. So it was probably something that, okay, it was around these domain events that we had there. And it started with that.

Miłosz [18:07]: I’m still not sure why.

Robert [18:11]: Probably we’ll never know. And yeah if you’re experimenting with time you are experimenting with time, it’s important to not keep this for you forever so create some version that will be useful for people and try to share it ASAP for feedback.

Miłosz [18:40]: Just mark it as experimental so people are aware that the API can change so they are not annoyed. It’s normal to have some proof concept and then keep changing the API if you find a better way. That’s fine. Just make sure people are aware. But this early feedback can be super helpful though.

Robert [19:02]: Yeah, and one of the best early feedback that you can have, and it can be also a nice hint how to find a, problem that people care about is using it in your daily job as long as you can because i know that it depends it’s maybe hard in some teams but like in our case so we’ve been building it after hours but we’ve worked in a company that it was fine to just use it in a real project that we worked in so thanks to that we had real-time feedback for that and we’ve been able to dog foot it with it but Obviously, it’s just a starting point. At some point, you need to share it outside so other people can give you feedback, and you can adjust to that, because you will be not always right from day one.

Miłosz [19:54]: And after a while, it’s probably a good time to start with documentation. We have some docs in place, and the go doc we have is not enough, usually. Because the go documentation is not that bad, but it’s still… It’s sometimes hard to get started with just the code documentation. So, we found it very useful to have this getting started page, so someone can easily see how to install the library and some minimal example to get started, also some minimal theory. And then lots of real-life examples are also super useful, which is sometimes… I’m super happy when I see some library and I see the examples directory, I often go straight there, instead of trying to figure out how it works. Because you can just copy some snippets and it gives you a better idea of how it works, rather than trying to understand from the code itself.

Robert [21:02]: Or clone it and run it locally and play with that. So it’s also pretty useful. And I think it’s also pretty close to our way how we believe that learning is the best. So actually learning by doing. So it’s really close to it.

Miłosz [21:21]: And lots of our examples we have for Watermill are real-world examples. So they are more complex than a simple hello world. So you should be able to copy them to your solution and change them, and it should be what you need.

Robert [21:36]: Yeah, so if you’re longer with us, you should know that we don’t like examples like to-do apps or some fake examples, because usually it’s just hiding a lot of complexity. Okay, maybe some Watermill examples aren’t that simple, but they’re as simple as they can to show how to solve these problems.

Miłosz [21:59]: Simple examples are fine to start with, but then when you try to apply this in your own projects, you will see it’s not that simple. I guess it’s better to have some more complex examples and see how it works in real life.

Robert [22:14]: Yeah, for example, we have the example with handing ordering in your messages. And this topic is not simple. There are a couple things that you need to do right. It also differs a bit for different PubSubs. But again, it’s easier to give you more complex example rather than you need to figure it out yourself. Because we know that it’s not easy because at some point we needed to figure it out as well. At the end, your life will be easier with those examples. One tip. So if you are building your library and you have pretty complex examples, it’s a really good idea to start with some end-to-end tests for them from day zero. So every example that we have, it’s included in the documentation, but we have also automatic tests that are checking if those examples are working properly. So it’s cool because when we are releasing a new version of WaterMill, we’re updating dependencies, we can just run tests on all examples and ensure that all of them are working previously.

Miłosz [23:18]: It’s easy to break them without a version bump or something like this.

Robert [23:26]: But it’s worth mentioning that it’s not that easy, unfortunately, to create such tests. Sometimes it’s taking a pretty long time to run them. But again, it’s still better than not having an automation for that. Because doing it by hand is 10 times slower. And it’s nothing worse than examples and documentation with snippets that doesn’t work. Because it’s just a super bad experience. Because if somebody is trying to use your library and your examples are not working, Well, probably the person may just leave your library and don’t use it at the end.

Miłosz [24:03]: Yeah, but assuming they stay, you usually will have some early adopters of your library. It’s nice to have a place for communication, for building the early community. So in our case, it was issues and then Discord. Our Discord and dedicated channel. So it gives you a place to listen for this feedback from people and your suggestions. And also sometimes they contribute, which is super nice. You have your first contributors, so it’s not just your project, but you start to feel like you maintain it, actually. It starts to get more real. What other things help?

Robert [24:53]: The next thing that is coming to my mind is the fact that Watermill is not a framework. So, we’ve covered it more in the previous episode about frameworks, but TLDR is that. We are not big fans of frameworks in Go for multiple reasons. And I think it also helps with building a library when this is not a framework.

Robert [25:14]: For one reason, because it’s easier to adopt if it’s just a library. Because when you are starting to use some kind of framework, it requires more changes in your code. And I guess that many people may be also not…

Miłosz [25:34]: So it locks you in.

Robert [25:36]: Sorry?

Miłosz [25:37]: It locks you in, so it’s harder to migrate out of it. And we chose this way of being more independent. So it’s easy to throw a water meal away, if you choose to do so, and replace it with something else. And that’s my design, because we prefer working with smaller libraries that are not as heavyweight as frameworks.

Miłosz [26:13]: Maybe something similar is to how public API design matters. I think in Watermill the publisher and subscriber interfaces are quite good examples of how having a simple interface and the complexity hidden behind it can lead to nice API of the whole library. And we have the higher level components built on top of them. So it’s… quite… easy to use and fun to try. Because you don’t need to dig through the documentation to understand how it works. Use those two functions together and you say, oh yeah, okay, this is how this stuff should work.

Robert [27:09]: And I think it’s also cool because it’s giving you a lot of flexibility. Because we are giving you some building blocks in Watermill, but we know that there are a lot of different kind of projects. And it’s just impossible to create a library that will fit everybody’s needs. But if it’s built in the modular way, it can be adopted much easier to totally different use cases. So you can just use some of the components, use some more lower-level components, for some cases, some higher-level components. So to give you some examples, so you have in WaterMill, you have high-level interface for handling events that is handling marshalling for you, but you have also lower-level API that you can use, interact with PubSub more directly. So it just will have message with slice of bytes payload, and that’s it. And it may be a great choice for some low-level processing or data pipelines. It was hard people that they were using it in some strange cases, let’s say.

Miłosz [28:17]: So you can choose what to use, basically.

Robert [28:20]: Yeah, and it’s just making a list of use cases that we can support much, much bigger.

Miłosz [28:28]: And what about testing strategy? That’s also something we use, a unique approach.

Robert [28:34]: Yes, it’s interesting because if you think about the abstraction over PubSub, how it works in Watermill, so basically PubSub is implementation of one interface with two methods. PubSub publish and subscribe. So you can guess that publish method is responsible for handling publishing of your message, and subscribe. Yes, you guessed. It’s for subscribing for those messages that you just published to your topic. And every PubSub implementation that we have basically needs to implement this interface. And at some point, when we’ve been thinking about how to test that and how to implement many of the PubSubs, we found out that actually, okay, if every PubSub should implement this one interface, basically every PubSub should work in a similar way. So your publishing message with publish and subscribing, and it should run back. Obviously, it may sound like a pretty simple thing. At the end, I think we have probably like 30 different use cases, or 30, 50, I don’t remember.

Miłosz [29:41]: You can configure them depending if this PubSub supports them or not. The funny thing here is how you can adjust and adapt many different systems to implement this interface. Probably one unusual thing is the HTTP-based PubSub. So you can just do it over HTTP requests and responses.

Robert [30:03]: It’s basically webhook.

Miłosz [30:05]: Yeah.

Robert [30:07]: We have also PubSub, for example, based on the file system. I mean, io.Reader, io.Writer.

Miłosz [30:14]: So you can do it over files or whatever else you… We should probably build one over email.

Robert [30:25]: Yeah, it would be cool.

Miłosz [30:28]: But this goes back to this design of the API, right? If you choose this nice interface, the rest kind of falls in place automatically. And I see many libraries that don’t care that much about it. And you have these public methods or functions that take many arguments, and you have to check how they work before you start using them. And they could probably use some more high-level APIs that would be easier to use. So if someone is wondering if there’s still a place for some other libraries, I think there’s plenty.

Robert [31:14]: And there’s one cool thing about having this one test suite for every PubSub that we support. This is the fact that if you’re implementing a new PubSub implementation, basically you don’t need to write a test because tests are already there. And to be super exact, we have implemented, for example, Google Cloud PubSub at some point without having access to any production system that was working on Google Cloud PubSub. But we know that, OK, it’s cool because it’s hosted. And we implemented it basically just based on test. And I think a couple months later, we joined a company that was using Google Cloud PubSub. And we think, OK, let’s try if it works with that. And we used this PubSub that we just implemented based on tests. And it just worked on production without, and a big adjust. So I think it was just some context cancelling or some really minor thing, but everything else, it just worked as expected. So it’s a cool thing because it’s ensuring consistency about multiple Babsops. So we can say that we have one standard of how PubSub implementation should work. And when you’re implementing a new one, it’s just much faster to do that, and even if you don’t have access to any system that works on this PubSub.

Miłosz [32:29]: It’s just a nice exercise to try to bring your own PubSub implementation because of those tests. So now you have this nice system that tells you what the code you wrote is correct or not.

Robert [32:45]: It’s a bit like a quiz. You have a quiz and you need to solve multiple challenges to make it working.

Miłosz [32:54]: It’s also quite easy to do it, right? I remember the NATS PubSub we created. We did it over one day on a hackathon with a couple of friends. Yeah, because it’s just one interface. If you can implement it, then you’re done.

Robert [33:10]: So if, for example, you are using some PubSub that we are not supporting in Watermill, well, we are open to accept contribution, obviously. But the most important thing, your tests needs to pass and be stable. If they’re passing and they’re stable, well, we can talk. But we’re also happy to help with that because we know that in some systems, debugging is a bit harder. It’s also useful to know how to use grep and pipe your logs to text files because sometimes… Oh, implementing PubSub may be not that easy when SDK is not best.

Miłosz [33:49]: But it’s doable. Yeah, and you can start with the happy path. Because there are many different configuration options, usually. And once the implementation is successful, you have people asking for more features or changing the default behavior, stuff like this. But the HappyPath scenario is usually quite easy to do. So you can focus on this and later add the configuration.

Robert [34:20]: There’s another thing that I think is also important when you are building a library that people should trust in. And this is some policy of how you are introducing breaking changes. For example, Watermill is since, I think, six years in the major version V1. So in other words, we didn’t make any breaking changes in the core library. We did some in some PubSub implementations just because some SDK was updated or we had some good reason for doing that. But in general we are avoiding that and in the core we didn’t make any breaking change so obviously it has a downsize because we need to even if we’re deprecating some methods we need to keep them backward compatible keep old functions still working but from another side it’s very, easy to update for every people and i remember that i was using some libraries that were doing some breaking changes without bumping a major version. It’s not the best experience.

Miłosz [35:21]: Even with major bumps, people often tend to not migrate at all. Even if it’s just a couple of methods that change, it’s some effort. Especially if you use this library in some part of the system that’s critical, like messaging, or maybe it’s a part that is not used much, so you don’t know if something breaks. Then you are more likely to postpone upgrading. So, yeah, we want to keep the v1 promise.

Robert [35:52]: And important disclaimer, so no breaking changes, it doesn’t mean only that the public interface doesn’t change, because it’s also about keeping behavior as earlier. Because I would say that it’s even worse, because if you are breaking your API in a way that the compiler will say you that, okay, it doesn’t work, it’s fine because you will see the compilation error and you can fix that. But if you’re changing your library behavior, but without breaking interfaces, it’s pretty bad because you may expect some, consistency in how it works. And in general, you don’t want to surprise your library users.

Miłosz [36:35]: And probably the best way to avoid breaking changes is using configuration structs for the top-level methods. I think we had to bump the Kafka PubSub implementation to v2 very early, because we did more positional arguments and changing them is a breaking change always. Or you need to introduce a new constructor, which is annoying.

Robert [37:03]: Yeah, I think we also made the similar thing with SQL PubSub because we needed to add more parameters in some interfaces that we have in that library. PubSub. And by the way, if somebody is surprised, you have SQL PubSub. So yeah, we have SQL PubSub. So if you are in the stage that you cannot effort, let’s say, to have extra infrastructure piece or Kafka, RabbitMQ or whatever, you can just use Postgres or MySQL. We have also benchmarks and we checked how fast it works. And for many systems, it will be more than enough. but it’s just a bit off topic.

Miłosz [37:52]: On this topic, one more thing to add is to keep the V0.1 before V1 so you can experiment with the changes and people are aware this is an experimental version. But at some point, you probably want to release the 1.0 because it builds trust. If someone says your software is always in a beta release, they might not be eager to use it in the project they run. They will keep asking, is it production ready or not? But this buffer, which in our case was two years or something, quite a long time basically. But it was useful to experiment with different approaches, and then once we were ready to finalize the API, we could just froze it, and that’s how it stayed over time. And we kept adding more features, of course, more high-level components, but they all worked with the low-level API.

Robert [39:03]: So the tactic that we often use is just keeping the old functions or old types that we had there, marking them as deprecated, and suggesting to use another one. So a good example is our SecureS component for handling events and commands. And at the beginning, we have some super weird, crazy, complex interface. Now, it’s still there, but it’s deprecated. And we have nicer API using generics now that you should use. But if you are using the old one and you don’t have time to migrate, okay, that’s totally fine. It’s still working as it worked five years ago. So I think it’s pretty cool.

Miłosz [39:43]: And it wasn’t huge efforts to keep it backwards compatible. Most of the time, it’s not. You just need some deprecation commands and maybe keep some old tests in the repository.

Robert [39:56]: I think the biggest challenge is often how to name the new version. Because often you use the good name already and deprecate that and have some good names. I think we didn’t do that, so we didn’t put any suffix like v2 or whatever, but in probably worst case, you can do it. I think we have also this with the protobuf marshaller that we have in Secure S component. So I think the old one was the protobuf and the new one is proto. Not perfect, but probably a bit better than v2 suffix or something like that. Yeah.

Miłosz [40:35]: Naming and choosing a good API is one of the challenges definitely. Another challenge that’s kind of specific to Watermill is we support 12 PubSoaps right now. And most of those systems is quite complex. Like Kafka can be configured in many ways and used in many different scenarios. And we are not able to run all of those systems in production. We use some of them. So yeah, that’s one of the challenges here is to keep up with the community. So you have to keep it in mind that once the library starts growing and future requests are coming and you need to keep up and review them. And if you are not using some Pub-Sub, in our case, in production, it’s sometimes hard to tell if the changes make sense.

Miłosz [41:41]: In general, maintenance is important, because if someone sees a project that has not been updated since a few months, they might be afraid to use it again. Like, is it deprecated? Is it maintained? Because I’ve seen there is no updates. So regular updates are often a good idea.

Robert [42:06]: Yeah, I think it’s often also challenging when your project is kind of feature-complete. Like, okay, I wouldn’t say that WaterMill is feature-complete because we have still many ideas what we can add there. But from the other side, the features that are there are feature-complete. I mean, we can probably just not touch WaterMill for the next 10 years and it will be enough. We maybe just can update other PubSub implementations, but that’s it.

Miłosz [42:33]: It’s stable.

Robert [42:35]: Yes, yes.

Miłosz [42:36]: Yeah, but very often some security updates also make sense or library bumps, version bumps.

Robert [42:44]: Or license changes.

Miłosz [42:46]: Yeah, that also happens. So that’s one of the challenges. So it’s not only challenging to start, but then you also need to maintain it over time.

Miłosz [42:58]: Maybe to change the topic a bit, we should talk about marketing. Which is probably not the first thing you think about when you consider starting an open source project.

Robert [43:11]: Yeah, but as we already mentioned, running open source project library, it’s pretty close to building a product. So I think we’ve been lucky to already build a couple less or more successful products. And I think it also helped us to learn multiple things. But I think one of the most important things is that it doesn’t work. So just creating great product, great library isn’t enough for people to use that. So you need to let other people know that it’s great product, it’s great library, because it doesn’t work in the way that people will find it. No, they won’t. The chance is really, really low.

Miłosz [43:54]: Yeah, and there’s no easy way to track how popular the project is. GitHub stars are one way, which can be a vanity metric. It doesn’t tell you very much.

Robert [44:07]: You can buy them.

Miłosz [44:08]: You can buy them, yeah. But it also gives you some social proof. And in general, if your library is used, people will be more eager to use it as well. I remember once, when we joined the company, at the time when we released the 1.0, I think. I remember we wanted to introduce it internally to other teams. And there was some kind of internal library like this before, but it was kind of limited and didn’t have a nice API. So there was this meeting of senior engineers and this kind of meeting where you have 10 or 15 people that gather to decide a thing in one hour

Robert [45:02]: And it costs a lot of money.

Miłosz [45:04]: Yeah it never ends with a decision it often ends with another planned meeting and I remember we drifted from the main topic and just kept talking about which library we should use and then one of the principal engineers starts reading mail on his phone because he’s bored and he finds Golang Weekly newsletter and water mirrors featured in there and he says out loud guys you are in Golang Weekly and somehow this ended the conversation it was like a proof that oh yeah this library I’m sure it’s worth something Which is funny, because it didn’t mean some company used it on production or whatever. Maybe they did, but basically social proof is something you need to succeed in open source too. That’s why often projects also add some logos of companies that use it to the readme or website. Because then you go like, oh, okay, this company uses it, so it must be a good library. But of course, they don’t tell you that this is used by some obscure testing framework this company uses or something like that.

Robert [46:33]: Or it’s just used in some service that two people are using in this company. But, you know, it’s often like that.

Miłosz [46:41]: But long story short, you need to promote it, whether you like it or not. And the obvious answer is where to do it? Is Hacker News and Reddit? What other places you suggest? Maybe some conferences?

Robert [46:59]: Yeah, so this is the thing that, for example, I like to do a lot. Maybe some of you had the chance to see my presentation on some conferences. I really love to talk about watermelon conferences. Also on every bigger release that we have, we’re trying to write some blog posts, that we can share what’s new there.

Miłosz [47:24]: Also the examples are one way to promote it sometimes we have more complex examples and some blog posts around it, like the one with servers and events which also features Watermill, but it’s not only about Watermill, it’s how to do something very specific in your application.

Robert [47:47]: Yeah, and some of you may ask, all right, but I’m posting that and nobody cares and nobody’s avoiding that. And I think it’s often all about going back to the one of the first points that we have. So maybe it’s something that doesn’t resonate with people, because maybe they don’t understand how to use that, or maybe they don’t have use case for this library. And yeah, I think that this is part of why Water on You is that popular. It’s solving real problems that people have and is the key. Again, if people are not happy about using that, maybe it’s not solving problems that they have. Maybe you should try to find another idea that you can help people with a form. Maybe people don’t understand what this library is doing, what problems it’s solving. Quite to focus on use cases, on emphasizing what problems the library solves. And it will help a lot when you’re posting it to Hacker News or to Reddit.

Miłosz [48:46]: I was browsing GitHub today and I found a library with README and there was a big banner that says share your ideas for your use cases so I think that might be a sign you don’t know why this library exists so if you can come up with some real world examples featuring your library, that’s probably a good sign yeah,

Robert [49:14]: So watch out for looking for problems for solution that you have, because often it may be not the best thing yeah, All right, so we’ve shared a couple things about watermills, what we believed it was successful. I hope that it will be pretty useful if you’re building your own project or maybe you’re looking for some inspiration. If also watermill and eventview application sounds useful for you, but you didn’t have any chance to build such kind of system and you don’t know how to do that, Obviously, we have our examples, we have our documentations, but we also know that just reading examples, reading documentation is not enough for learning. So we already covered it in the previous episode, I think. But if you are looking for something about event-driven applications, you’re pretty lucky, probably, because twice a year we are running our online training about building Go event-driven applications in Go. And actually we’ll be starting in next three weeks. And it’s also a pretty special edition because we have a lot of updates. So it will be V2 of Go event-driven training. Breaking change. Yes, but yeah, it’s exactly breaking.

Miłosz [50:43]: What didn’t happen to Watermill. But sometimes it’s better to do it.

Robert [50:47]: Yeah, yeah. So what we have made, we’ve gathered all the feedback that we had from people from previous editions. and we spent a lot of time to implement them, and we believe that it will be better than it was earlier. This is also the first time when we will be doing it in the cohort way, so we will go with the content over four weeks with a group of other people. So let’s see how this form will work. We believe that it will be much better.

Miłosz [51:23]: And everyone who bought it before will have access to this new one, like we promised. So we get all updates for free.

Robert [51:34]: Promises are important for us, so we’re keeping that. One of the promises is that when you’re buying at this point, it’s the best price that you have all the time. So you can join in-house here, probably more or less, but it will be more expensive. So it’s definitely we encourage to join earlier. And you have also lifetime access for that. So it’s not like you need to finish it with this cohort now. You can do it in your own time. When you have time, you can do it now, in one year, whatever works for you.

Robert [52:09]: But yeah, it’s just available from time to time. It’s not available for the time. So if it sounds cool for you, just Google it. It’s named Go Event Driven. There’s a page about that. And then again, it will be available in three weeks. So stay tuned.

Miłosz [52:26]: And actually, Go Event Driven is also a nice synergy with Watermill. Because we feature Watermill in the training. Because it’s about event-driven applications. And it lets us show how to build event-driven applications from a high-level perspective. But it’s not a training about watermill so we don’t make money on watermill directly but it’s this synergy where we have this incentive to keep improving watermill to make the training even better so I think it’s a quite unique example of how you can take an open source project and connect it to something commercial but also So it’s not about sponsorships, or changing the license to closed source, or maybe even running an enterprise edition. Which often doesn’t work, because some cloud takes your source code and does it cheaper.

Robert [53:30]: And it’s also worth mentioning that we’re also not stripping documentation just to force you to buy the training. I mean, if you already build event-driven applications in other programming languages, Watermill documentation will be totally enough for you. You can go over the documentation, understand how Watermill works, and that’s it. Because some people may think that, oh, you need to buy training to understand Watermill. So it’s separate. And it’s also separate in the other way. So it’s not training about Watermill, it’s about universal patterns that works in every programming language. So we’re using Watermill because it’s making stuff easier, but we’re planning in the future also to create additions in other programming languages. Because again, there’s no training about Watermill. You can replicate it in any programming language and it’s universal knowledge that works now, worked 20 years ago and probably will work in 20 years.

Miłosz [54:28]: Well, we will have even given AI systems.

Robert [54:32]: Maybe AI will learn from our training. And as always, when we’re mentioning WaterMill, we always are sharing a big thank you to all contributors. We mentioned that we have more than 100. It’s hard to say because it’s across almost 20 repositories, and it’s hard to deduplicate and count them, but I think it’s more than 100 people across all repositories. Hard to say the exact number. But yeah, real thank you, because without you, Watermill wouldn’t look like you now, because we’re just two, we’re 100. So it’s really, really helpful.

Miłosz [55:14]: Right. Let’s move on to the Q&A then.

Robert [55:17]: Yeah, so if you have any questions about Watermill, about anything, just leave it on the chat, and we’ll be more than happy to answer them. Yes, I see that there is some comment about focus on me. Yeah, so we’re fighting with cameras to get the good settings, but yeah, I see. Probably it’s pretty good now, but light may be not perfect. Although, yeah, we are trying to improve every episode. Adrian is also asking, when creating a project, do you consider performance at the start of the project? Is it wise to make it run first, then think about performance later? Because performance plus design do go together right. What do you think, Niosh?

Miłosz [56:14]: Yeah, good question.

Robert [56:16]: It depends.

Miłosz [56:19]: So for what I mean, we have benchmarks. and that’s kind of objective way to measure how fast it is. But we never explicitly focused on performance, because most of the time it’s not that critical. Even for the SQL PubSub, which is super slow, we calculated that you can process, I don’t know, a few million messages per day or what was the count?

Robert [56:55]: Per hour probably, I don’t remember.

Miłosz [56:57]: It’s slow but still pretty fast for most use cases and I think we never had anyone complain about performance but it’s a good comment about the design. I guess one example could be if the design didn’t support batch requests, that could be pretty bad. That’s probably something to keep in mind. And that’s probably different in open source projects than some internal libraries. Because if you finalize the API and then it turns out it’s a blocker for some performance, action, then it’s a problem. So, for example, the publish of Watermill Publishers supports multiple messages in one batch. I guess that might be one example how it can help. If all you could do was publishing one by one, that might be a big factor for performance reasons.

Robert [57:58]: Yeah, but I think the most important thing is about thinking about use cases for the projects, because in some libraries it will just doesn’t matter and in some cases it may bother yeah.

Miłosz [58:09]: Very often it won’t matter

Robert [58:11]: Yeah for example if you’re building i know some ai stuff probably doesn’t matter because okay you can optimize every memory allocation order your struct fields to use as least memory as you can but when you just call lm it will be super slow and what you can do Oh.

Miłosz [58:29]: I know. Good example. Very popular in Go ecosystem is doing benchmarks for HTTP router libraries. And each library claims they are the fastest. And they have charts where you see that they are, I don’t know, faster in nanoseconds than the other ones. But the truth is, if you use HTTP, then you communicate over a network and each request takes, I don’t know, 50 milliseconds or 100 milliseconds roundtrip. Then this micro-optimization is not that important really.

Robert [59:14]: And probably you’re using JSON for communication. So come on, if you’re using JSON for your API and you’re talking about optimization, probably it’s not the issue.

Miłosz [59:25]: Someone will now say they have a very specific use case for this and they need to process billions of messages per second. And yeah, maybe it’s important. But for many projects, it won’t be important. Yeah, I guess advice is keep it in mind in terms of API, but probably don’t focus on it as the priority for the first release.

Robert [59:52]: So in other words, if you can optimize performance later without breaking changes, maybe it can wait for a while. Because it’s all about not introducing breaking changes. Like in Watermill, we can probably optimize a couple of things in terms of performance, but it will not make any breaking change, basically.

Miłosz [1:00:13]: Unless maybe you want to create a similar library, like one that exists already but that is faster or makes zero allocations or something like this, then it might be useful.

Robert [1:00:31]: All right, so the next question, when creating a wrapper around, say, GCP pops, what separates from lower-level implementation versus higher-level implementation? What typical feature should be included?

Miłosz [1:00:58]: Higher level implementation. Do I get it right that it’s about higher level GCP pubsub features? So basically, the Watermill wrapper around GCP pubsub is this… You implement this low-level interface only. So you implement the Publisher and Subscriber, and then all other components of Watermill work with those two interfaces. So you get the rest working out of the box. I’m not sure if this is the question, but the typical features are basically publish and subscribe. That’s the short story, but there are often very a PubSub-specific configuration you need to include. Okay, probably one very often feature that appears in most is consumer groups. Some kind. Or in the case of GCP PubSub, this is subscriptions. So you need a way to configure those. I’m not sure if this is the answer. So if you have a Yeah, if you have a follow-up question, you can let us know if this is what you meant.

Robert [1:02:24]: Yes, Adrian mentioned, true, at the end of the day, which HTTP package we use, it may not be that important. Yeah, it’s also worth mentioning that, you know, in this benchmark wars of HTTP routers or whatever, often the routers that are winning those wars are not fully compatible with HTTP protocol. It’s also worth mentioning. It’s cheating a bit, but for specific use cases, they are faster in it. But again, we are mentioning it very often that you can optimize allocations, struct, order, but just making one more SQL query will just kill all your optimizations. Again, there are cases when you’re not doing SQL queries, you’re just doing some CPU intensive operations.

Robert [1:03:20]: In this case, it totally makes sense to optimize. And if you’re a big library, just think about value kpi in this case and you can optimize later. And I think it’s also useful to if the library is when performance is important, just share benchmarks with people because, if they care about performance, it may be good. It may be just using the Go benchmark that is built-in. For these cases, it may work. For example, in Watermill we have special benchmark, because we are not measuring it in go-run time. We are more interested in how many messages we can publish or subscribe in one second.

Miłosz [1:04:01]: Yeah, I agree that the HTTP package is probably one of the least interesting things about the project I want to decide. I mean, I want a nice API and probably middleware support. But other than that, I want it to be like Watermill, so I can replace it with anything else if I want.

Robert [1:04:24]: If you feel that maybe you need to optimize your HTTP communication, maybe use gRPC, for example, that will be faster if you really need. Again, optimization when you’re using JSON for communication is a bit… Yeah.

Miłosz [1:04:42]: Yeah, I wouldn’t spend too much time deciding on the library, basically. Just pick something you know or you like, and that’s it.

Robert [1:04:52]: Right, so it seems that this is the last question so far. So let’s wait for a couple more seconds for questions.

Miłosz [1:05:06]: So probably a good time to remind you about the subscribe button or the five star writing button on

Robert [1:05:15]: Podcast applications or even better joining our newsletter so you will be sure that will not miss an episode notification so you can have notification before life one week before or summary and you know when you’re clicking subscribe it’s also cool but you are never sure that your notification will be there with our newsletter you are always sure plus you also know what all other updates that are doing uh what else so yeah five star reviews uh share with your friends so it would be also cool when this knowledge can go wider and if.

Miłosz [1:05:55]: You have any open source libraries share with us in the comments so we can take a look

Robert [1:06:01]: Or if you’re using watermill production also please let us know in comments it would be cool to know if it’s useful for you because again we don’t know basically we just know from some people that we’ve seen on conferences but, we don’t have any telemetry or whatever so we don’t know if you are using watermel.

Miłosz [1:06:22]: Right i think no more questions thank you everyone for joining us today

Robert [1:06:27]: Yeah thank you Miłosz for today. So see you in two weeks. In two weeks we are discussing what’s the difference between sync and async architecture. When we recommend to use that, we’ll also go deeper into the topic what’s the difference between PubSub and Message Queue.

Miłosz [1:06:46]: So wehave a very related topic in two weeks.

Robert [1:06:50]: Thank you for being with us and see you in two weeks.

Miłosz [1:06:52]: Bye-bye.

Unpopular opinions about Go

Wed, 30 Apr 2025 16:00:00 +0000

Quick takeaways

Simplicity isn’t enough for complex applications - while Go’s syntax is simple, complex applications still need proper design patterns; primitive code easily becomes spaghetti code in large projects.
Reading the standard library isn’t the best way to learn Go - it’s optimized for different goals than typical applications and might be confusing for beginners.
Router libraries are better than the standard HTTP package - libraries like Chi or Echo come with a nice high-level API.
Struct-based configuration is better than the “optional pattern” - structs are easier to document, discover, and maintain than the popular With-options approach.
There’s no one best project structure - starting small and evolving your structure as needed is better than following a dogmatic approach like the unofficial “Go project layout.”
Writing stubs by hand is better than using mocking libraries - manually written stubs are easier to debug and encourage better interfaces than reflection-based mocking libraries.
Code generation is better than reflect - for ORMs or dependency injection, it gives you compile-time checks and better performance.
Generics are mostly useful for libraries, not application code - while everyone waited for them, they’re rarely needed in typical service-level code.
Channels and goroutines can be overused - they add complexity and should only be used when concurrency is actually needed, not as a default approach.
Go’s error handling is fine for most projects - explicit checks make code easier to read, though built-in stack traces would be helpful.
Memory optimizations are often premature - micro-optimizations waste time for typical API services where network latency is the bottleneck.

Introduction

In this episode of No Silver Bullet, we share some of our unpopular takes on the Go programming language. After working with Go for eight years on all kinds of projects, we’ve seen many discussions about what idiomatic Go means. We talk about what worked for us, but we keep in mind that different projects have different needs. We question some common Go beliefs and share tips we’ve picked up along the way.

Notes

Wild Workouts - Our example Go project that shows a more complex application structure
Watermill - Our Event-driven application library for Go
HTTP Routers we recommend: Chi and Echo
Go in One Evening - Our hands-on Training for learning Go quickly
Clean Architecture episode - Previous episode that goes deeper into project organization
Go Developer Survey Results - Shows that ~75% of Go developers build API/RPC services
Google’s Go Style Guide - Many useful ideas, but be careful about being too dogmatic about it

Quotes

Go is simple, but sometimes it will end up as a disaster if you write too simple code for complex applications. You can write simple code, as long as we are building a simple application. But for something more complex, it starts to be more tough.

Robert

The common question in the community is ‘how to learn Go the best?’ One common suggestion is to read the standard library because that’s where the idiomatic Go code lives. But the standard library has completely different goals than a web application or API server or CLI application.

Miłosz

The biggest risk about having strong opinions is starting to be dogmatic and hard to be convinced by somebody else. We’ve seen people that were very strongly opinionated, and when you try to convince those people, it was super hard, even if your arguments were pretty logical.

Robert

This podcast is called No Silver Bullet because we believe there are no easy answers. To grow as an engineer, you need to keep an open mind. You can’t get stuck on a single fixed point of view because things get updated very fast What works in one context may not work in another.

Miłosz

When you create your application code, you definitely should not start with something generic. When you start with something generic, it will be over-abstracted and much more complicated than it needs to be. After a year, you’ll find out it didn’t need to be generic because you just have one use case.

Robert

Universal patterns exist for some reason. Probably someone wanted to solve some issue before. Design patterns are especially problematic - many people have this allergic reaction when you create a strategy pattern in Go and some people might react, “Oh no, this is not Java, please.”

Miłosz

Is structuring your project important? Ask a question: Will it help? Maybe it will not help. Maybe it doesn’t matter.

Robert

If the network roundtrip takes 100 milliseconds, why would optimizing the nanoseconds matter? Most of the time it’s a waste of time.

Miłosz

Timestamps

00:00:00 - Introduction

00:05:20 - Simplicity is not enough

00:10:55 - Don't read the stdlib to learn Go

00:15:38 - net/http vs other libraries

00:18:44 - Avoid the options pattern for config

00:21:48 - What's the best project structure?

00:34:38 - Avoid mocking libraries

00:38:53 - Code generation is better than reflect

00:44:32 - Generics are overrated

00:48:03 - Channels and goroutines can be overused

00:53:10 - Error handling is fine

00:58:58 - Memory optimizations are often premature

01:03:54 - BDD style testing libraries aren't helpful

01:09:15 - testify is useful for asserts

01:12:58 - Interface naming conventions

01:15:26 - Q&A

Transcript

Miłosz [0:00]: Do you know what is the best project structure? Should everyone read the standard library? And will error handling in Go ever improve?

Miłosz [0:07]: Go may be a simple programming language, but there’s a never-ending debate about what idiomatic Go means. Today we discuss our takes on some of the most controversial topics. I am Miłosz.

Robert [0:19]: And I’m Robert. And this is No Silver Bullet Live podcast, where we discuss mindful backend engineering. We spent almost 20 years working together across different projects and teams. We have seen Go evolve from its early days to what it is now. We learned that following advice like always do X or never do Y doesn’t work and it can limit your growth. In this show, we share multiple perspectives that will help you to make smart choices and grow into principal engineer level.

Miłosz [0:48]: And depending on your background and experiences, some of the opinions we share might seem as popular or unpopular. and this is fine. We share what worked for us before. So keep in mind, it’s our opinion, but let us know in the comments what you think, what’s your take on the topics and share your questions for the end when we will have Q&A and if you have any controversial opinions of your own, that’s great. We can discuss them together.

Robert [1:24]: Yeah, totally. would love to hear your unpopular opinions and include it to this episode.

Robert [1:31]: And yeah, probably it’s worth starting why this episode is about Go. So if you don’t know us for a longer time, Go is our language of the choice, let’s say. So I think it’s the first language where we stayed for longer and we feel that it’s, we feel productive with that and we’re quite happy about this language. But as every language, our technology it’s not perfect and there are some things that are not that

Miłosz [1:58]: Perfect it’s probably quite common story I’ve heard it a few times before but someone was moving from one language to the other and they stumbled on Go at some point and just stayed and maybe the reason is it’s a nice, get stuff done language so you know you can focus on what’s important and not about and the language specifics or features and so on.

Robert [2:26]: Yeah, but it’s definitely the best one from the worst. If we assume that every language has some downsides. But I think also probably many things that we’ll cover today should apply to other programming languages. So if you’re not writing in Go, don’t leave us, because you may also learn something that maybe can apply to your other programming languages.

Miłosz [2:51]: And we’ve been working with Go for about eight years now on production professionally. So everything we talk about, we base on this experience. And have we seen it all? Probably not. But we’ve seen different kinds of projects and teams and companies. So basically, the rule is simple. If we say that something works for us, It means we’ve seen it work in a few different contexts. But it doesn’t mean it is the same in every other context. This podcast is called No Silver Bullet because we believe there are no easy answers like this. Once again, show your perspective. Maybe there’s something we consider as a good practice and you’ve seen it doesn’t work. So that would be great to hear. Especially to grow as an engineer, I think you need to keep an open mind. You can’t get stuck on a single fixed point of view because things get updated very fast. So what works in one context in one project may not work in another.

Robert [4:13]: Yes, I think it’s important to be open-minded and not be dogmatic because I think the biggest risk about having strong opinions is starting to be dogmatic and hard to be convinced from somebody else because often we’ve seen people that were very, very strongly opinionated about something and when you try to convince those people it was super hard, even if your arguments were pretty logical.

Miłosz [4:40]: And on the other hand, there are people who start writing in Go and they worry that they are not doing it in the idiomatic way, which can be also a trap because it’s hard to tell what’s idiomatic without the full context. So it may be also helpful to hear different perspectives today.

Robert [5:02]: Yeah, and if you would like to, we already covered this topic pretty deeply in, I think it was first episode about low quality code. So please check this one. So we discussed more about this, what’s worth doing

Robert [5:15]: and when it’s worth doing shortcuts. Okay, I think we can start with the first unpopular opinion about Go, and it’s that simplicity is not enough. So one of the selling points of Go is that it’s a simple language, so some people are going pretty far with that and saying that, okay, Go is simple, you should write simple code.

Robert [5:38]: I kind of agree that I think we should write as simple code as possible, but no simpler, because usually it will end up as a disaster. So some examples that we’ve often seen was trying to build some complex applications with some complex domain logic, like a financial application, like a crowd. And okay, maybe it will work for a while, but after adding more and more logic, it will suffer a lot. And it’s not that language specific, I think, because, okay, we’re now talking in context of Go. Again, a lot of people are saying that it’s simple language, but I think it’s something that is happening in every language. So Go have pretty simple syntax, that’s correct, but if you are writing complex logic, it doesn’t matter if you are doing it in Go, in PHP, or Java. But if you approach it in a too simple way, as a crud, it will just start to be overcomplicated.

Miłosz [6:37]: Maybe the reason Go is more prone to this is people come to it from other languages, maybe some very complex projects where someone went too far with patterns and it was all overcomplicated. So when you start to write this simple syntax, it feels great, like a fresh air. So that’s pretty cool. But you can also go too far and try to do it. Trivialize it, write primitive code, let’s say.

Robert [7:10]: Yes, like none of design patterns are useful, just write simple code and it will be simple and nice. And that’s true, as long as we are building simple application. Because for simple application it will totally work. But for something more complex, it’s starting to be more tough.

Miłosz [7:28]: Design patterns are especially problematic. I think many people have this allergic reaction when you create, I don’t know, a strategy pattern in Go and some people might react. Oh no, no, this is not Java, please.

Robert [7:47]: Don’t make Java in Go.

Miłosz [7:51]: You probably don’t want to do it if it’s too early, but if you have an else-if structure that goes to 20 cases, maybe you could refactor it into a pattern that’s easier to read and maintain and so on.

Robert [8:06]: Yeah, totally. I would say, again, it’s important here to not be dogmatic. It’s important to be mindful, let’s say, know those patterns, not say like those patterns are stupid and obsolete. It’s not the case. I mean try to go with simple code, but know what to do if you have some more complicated cases to tackle.

Miłosz [8:27]: The universal patterns exist for some reason. Probably someone wanted to solve some issue before.

Robert [8:34]: And they were doing it like 50 years ago.

Miłosz [8:37]: In a language that’s not so far from on-go looks like right now exactly it’s still the same paradigm i think if you compare java a few years ago we’ll go it probably it’s not that far with syntax so yeah it’s a similar case, Maybe besides patterns, what comes to my mind is constructors. So I know some people also don’t like creating them in Go because, yeah, you can create just a type with proper zero value. And it often works well, sure. But sometimes you want validation. In web applications, very often you want the most entities.

Robert [9:22]: Yeah, and if you’re building, again, simple application like your team, if it’s three people, five people, okay, it’s totally fine. You can live without constructors. You can watch out to not break things. But if your team is 10 people, 20 people, 30 people, it’s starting to be critical because anybody can actually touch your code. And you need to have some way to ensure that this person will not break this code.

Miłosz [9:49]: And this validation will be in some place. The question is, do you want it in one place behind a constructor or do you want it scattered somewhere in your handler?

Robert [9:58]: And this is a good example. I mean, simple code to… People that love simple code may say that no, no, no constructors. And I know that it may sound crazy, but unfortunately it’s what sometimes we’ve seen.

Miłosz [10:15]: So simplicity won’t save you without the proper design. It’s possible to create spaghetti code in Go as well.

Robert [10:26]: Exactly. I think it’s an important point that Go is not different in this case. You can create spaghetti code in every language. Maybe in some languages it’s a bit easier because you can write obscure code. In Go it’s a bit harder to do. But again, you can still have code with very big cyclomatic complexity like City, and it will be also unreadable.

Miłosz [10:54]: Okay my next take is about ways to learn go and it is a common question in the community how to learn go the best and there are many many options but one of the common suggestions i’ve seen is, you should read the standard library because this is where the idiomatic go code lives, And I guess it might be because in the early days, there were not many open source libraries. It was hard to see some real life examples of Go. But I think for learning, it is probably not the best way to learn it. Because the standard library has completely different goals than a web application or API server or CLI application. or whatever you create.

Robert [11:49]: Yeah, probably you can compare it to advice like if you would like to learn driving a car, you should go to Car Factory and learn how to drive a car.

Miłosz [12:00]: Yeah, I would. That’s a good analogy. Unless you want to create a standard library for your own language, that is a great choice.

Robert [12:08]: Or your own car.

Miłosz [12:10]: And there are many great ideas in standard library, of course. So it makes sense to check how something is done. But I think if you are just starting out, it might be confusing because some of the code is not that readable. It’s optimized to be fast, to be generic, to cover all possible use cases. It has to be forward compatible because Go will never break the API, I promise. And, yeah, it’s probably not what you want to learn first. And one example might be that, speaking about constructors, I think the standard library quite rarely uses them, which might be fine because the zero value has some same defaults. For example, the default HTTP client works like this. You don’t need an instructor because the zero value is fine, which is a cool design. But sometimes you need to validate the inputs, basically, and it’s something you want in your application, probably. So I will be careful with this.

Robert [13:31]: I think the thing that you mentioned a bit earlier, the promise of backward compatibility, it’s making it a bit harder. So, if you’re making constructor, it’s harder to add another parameter because it will be a breaking change. So, we know it from the library that we are building Watermill, and sometimes it’s actually pretty tough to avoid some breaking changes. And yeah, if you’re doing a standard constructor, it’s just hard to add one parameter. But if you’re building web services or any kind of web applications, you basically can You can do breaking changes in your internal code because compiler will say that it’s a breaking change, it doesn’t compile, you can update all the code, but it’s not a standard library.

Miłosz [14:20]: So what to do instead? There are tons of resources and books, two of Go, documentation, videos and so on. We are a bit biased because we like learning hands-on, which we mentioned in the previous episode. So if you want to learn or check it.

Miłosz [14:38]: We also have the going one evening training where you can use hands-on approach to learn by writing code.

Robert [14:47]: And we know that a lot of people don’t believe that Go, you can learn Go in an evening, no way. But yeah, you can do that.

Miłosz [14:54]: Yeah, but if you like to just read instead, there are tons of good resources. And there are many, many open source projects right now that you can take a look at how they work.

Robert [15:04]: We actually have R1s also, Wild Workouts. It’s a pretty popular Y, I think, on GitHub. I don’t know how many stars it has now, like 5,000, 6,000, something like that. Also worth checking because you can learn from a project that looks like a project that you will do in your job probably as long as you’re not building a language or libraries and you can just see how to do it properly and not learn from your mistakes but from our

Miłosz [15:36]: Staying with standard library there is this opinion that the standard HTTP package which NetHttp is quite powerful and good to go compared to other languages, which I agree with. But I would also add that it is not that great to be used by default in most projects when you start. Because the API is not that great to work with compared to router libraries, and it doesn’t give you much to stick to the defaults, unless you care about dependencies size or something like this. So what we like the most is either GoChi or Echo, which gives you a very lightweight router out of the box. And Qi is compatible with the standard library. Echo is not, but gives you more sane error handling.

Robert [16:41]: It’s actually pretty compatible. I mean, it’s not using totally different types. You still have access to the standard library types. Yes, the HTTP handlers are not fully compatible, but you can have some magic adapters, let’s say.

Miłosz [16:54]: Yeah, so in the very beginning, we stick to 3 because we wanted to be compatible with the standard library for some reason. I’m not sure why. Probably because it felt idiomatic to do it. And then we switched to echo because the error handling was promising.

Robert [17:11]: In other words, I think it’s pretty cool that you can return error from HTTP handlers. Because from a practical point of view, it’s a very common thing that you’re returning error from your HTTP handler. And it’s just easy to mess it out if you not return from HTTP handler. So if you have checking if the error happened, but you will forget to return. It happened probably to any person and it’s not perfect.

Miłosz [17:37]: The error handling gets verbose otherwise. And I think once we try to do it ourselves, like create our own wrapper on top of G to have this error handling. But Echo does it for you, so that’s pretty cool. And maybe Barton to mention that those are just routing libraries, not full frameworks as some other. And we prefer it this way because it’s easier to replace or doesn’t interfere with the rest of your code. Just HTTP handlers.

Robert [18:09]: So we covered it a bit in the episode about frameworks. But TLDR is that you can pretty easily replace that. So if you decide that, okay, I don’t like Qi or I don’t like Echo. You can switch to any of this and it will take you probably less than a day for a pretty big project. So that’s cool. And I will be not afraid to be vendor locked in.

Miłosz [18:32]: Unless you couple it with your application. Hopefully not.

Robert [18:36]: Yeah, but you can also do it with the standard library. You spend some time.

Robert [18:42]: Alright, so next unpopular opinion from me is using the optional pattern for configuring things. So you know those with XYZ configuration instead of using struct. And there are a couple of reasons why we don’t like this pattern. So the first reason is it’s hard to find what options you can pass. So you need to go to file and…

Miłosz [19:07]: Sometimes it has the same package as the constructor. Sometimes it’s a completely different one called options so you need to look it up sometimes.

Robert [19:15]: You have two packages for example in cmp library for comparing stuff you have two packages where options are stored

Miłosz [19:24]: Many libraries also use options package for keeping those options which is cool but we have a big project with many dependencies and in your ide when you type options it gives you 10 different libraries.

Robert [19:40]: Yes, and I actually don’t have a good idea why this pattern is so popular.

Miłosz [19:45]: Yeah, I’m the same. I think I’m missing something. Why is it that much used?

Robert [19:52]: But long story short, what we are doing in, for example, Watermill, we are just using structs because it’s super simple. So you are just navigating to the struct. You see all the options that you can pass. You can add a method for set defaults, for example, so you can see what are the defaults. That’s it. It’s deadly simple, just works great. I’m not sure why it’s not the standard. I hope it will be, but…

Miłosz [20:16]: It’s easy to find the documentation. You just jump to the structure. You have all the fields there. What I also like to do is an IDE. You can have this command, fill all. So you just create empty struct, fill all and delete what you don’t need. Exactly. It’s very easy to configure then. and zero values can have the same defaults anyway. So it’s not that different from using the with options. It’s also easier to conditionally change some configuration. For example, if you have some input parameter, depending on which you want to use different options or not. If you use the options, you need to create a slice of options first, and then append to it. And with config, you just set the value and that’s it. And so if you’re just some fun of using this pattern in the chat, please let us know. What are we missing? Because maybe there’s some use cases that we didn’t see yet.

Robert [21:19]: I think we can just recommend using struct because again in Watermill we are using it with zero problems, zero backward compatibility issues. I think standard library is also not using this optional configuration pattern.

Miłosz [21:34]: A good example where simpler is actually the simpler option. I mean, when it’s useful to use a simpler one, just a struct.

Robert [21:44]: Yep, definitely. All right, so the next unpopular opinion is about structuring your project. And I think…

Miłosz [21:52]: Oh, most common discussion.

Robert [21:55]: Yeah, it’s a controversial thing. So I think… It’s the thing that structuring your project is the thing that a lot of people are often very dogmatic and blindly following some patterns that they don’t have idea why they are doing that. So I have one good example, the Golang standards project layout that a lot of people are following.

Miłosz [22:22]: Renamed.

Robert [22:24]: Yeah, yeah. And it’s quite funny because a lot of people are following that thinking that it’s official guideline, but it’s not. If you go to issues, there is one closed issue that Golang by trainers are saying that it’s not official. Maybe you should rename it or something because it’s misleading so this is one thing that i often found the other thing that is sometimes controversial is about using some more complex structure like clean architecture or hexagonal architecture because many people are saying no no it’s making java from go we touched it very deeply in the previous episode, I think, about architecture.

Miłosz [23:10]: Two episodes ago, yeah.

Robert [23:12]: OK. If you are interested, you can just search for it.

Miłosz [23:17]: And this is also a common question. I think maybe because of this empty page syndrome, when you start a project, you have nothing in your editor. So you feel like you need some foundation in place to start. And many people try to find the best way to start the project, to organize it. And you don’t really need to do it yet before you know how the project will look like in the future. So we recommend starting small and not sticking to any particular project structure if you don’t need it yet. Because it’s easy to go too far and overcomplicate it before it’s already needed. There are some nice, maybe not patterns, but some tips people use, like the cmd package for keeping all the binaries. I think that one is pretty cool, because it’s very easy to find the entry points to your application. Unless you create a library, maybe. I would probably use it in most projects.

Robert [24:28]: Yeah, it’s also good to remember that you can use internal package, So the package that you can only import from the same or deeper level of your application. So if you are building a library or some services that are shared in the codebase, for example, you have modular or monolith, it’s pretty useful to use this package.

Miłosz [24:49]: Some people call it PKG or SRC. I don’t really care that much, but I like to have this separate directory in the project, Because very often you have other stuff like Docker or Dev or Tools or API definitions. Basically some other files that are no Go files in your project.

Robert [25:16]: Yeah, and it’s starting to be pretty messy if you’re just keeping each of the directory.

Miłosz [25:20]: Yeah, especially for someone who is just joining the project, it might be very hard for them to understand where is the Go code, There is some helpers, some not ready to go files. So whatever you call this internal package, I think is a pretty cool idea. And it helps to find your way around the project. Also depending on the project type it might be a bit different right for applications for cli apps for libraries you probably will have a bit different structure, so it doesn’t make sense to stick to one everywhere you don’t need to look for one perfect solution and.

Robert [26:06]: I think it’s probably also not that important that as people are

Miłosz [26:10]: Exactly going.

Robert [26:12]: Into very deep discussions i mean it’s just structure it just directories you can change it pretty quickly that’s it

Miłosz [26:23]: But there is one package structure i dislike and it is the flood structure and, maybe it is also because of how the standard library looks like So I’ve seen many projects use one level of packages, and it’s supposed to be simple. But I think this is exactly this issue where trying to be simple tends to be something that’s hard to grasp. And I recall projects that have the same level, have packages like app, user models, database, handler, all kinds of names. And all those packages import each other in different configurations. So it’s quite hard to understand what’s going on. And again, especially for a newcomer, you just land on this project and you have to basically read it end-to-end to understand what’s going on. So that’s one structure I would not recommend, probably.

Robert [27:41]: And yeah, I remember that there are some people thinking that, okay, if you’re, I mean, that project structure can work for really simple services, and it’s fine. If you have really, really simple service, and it for some reason needs to be a separate service, it’s fine. But I think it can also lead to some kind of pathology when if you see that, okay, my service is too complicated to be in one package, I need to split another microservice. And it’s a trap because you start to have really granular microservices and will start to have more microservices than people in the team.

Miłosz [28:20]: Jim in the chat mentions that maybe we should read Robert Martin’s packaging principles. So I think this is clean architecture, basically what we covered in the previous episode. And yeah, this is what we usually suggest for more complex applications. And in short, it’s just grouping the packages in a few layers. So it’s easier to understand where’s your logic code, where’s your database implementation, and where’s your HTTP handler. And even if you look at standard library, there are some grouping packages like encoding for JSON and other encoders and HTTP in net. So it’s a similar idea.

Robert [29:04]: And I know that people are sometimes thinking that, okay, it’s actually not like that in standard library. In standard library, you have package per one thing. But yeah, I agree. that it’s totally compatible and it can be idiomatic in Go. It’s just about not going too far.

Miłosz [29:27]: Another package that’s often called unnecessary or forbidden is the common package. Which is an interesting idea because it’s hard to imagine a project where there isn’t common code. So I guess the standard recommendation here is to just move your code from common to where it belongs. Sometimes you can but sometimes it doesn’t make much sense and I think it’s pretty fine to keep this common package for stuff that don’t belong anywhere and I don’t remember it being a problem except for where you keep your entities and logic there that is pretty much it’s.

Robert [30:15]: Just you know this hard discussion what domain logic is and if it should be there or not. But I think what’s pretty useful here is thinking in terms like if you are building your common package, is it something you could publish and other people can use in totally different projects?

Miłosz [30:38]: Yeah, that’s a good idea.

Robert [30:39]: Yeah, this filter can be pretty useful for filtering out things that are your business logic of your application and are not. And sometimes also some people are recommending that instead of creating some shared code, you can copy it. And okay, sometimes it works, but it doesn’t work for things that should change together. Because there is sometimes some logic that should always work the same, and it will be not a problem when it will be replicated in multiple places. But sometimes you have common logic that you need to be sure that it’s working the same way across entire codebase. And if you forget to update it in other places, you may have a problem.

Miłosz [31:26]: Yeah, so maybe the issue here is not having the common package in itself, but deciding what goes there. If you just put too much in it and too many domain-rated stuff, then it becomes a problem. Similarly, you will create a common library that all services use. So I prefer to use the smaller libraries. And similarly in a package, you don’t need to put everything inside the common package directly. You can use sub packages. But I think this grouping is quite handy.

Robert [31:59]: And there is a related question. So if you have considered packaging by feature, not a technology layer. So I would say that it’s a different layer, it’s a different level of packaging. So usually in a more complex project, we have, let’s say, two levels of modularization. So one is by, let’s say, technology or layer that you have like application, interfaces, adapters, etc. And above that, you have like users, orders, okay, it’s probably a very bad example.

Miłosz [32:32]: More like modules, maybe?

Robert [32:34]: Yeah, yeah.

Miłosz [32:34]: I’m not sure if this is not about vertical slice architecture where you slice the project by each feature. Which I tried to play with it i didn’t come up with anything that i liked yet so I can’t really tell if it’s a great idea maybe in the future but.

Robert [32:57]: Uh you know also separating by um so i said that you know users orders etc it’s basically example because it’s actually in practice it often doesn’t work.

Miłosz [33:09]: It sounds too easy, too naive.

Robert [33:12]: Unfortunately, but I think the proper modularization is really close to separating by feature. You should separate your project by independent things that you can modify to add some functionality. So for example, you are modifying some functionality, you should not need to modify 10 modules for example for that. You should be able to do it in one module and it means that your modularization is proper. So I think that it can actually be close to that. Unfortunately, often it’s not the case. But from the other side, you don’t need to also have very granular modularization for simpler projects because it can just overcomplicate stuff. And I think I’ve… Okay, it’s hard to say because I’ve seen multiple projects where they’re too granular or they were… Well, they could be more granular because it was just building with the time with more features in one module. So, unfortunately, it’s sometimes hard to get balance here. Well, yeah, I think it’s something around that. So even if you have modules, probably it should be pretty close to features.

Miłosz [34:19]: Yeah, probably. I will try vertical slides at some point.

Robert [34:24]: Easier to say than do. Unfortunately.

Robert [34:29]: Alright, so time to next Antipolar Opinion, if you don’t have anything to add to the previous one.

Miłosz [34:35]: Let’s go.

Robert [34:36]: So, if you are with us for a longer time, you may know that we are not big fans of mocking libraries in Go. And there are multiple reasons for that. The first reason is that most of them, or I would say all of them that we tried are just hard to use because of a lot of empty interfaces, a lot of reflection. So because of that it’s pretty hard to work with that because you can basically pass anything to the assert or to the implementation and it’s really hard to debug that. And our solution for that is quite simple and I know that it’s a bit controversial, but our solution for that is actually writing stops by hand. And some people are saying that, but okay, it will be super hard, sometimes interfaces are super big, it will take ages to write those stops. I think I would think about that in another direction. Like if you have big interfaces, it’s not a problem of your stops. It’s a problem of your big interfaces.

Miłosz [35:39]: Small interfaces make it much easier.

Robert [35:41]: Yeah, yeah. And we already in some episodes were covering that it’s a similar problem with DI. So if your DI is complicated, it’s not a problem of your DI. It’s a problem of your dependency graph, basically. So the similar situation is with mocks. and okay sometimes in some projects where you are adding some tests it’s just hard to write those stuff by hand okay it it may be fine to write the use some library that can generate it for you it’s quite fine but i would recommend to try to write them by hand and i know that for some people it’s not obvious but in this case you can just keep this stub implementation next to the real implementation so thanks to that you don’t need to copy this implementation over multiple places so it doesn’t require much more effort to use that than using real mocks and at the end it’s giving you much more flexibility and it’s much easier to debug and that’s nice and

Miłosz [36:47]: I think it also encourages you to test your your code and not not the mocks What I don’t like about mocking is that many libraries give you those methods that you can test what method has been called and with what arguments and so on. And I think it kind of misses the point of testing because you don’t really care what some method, what some repository or client has been called with. You care about the end result. So I like to design those fakes like in-memory repository. That works kind of similar to the original one. It’s much simpler. And what’s nice about it is you can replace the implementation and then run the tests on a very high level and everything should work as a production. And you don’t need to care about what has been called, what hasn’t been called. If the end result is correct, then it’s fine.

Robert [37:47]: Yeah, it’s the worst if you are changing some logic and you are running your unit tests and everything explodes because you called one method that were not called earlier and you need to fix 100 tests just because you called something extra. It’s a disaster. And it’s even worse if you’re using your mocking library in the wrong way and it’s not thread safe. So if you’re running some tests in parallel, it explodes and you have no idea which test is failing. So yeah, if you wrote your stubs, well, the problem doesn’t exist. And if you are looking for some inspiration how to do that, so on our blog you can check article named the Go libraries that never failed us, 22 libraries you need to know. I know that it doesn’t sound like it covers how you should write stubs, but it’s actually There’s one example there that you don’t need HF library for mocking.

Robert [38:51]: Okay, another thing that I think it’s sometimes a bit controversial, especially for some people that are newcomers to the language. So this is the fact that in Go code generation is pretty popular, but I think it works pretty well. If you will take example of ORMs or dependency injection libraries. So it’s similar case like with mocks. So if you are not using code generations in many things like your ORM, you end up with a lot of empty interfaces. You will have no compile type check format.

Miłosz [39:32]: You have no strong types in a language that has static types, so you lose some of the benefits of using Go in the first place.

Robert [39:39]: Exactly. And for ORMs, it has also one important thing that you probably care about performance pretty much. And if your ORM is pretty heavy on reflection, it will just slow. You cannot overcome that. And if it’s generated, it’s just static and it’s fast. So it’s quite cool.

Miłosz [39:58]: Also easier to understand. If you really like to know how it works under the hood, you can open the generated file and very often it’s quite readable.

Robert [40:08]: Yeah. And a good example is dependency ejection. So for example, we quite like wire. So in our projects, we don’t use a library for dependency injection. We are just doing it by hand. But if you really need to use any of it, we recommend wire because it’s code generation based and you can just go to the generated file and see how everything is injected. So debugging of that is pretty simple. With reflection, good luck. No way.

Miłosz [40:34]: Yeah, it’s quite hard to understand how Reflect works, especially in a library you don’t know. So it’s quite hard to debug. And one other thing, libraries using reflect tend to overuse i’ve seen is struct tags so struct tags are pretty cool idea for marshalling it doesn’t get in your way and you know it’s easy to specify, but for other things i think it’s questionable because it gives you this this magic, behavior that’s not compile time checked one example is validation libraries i’ve seen some quite popular ones that let you specify so many things there, like how long you expect a string to be. Basically, if you look at the syntax they let you specify in a strike tag, it’s like a completely different language.

Robert [41:34]: I think when the fun starts, when you are using this kind of library and you are just not checking how long the string should be, It’s a fun start begins when you are checking something if some field is set.

Miłosz [41:48]: Yes, they also let you do it. So you need to write tests for it, which you probably want to do anyway. But I think if you use Go for its static types and compilation and all this good stuff, It kind of misses the point to overuse Reflect like this. And ORMs are similar with how they let you specify field types and relations in struct tags and so on. Feels like a cool idea because you have this one model that describes everything, but it can be overused. One of the things I like Go for is that it’s easy to read. You can take a look at the code and you understand what’s going on. There is no magic behind the scenes like in dynamic languages, where a function call can trigger something you don’t understand because there’s some decorator used somewhere or something like that.

Robert [42:57]: Yeah i have ptsd when you are saying about it

Miłosz [43:00]: Yeah i remember such stuff from python um or.

Robert [43:05]: Php you remember magic methods

Miłosz [43:07]: Yeah so yeah so if you use go i would stick to the, classic code as much as you can and avoid to reflect entirely unless you really want to do it. I remember one project when we tried to create our custom tags and be smart and be smart and avoid writing code by hand and yeah it wasn’t a good idea in the end, because just digging through the reflection code is not fun at all.

Robert [43:43]: Yeah so alternative if just write those ifs. It’s not that hard. Maybe it looks super explicit, but it’s not bad.

Miłosz [43:53]: Or generate your code, which is also quite easy to do with Go.

Robert [43:57]: But probably not for validation. I think for validation, I think it’s just a matter of a couple ifs that, again, it will look boring, but don’t be afraid of boring code. Boring code is not something bad it’s something good actually

Miłosz [44:13]: Yeah and with you know with good ideas and with things like copilot becomes much easier to to write boilerplate so it’s one less argument it shouldn’t take you much time to do it anyway definitely all.

Robert [44:31]: Right another thing that i hope it will be controversial because you may remember that one of the most anticipated features on Go were generics.

Robert [44:41]: And you’re here. We have generics for, I don’t remember how many years.

Miłosz [44:47]: Two years, probably.

Robert [44:48]: Yeah, and well, how often you are using generics in your service code? I don’t remember when I last did that.

Miłosz [44:59]: Yeah, not much. For some for some more internal libraries. I’ve used them. We use them in Watermill for CQRS handlers. We also use them in event sourcing library. That was cool. But I think for service level code, Probably not that much.

Robert [45:23]: And I think the secret may be in the name. Generics. So like the name states, it’s for some things that are generic. So for example, libraries that we mentioned. But usually when you are creating your application code, you definitely should not start with something generic. Because often when you start with something generic, it will be over abstracted and it will be just much more complicated at the end than it needs to be. And probably after a year you will find out that it didn’t need to be generic because you just have one use case that works in exactly one way.

Robert [45:58]: And even if you have two cases just copying this part of code and not having it generic it will be much much easier and it will make your code much nicer. But for libraries definitely it’s useful but I wouldn’t say that it’s that big game changer compared to how much people wanted to have generics in Go. And yeah it’s cool that we have it because the secure component in WaterMule, it’s cool because it’s making the syntax much nicer. I love it. But I would say that I don’t see that many more cases when it helps. And I’ve seen a lot of cases when it just makes code unreadable. Really, if you know some generic quirks, you can do such ugly code that nobody will have an idea how it works and how to make it compiling. It’s sometimes like when I play with TypeScript. I mean, I have no idea how to make it compiled, so I need to use compiled or whatever. But I’ve seen such kind of code with GoGenerics sometimes, so watch out for that.

Miłosz [47:03]: So it’s a similar rabbit hole as Reflect. It can be useful, but it can be overdone. So you can find yourself trying for a few days to make it work instead of just creating something straightforward, but for stuff where you want, It’s something universal, that might be a good idea. I remember one thing on the service level we had was decorators for application level commands. So we had stuff like metrics, logging and tracing decorator. And it actually worked pretty cool with generics.

Robert [47:40]: Yeah, but it’s also something that we’re able to do with code generation. True that it was a bit tough because if you are doing your own code generation, Unfortunately, it’s a bit hard to do, because there is some tooling, but it’s not that great, so…

Miłosz [47:58]: Yeah. One more feature that’s not always needed and can be overused are Tunnels and GoRoutines, which are one more great part of Go that everyone loves, I guess.

Miłosz [48:15]: And for running concurrent code, yeah, that’s a great idea. But sometimes I’ve seen projects where they were used for some unknown reason. Like instead of calling a function, you would have some worker in the background that would communicate with channels with some other function. And it may sound like a cool design, but remember that it also has lots of complexity, especially for stuff like synchronizing routines, closing channels, accessing some common memory and so on. So it can be also painful. And also the concurrent code is not always faster than the segmental code.

Robert [49:05]: Yeah remember that running a routine it also does some allocations

Miłosz [49:11]: And if your code is CPU-bound then you are limited by the cores anyway so it might not be the best idea.

Robert [49:18]: But i think it’s similar case like with generics so when you are building libraries channels may be pretty cool to simplify your life and build pretty complex stuff. But in typical application code, maybe it’s not that often useful. Sometimes it is, but often you can use, for example, error groups to do some synchronization. It also works nicely.

Miłosz [49:44]: And it has more high-level API. It’s harder to misuse. And with channels, it’s easy to have some unbuffered channel that makes you stuck at some point because no one is reading from it and so on. So you need more tests, you need to be more careful with design overall.

Robert [50:07]: That’s true, but it can simplify live with libraries. like an example of WaterMule. So I think it’s a nice example because we’ve been able to use a lot of patterns from Go that make implementation much easier. But it’s a library. I think we should do an episode about WaterMule.

Miłosz [50:24]: Let’s make it the next one.

Robert [50:26]: Sounds like a good idea. And one more thing about channels is that if you are using it in your application, you need to also remember that they are not persistent. So, for example, if you are triggering some workflow from your HTTP handler to channels and your service may die, for example, now you’re using Kubernetes, you never know. The service may die in the middle of processing your request. Your operation will be just lost. So remember that channels are nice, as long as losing this operation because it’s in memory will be not a problem. And if it will be a problem, remember that something like Message Broker exists, like PubSub, like Redis,

Miłosz [51:12]: RabbitMQ, Kafka. Or cloud-based PubSub. I often see people ask about Watermill and the GoChannel PubSub we have. So we have this PubSub implemented in a water mill where it communicates through Go channels, so we don’t need a separate infrastructure. And it seems like a good idea because you can just have it running with no other infra. But it’s also quite limited. It’s not the same as most PubSub are. We use it mostly for testing purposes or stuff like that. It might be tricky if you try to use it for some production-grade system, because there are some… Persistence is probably important, or another thing that comes to my mind is buffering. So if you use unbuffered channels or the buffer overflows, then your handler will hang. There’s many complexities you have to think about.

Robert [52:17]: Yeah, and if you are not building that big application, like you are building a smaller application and you have persistent file system, you can also consider PubSub based on SQLite, because we will very soon have it in Watermill. Thanks, Dima, for implementing that. And I think it will be a pretty cool alternative for channels, because you can persist your PubSub with SQLite, that will be super fast, and you can just store it in your file system. And it’s a nice alternative. If you are going into a more complex system, just use some production grade.

Miłosz [52:58]: Or use SQL-based PubSub, if you already have the database, you probably have.

Robert [53:02]: If you’re in the middle, let’s say, of your pop-up journey.

Miłosz [53:08]: Okay, my next one is will be either unpopular or super popular, depending on who you are. So my take is that error handling is completely fine. In go. The checks are boring to write, but they are great for reading the code, because you know what’s happening. And you can just use a compiler ID shortcuts to generate them. But one thing we miss is stack traces by default. That would be cool.

Robert [53:43]: That’s true. Because you always have this code that has no raps, and you’re later spending minutes, hopefully not hours, to try to find from where this error went to where it went. It’s tricky part. for stackraces, well, it could be helpful here.

Miłosz [54:02]: Yeah, I wonder if the next proposal will make it to the spec, because I’ve seen there’s another one for using some syntactic sugar. So, let’s see.

Robert [54:16]: Okay, in general, for example, I’m super happy about the error handling in Go because it’s explicit. It’s not like with exception, you’re quite not sure what can be drawn from the bottom of the stack. And in Go, you’re sure that, okay, this returns error. Should I do something with that? Shouldn’t I do something with that? And it’s super strict, and I think it’s making implementations much more robust.

Miłosz [54:43]: Yeah people complain about handling the errors but the other side is if you have function that returns no errors that’s actually super cool because you know it is you know pure function, and it doesn’t have side effects probably.

Robert [54:59]: Or for example have some function that you’d know that you don’t care that it returns error so for example sometimes you have cases when you are converting string to boolean but for example you don’t care if it’s drawing error if it’s just false. Okay, it’s false. Whatever, I’m just ignoring this error. But you’re doing it explicitly. With exceptions, it will be a big chance that some exception will be thrown. It will go up to the stack to your handler and it will throw to person 500 just because you were not able to parse bullying.

Miłosz [55:28]: Like a panic. What do you think about panics?

Robert [55:31]: I love panics. So I know that a lot of people are like, don’t panic. Yes, yes, I know. But I think it’s not that easy. So there are many cases when you shouldn’t panic. You definitely shouldn’t panic in goroutine because i’m not sure if all of you know but if you’ll panic in your goroutine and you will not recover it your entire process will exit with non-zero code not the best thing in http servers not gracefully

Miłosz [55:57]: Shut down but just exit.

Robert [55:59]: Yeah yeah so it’s if you are using channels for your asynchronous operations it will be lost so it’s not the best thing. In general, if you’re writing libraries, for this reason, probably you should watch out when you’re panicking because you can just accidentally do this favor of killing service for somebody. And it’s also pretty hard to debug. So not bad. But there are some cases when I like to use panic. So obviously in some dirty code, it’s nice. So for example, in my IDE, I have shortcuts to do if error is not nil, panic. It’s pretty useful. But there’s one case when I really like panics and it’s a production code and it’s related to dependency injection. So if you’re injecting things and when running some constructor in your main, return some error. And I think it’s totally fine to panic in this case because it’s a non-recoverable situation. So if you’re not able to inject dependency, your service will not work. So it doesn’t matter what you will do at the end. And panic is nice because it will give you the stack trace and it’s making it a bit shorter. And I think in general, this is the situation when panics are good. So when you have some non-recoverable situation, basically.

Miłosz [57:19]: Yeah, sometimes also for a better API. We have a function that’s very often called and handling the error there would be annoying. And there’s just one error you need to check inside it, but it’s basically a sanity check. It never happens, but just for completeness, you need to do this entity check, then I think it’s pretty fine to…

Robert [57:48]: I no longer make errors like it should not happen because it always happens.

Miłosz [57:54]: What comes to mind from our project is about getting some config value inside the function.

Robert [58:03]: Yeah, and without this config, it will totally not work.

Miłosz [58:06]: So in theory, the config library should handle this for you, but you never know. So why not just throw a panic in this case instead of making your API harder to use in this case?

Robert [58:20]: Yeah, exactly.

Miłosz [58:22]: But it’s an exception, not a rule.

Robert [58:26]: Always having exceptions in Go. Yeah, but again, I think what’s important here, again, not be dogmatic, because I think a lot of people are dogmatic that don’t panic, never ever panics are evil. But I think it’s not like that. There are some cases when it’s fine but just be mindful when you are doing it.

Miłosz [58:48]: And remember about recovers on many levels.

Robert [58:52]: Yeah, definitely.

Miłosz [58:56]: Cool. My next one is about memory optimizations or optimizations in general. So So, I fancy discussions like this, like knowing what makes more sense, what’s a better trick. One example is struct field ordering. So if you order fields in your struct in a certain way, then they fit better into memory and it takes less memory allocation.

Robert [59:26]: And you feel so smart. I just ordered my struct field to use a bit less of memory.

Miłosz [59:33]: Yeah, and there are many such tiny improvements you can make, like pre-allocating slices instead of appending in a loop, let’s say, if you copy a slice. Or using a buffer builder instead of appending to a string. like our, yeah, my other ideas. No. Oh, yeah, I know. The string with map that maps to an empty struct instead of a boolean.

Robert [1:00:10]: I actually remember one more. So concatenation of strings with this function. I don’t remember the name. But to not concatenate strings with plus, but instead using this…

Miłosz [1:00:23]: The builder. Yeah. And, you know, GoTest has this benchmark feature that lets you write two functions and compare them and you can easily see, like, wow, this is five times faster. But, you know, if you take a look at the Go survey, I think 75% of people write API services in Go or web servers or something similar.

Robert [1:00:50]: And what’s the bottleneck in this kind of servers?

Miłosz [1:00:53]: Yeah. So, you know, if the network roundtrip takes 100 milliseconds or 50 or something like that.

Robert [1:01:01]: Let’s take one millisecond even.

Miłosz [1:01:04]: Yeah. So why would optimizing the nanoseconds matter in this case? So most of the time it’s a waste of time. And of course, a huge disclaimer depends on what your application does. If you write some, I don’t know.

Robert [1:01:20]: CPU intensive or memory intensive application.

Miłosz [1:01:23]: It’s tough for microcontrollers or whatever, of course, it matters in many cases. But I’m talking about when you create an HTTP handler and then you spend time discussing this in a PR, whether this should be pre-allocated, sliced or not. And it’s probably a waste of time.

Robert [1:01:41]: Or maybe you are also writing a service that is handling really, really huge traffic and your servers’ costs are important, it may be also the case. Because sometimes it’s the case. But again, from our experience, it’s a smaller percent.

Miłosz [1:02:02]: But the difference is you optimize once you feel the pain. Instead of trying to predict that one year later we will have such huge scale and that we will need this optimization.

Robert [1:02:12]: And I think it’s useful to actually use this benchmark in opposite ways. So if somebody is suggesting you like, oh, no, you’re wasting such much of memory or CPU. You can use benchmark and show that it’s just that small difference and compared to, for example, SQL query, it’s like nothing.

Miłosz [1:02:34]: Yeah, maybe just add an index in your table and it will be much, much faster.

Robert [1:02:39]: Undo one less SQL query. Or maybe if you’re calling some external service, maybe do it in Go routine if you can. and just do things in parallel. It will give you probably a couple orders of magnitude more and then optimizing some allocations.

Miłosz [1:03:01]: A similar concern is about garbage collection often. And I’ve seen it most often in regard to games. Like this opinion that you can’t make games in Go because of garbage collection. But chances are you’re not creating a game or even matter. And it turns out it’s actually pretty fine. There are many games created in Go that are all fine. And most of the time you don’t need to care about it that much. Unless, of course, you build something that’s very specialized. But I’m talking about a very common scenario of creating API or HTTP servers.

Robert [1:03:39]: So again, be mindful. Just not apply some dogmas blindly.

Miłosz [1:03:48]: Yep.

Robert [1:03:52]: So maybe one more thing about testing again. So, the BDD style testing libraries are not helpful, or I would say even more, they are harmful, for multiple reasons. So the first reason, I think all of them are not compatible with standard libraries. So they are kind of using some testing from standard libraries, but in general they are not compatible. So you need to learn some new tools, how it works, and your assets will be not compatible. The new people in the team will need to learn something new, and it will be just less readable.

Miłosz [1:04:31]: They have nice outputs, though. That’s a selling point most of the time, I think.

Robert [1:04:37]: That’s true. But I think one of the biggest arguments for using this kind of frameworks is that it’s making a nice structure of your tests and I agree. I think it’s nice that you have this given, when, then, and it’s nicely separated. But from the other side, you can achieve the exactly same concept in standard library. Because nobody is saying to you that you cannot split every of your tests to three parts. With given, when you are doing some prerequisites, in when you’re doing some operation and at the end you’re ensuring that everything, what should happen. And I think it’s, I’m trying to do that because it’s making tests much more readable. But again, I don’t need to have special library for that. I can just do it with the standard library testing. So I think the most important thing is mindset. And it may be actually a good idea to maybe try to use this kind of libraries to try to understand this mindset. But I would avoid to use those kind of libraries in your services.

Miłosz [1:05:46]: The big promise is using Garukin syntax to collaborate with your product owner or maybe some stakeholders or whoever. Yeah. We tried it before.

Robert [1:05:59]: Yeah.

Miłosz [1:05:59]: But we never made it to work really. So the idea is that you can create this framework for someone non-technical, I guess, to create the test cases for you in this given window scenario. And then you can just run those tests.

Miłosz [1:06:19]: But I have never seen it work out in practice. Usually it’s developers who write those scenarios anyway.

Robert [1:06:27]: Yeah, and you’re just wasting a lot of time on doing the mapping because you need to have parsers for this text, you need to do a lot of mapping. And at the end, when developers are adding those I would say now three times more time because you need to support that many things. Again, you can have very similar concepts in your standard Go library code tests. And again, maybe somebody like your product owner will be not able to edit that, or probably your product owner will anyway not write this. And if you follow those principles, you can even sit with your product person and if you did a good job with writing nice helpers, it can be even readable with your help to this person. So I think this actually worked for us a couple of times, so you can see with this.

Miłosz [1:07:20]: If it reads like in English, you can probably just show it to someone who’s not technical and they will be able to read it and suggest what’s missing.

Robert [1:07:29]: So again, I recommend to play with that, to maybe understand those principles and see how it can look like and give you some inspiration. But from our experience, it never fulfilled both promises that it gave.

Miłosz [1:07:45]: It was usually easy to start. And I think most examples in Garek can show you this very promising idea. You have this table with quite simple use case. We have something like, when I do this and do this, then something happens. But once you try to do something more complex with it, you need to add more cases and it starts to be kind of bloated. Hard to understand.

Robert [1:08:15]: Yeah, because Gherkin works nicely for some really high level tests, like end-to-end tests. But often it’s not enough to have some HTML selectors in Gherkin because you need to match the element that you need to click to do something. And it doesn’t make sense at the end. So you just

Miłosz [1:08:31]: Keep switching between the scenario and the test code to make it work together instead of just doing it in your tests.

Robert [1:08:40]: And parsing is harder and harder when you need to use some special characters.

Miłosz [1:08:46]: Yeah, well, maybe it’s worth to take a look and try to at least to inspire yourself to write tests in this given one then approach. That’s actually a pretty good idea.

Robert [1:08:57]: Yeah, and I think I would be happy to see it much more often because it’s actually not as often used as it could be.

Miłosz [1:09:09]: Okay, maybe the same with testing libraries. I see a few people quoting Google’s style guide which mentions that you shouldn’t use assertion libraries. But we actually like Testify for asserts and it’s pretty easy to use, doesn’t get in your way. Most it does what you want.

Robert [1:09:34]: And I think we can probably also say that it’s industry standard. In most companies it’s just used, in many libraries it’s also used.

Miłosz [1:09:46]: Yeah, and that’s the thing about following those style guides or official guidelines. I think they are helpful, there are lots of useful ideas there, but it’s also easy to take them too seriously and dogmatic. And you need to remember you are not Google. And the best example probably is how Go started with GoPath, which worked well for Google because they used the monorepo and so on but for everyone else probably almost it was painful and weird to work with Dependency management?

Robert [1:10:20]: Why you need that? This is everything on our monorepo

Miłosz [1:10:24]: It was this vendor, yeah I’ve never seen why using Testify would be a problem it gives you nice outputs mostly out of the box you can add some extra context if it’s missing.

Robert [1:10:43]: I think of an argument is that it’s implicit, you need to learn how it works under the hood and I agree but from other side you will do it once But for something like equals,

Miłosz [1:10:55]: We don’t really need to understand how it works.

Robert [1:10:57]: It’s just equals, basically. And you have to remember that sometimes you need to use equal values, for example, if you’re not asserting exact types, for example.

Miłosz [1:11:06]: Yes, there are some caveats, but maybe it’s one library where actually Reflect is fine. But it’s a testing library, so that’s a bit different.

Robert [1:11:16]: It’s a library.

Miłosz [1:11:18]: Yeah, you don’t need to care about compile time checks in your tests because it’s just testing your code.

Robert [1:11:24]: That’s true, that’s true. So again, I think it’s good to have exceptions from…

Miłosz [1:11:31]: Guidelines, yeah, and maybe make it work for your project, your team. And not be dogmatic. Yeah, agree on something together, which is also a nice idea to have some internal guidelines instead of just committing 100% to something external that maybe is generic and works fine in many contexts, but for your specific use case, maybe not.

Robert [1:11:57]: And I think that there is one thing in Testify that I don’t like. So those are testutes. I’m not sure if it’s still valid, but last time when I tried to use that, testutes have one big bad limitation. so they were not able to run tests in parallel. So basically, your tests needed to run sequentially. And from my perspective, it’s a deal breaker because many of the tests just need to be able to run in parallel when there’s any I.O. or any slow operation. And without that, your tests will be just extremely slow.

Miłosz [1:12:32]: Right, so we had to use the test main pattern.

Robert [1:12:36]: Exactly.

Miłosz [1:12:37]: But some years passed, so maybe we should take a look at it again.

Robert [1:12:40]: Yeah, and again, sometimes it’s just a matter of running some helper function before or at the beginning of your test, and it’s fine, and using subtests from the standard library. And I think our last unpopular opinion, so it’s about naming. So you may read in, I think, Effective Go probably, that you should name your interfaces with suffix like error, like reader, writer, follower, user finder.

Miłosz [1:13:21]: It makes sense most of the time, but sometimes not. yeah.

Robert [1:13:25]: So for example if you have interface that have four methods it can lead to something strange

Miłosz [1:13:33]: And i’ve seen people overuse it sometimes like you have a method called user by id and then the interface is user by idr come on can you just call it user finder or.

Robert [1:13:46]: Repository i mean it’s just user repository.

Miłosz [1:13:50]: That’s probably a very minor thing, but I’ve seen it overused sometimes.

Robert [1:13:56]: That’s true. I think it’s again about this being mindful when you are writing software, and if you see that you have an interface named user by IDFIR, it’s something wrong there because it’s just hard to read it.

Miłosz [1:14:12]: Yeah, exactly. It won’t tell you much about what it does. If you just stick to this pattern everywhere. Maybe just come up with something that describes it better, even if it doesn’t follow this idea. And my final one about naming is, I don’t care if you say Go or Golang, which for some reason is controversial. I don’t know. It’s like the common discussion. Someone says, how to do X in Golang? Someone comes in and says, first of all, it’s not Golang, it’s Go. I don’t know why. I don’t care. Everyone understands what it is.

Robert [1:14:59]: That’s true. But I think it doesn’t help that language name is not that unique around everything.

Miłosz [1:15:07]: Hey, it’s not the worst name. It could be called JavaScript. Does it make sense? No. Naming is hard. Everyone knows it. Just live with it.

Robert [1:15:24]: All right, cool. So let’s see what questions we have on the chat. All right, so Jim mentioned that obscure is not spaghetti, and spaghetti is not cyclomatic complexity. And yeah, that’s right. I think maybe we mentioned that it’s close to be equal, but no, it’s not, I would say that it’s maybe some shortcut, but it’s often connected in some magical way that when you have obscure code and cyclomatic complexity is big, there is a big chance that it may spaghetti code but yeah it’s

Miłosz [1:15:59]: Yeah i guess those are separate things obscure will be hard to understand spaghetti would be many side effects that you don’t see something similar but maybe jim you can tell us more what do you mean what do you mean by this uh but yeah there are many aspects for sure and this comes to this discussion about simplicity right so it’s it’s hard to, to decide what’s simple because of it, because there are many aspects.

Robert [1:16:32]: Another comment from Jim is that dry is tension with loose coupling. Any removal of the application implicitly creates coupling that didn’t exist and may be a source of future problems. And it’s true, but sometimes things are coherent. So basically if things are coherent you would like to have them coupled because it’s kind of expected. So if you have one logic that is expected to work everywhere in some way in every place. Well, unfortunately, it’s coupling, but you cannot close your eyes and say, oh no, there is no coupling there and just copy it everywhere because we’ll have a lot of problems with keeping it up to date everywhere. Sometimes maybe it shouldn’t be some library code. Maybe it should be in a service. It’s hard to say because there are many examples, but I would say that I agree with that, but there’s sometimes also one more complexity Sometimes things are just coupled because they need to be coupled and it’s making your life easier. So I would say that coupling is not always bad. Coupling is sometimes coupling and it exists and we cannot ignore that.

Miłosz [1:17:43]: Yeah, but if you overuse dry, you can end up with too much coupling. I think that might be the point here. And yeah, duplication is the best way to lose the coupling as a tool. We have a blog post about this as well, on our blog. And yeah, we’ve also seen that overusing dry is usually a bad idea, unless you know what you’re doing.

Robert [1:18:10]: Yeah. And I think it’s also one tricky part here that we’ve seen a couple of times, that it’s not silver bullet that you can always get common logic in library and just use it everywhere, or you can move it to some services and service can handle that. So in other words, sometimes it’s better to have a service that is keeping this common logic and it’s good, and sometimes it’s better to keep it in the library that is shared across. Because sometimes we’ve seen migrations in 2A. So for example, there was some library, some common code, it was moved to the service, just to figure out that actually having a service handling that is the worst idea, it’s better to have a library that keeps this logic common somewhere. But again, it unfortunately depends. Another comment from Jim. So yes, the focus on testing should be almost always on results, not with internal functions or code.

Miłosz [1:19:19]: Yeah, yeah, that’s why be careful with mocks and at least we prefer stabs or fakes instead of mocks for this. And black box testing.

Robert [1:19:34]: Right. Strange combination of opinions. Love panics, but hate exceptions. Love explicit error returns, but one stuck traces.

Miłosz [1:19:43]: I wouldn’t say we love panics. They have some uses. We don’t use them for normal errors.

Robert [1:19:52]: I hope it didn’t sound like that.

Miłosz [1:19:54]: That’s maybe the difference. Exceptions are just used everywhere in some languages instead of errors. And panics are just for the very exceptional for the exceptional cases not for errors yeah.

Robert [1:20:11]: And fact races so I would say it’s a matter of pragmatism that well, error wrapping is cool but what you’ll do if somebody forgot to do that and it’s also not perfect that you need to do that again, co-pilot helps with that sometimes those messages are super stupid but the reality is that But spending hours on figuring out the wrap strings, it doesn’t make sense.

Miłosz [1:20:38]: Maybe one of the issues where it shows is if you use a library that doesn’t wrap errors and you receive some obscure error from it, and then you have to look at the sources and find what it is and understand what’s going on. And what stack trace could help you. And it doesn’t need to be stack trace of all the levels, just of the most inner level, where the error originated. So you see the previous function calls.

Robert [1:21:14]: Yeah, and I think I talked to somebody from Go team about the problem of stack traces with errors, and I think one of the problems was performance.

Miłosz [1:21:27]: And that’s again the issue of standard library it needs to be universal yeah yeah probably in some projects it wouldn’t make sense to allocate memory for it so yeah that’s but.

Robert [1:21:40]: Yeah maybe someday because from other side we have the runtime tracing that it has it and it’s super fast. The overhead is like 5%, 1%, I don’t remember. Maybe someday. Please, please, please. Mate of Clive saying that I feel the least thing dev use Go is building web services. I always find jobs for DevOps engineers.

Miłosz [1:22:17]: So I mentioned the results of the Go developer survey in the official one. And if you take a look there, the top one is API servers, about 75%, and some other high percent for servers returning HTML and such. So maybe this means everyone writes web services, but they have no jobs.

Robert [1:22:45]: Yeah, but probably it depends.

Miłosz [1:22:47]: Yeah, I think this is an accurate observation that there are not that many jobs where you can see Go used for web apps.

Robert [1:22:56]: For other words, how many Kubernetes operators you can write? Come on so I think it may depend a lot on the market but yeah our observation is kind of consistent with the survey that we mentioned that I think most of the jobs were around services but again I think it may be also depending a lot on your bubble so if you are more into DevOps bubble you may see more jobs postings actually

Miłosz [1:23:26]: It made me think because it’s true that But I don’t see many jobs about using Go for web apps in general. So what do all those people write that reply in the survey about web services? What do they do? Maybe just some open source apps? I don’t know. But at least, yeah, maybe we are biased, but we mostly write to web apps in Go and API services. We don’t do any DevOps stuff.

Robert [1:24:05]: And we didn’t struggle to find any web services, probably it’s better words. Again, but it depends on the bubble a lot. Probably it depends a lot on your local market. If you’re looking in the local market remotely, remote is different. Okay, and the last question so far. So if anybody has another question, we’re waiting for a while. And the last question so far is saying I joined kind of late to this podcast and I’m not sure if you mentioned about naming convention in Go using name variables with one letter or three as maximum I mean code should be readable I think it’s actually a pretty good one we’ll cover this one what do you think?

Miłosz [1:24:50]: Yeah, code should be readable I agree, it’s hard to give some rule here here. Of course, short variables are nice if it’s used in a small loop or something like that. Sometimes it’s easier to read one word than four words together, but I wouldn’t have any strong role here like three letters max. That’s a bit crazy.

Robert [1:25:24]: Yeah, but I think sometimes it may improve, my opinion, readability a bit, but I think it’s all about not overusing that. So sometimes when you are… obviously, if you have struct receivers, it makes sense because you know that you are within some struct and you have receiver of the struct, so you know that S is, for example, service or something like that and it will just make a lot of code much shorter but if you are using all variables that are at most three characters length it may be not readable so i think it’s all about the context so if you see that in this context you know this small shortcut what it means it’s fine but it’s all about not going to extreme i

Miłosz [1:26:09]: Think there are some guidelines like this in effective go maybe or some other document where it’s like this recommended exactly as you said. Depending on the context, just longer or shorter variables. If it’s a long function, and you use the variable across it, many times maybe longer is better, so it’s clear what it does.

Robert [1:26:29]: Yeah, so I think it’s about some common sense. So if you see that it’s not readable, just use the entire name. And I would say that in most cases, we’re using the entire name to make code readable, but we’re also not afraid of using shorter ones.

Miłosz [1:26:42]: I don’t like rules for stuff like this, and a similar one is what I actually like from Google style guide is there is no definitive rule for maximum line length. It just says, you know, try to refactor if it’s too long, but it’s not something like you should use 80 characters per line. Because then you have this fictional, artificial rule that everyone tries to apply but doesn’t help anyone. It seems similar with short names.

Robert [1:27:19]: I think we covered it in an episode about writing low-quality code. At the end of the day, ask a question. Will it help? Maybe it will not help. Maybe it doesn’t matter. Safety doesn’t matter. Whatever. if it’s good if you have enough time to think about such things you have a lot of time if you

Miłosz [1:27:42]: See variables like A, B and C probably something is wrong we can probably improve it, you can also go to the other extreme and call everything by too long names which is also not readable that’s true, I agree that code should be readable And everyone has their own opinion about it, so of course that’s hard to tell.

Robert [1:28:09]: Yeah, so I think this readable, it’s something in the middle. So you’re just not going like writing off your state code with one character length of variables. But sometimes you can also just use those short ones and it can really, for example, for receivers.

Miłosz [1:28:32]: Yeah makes sense.

Robert [1:28:33]: Okay so jim is saying that he appreciate that balance in your commentary too much of go community ime in my experience oh makes sense quite simple with simplistic thank you thank you to jim for your comments thank you

Miłosz [1:28:49]: Jim i think that yeah the simple with simplistic comparison is a good one like simplistic might be too much simply is fine.

Robert [1:28:56]: Yeah yeah definitely cool so we have no more comments so a couple more seconds for last comments if you have any yeah

Miłosz [1:29:07]: So maybe we can mention if you know if someone’s listening to this and doesn’t know go yet or anyone would like to try we have going on evening training on our platform where you can learn by writing code.

Robert [1:29:19]: And it’s not our opinion but some people are saying that it’s the best thing that they did online for learning stuff, so I don’t know. Hopefully. And yeah, it can be one evening because Go has simple syntax and we did really a lot of work to compress the most important knowledge that you need. So you don’t need to read the entire Go standard library to learn to Go code because it’s not needed usually. There are some things that are most useful. That we know from the teams that we’re running because a lot of people actually that joined those teams never worked in the Go earlier and they needed like one week to be totally proficient in Go. Yeah, yeah. So it’s cool with Go.

Miłosz [1:30:08]: Yeah.

Robert [1:30:09]: It’s not perfect language, but no language is perfect. And I think Go is closest to this perfect one. So I recommend it if you don’t know Go yet. Cool. OK, so no more questions.

Miłosz [1:30:26]: Yeah, what should people do if they want to not miss the next episode?

Robert [1:30:31]: So we recommend to join our newsletter for one reason. So Deadly Algorithm may not send you a notification when a new episode is going out. So yeah, our newsletter is the best way to be notified about that. But we’re still under the rule of Deadly Algorithm, so it would be cool If you can leave thumbs up, leave a five-star review in podcast applications, leave some comments if you have any questions who are answering and reading all the comments. So if you have some extra questions, we’ll be more than happy to help you. What else?

Miłosz [1:31:14]: There’s one more comment. Let me read it. So is it too disrespectful to say that Go feels like Node.js?

Robert [1:31:24]: It is.

Miłosz [1:31:26]: When you start a project with them you start from scratch one index.js or main go and there is no structure from the beginning, yeah guess on how you approach starting out like there are some scripts in node i think that set up this scaffolding for you yeah that’s quite common but the approach yeah i think it’s pretty a sane approach. Start with one file and then see what happens. Start creating packages as the main guy becomes too big to handle.

Robert [1:32:04]: If you probably compare it to other languages, like, I don’t know, there are some languages that you need to use some kind of framework to build application and you have a lot of portal plates to start with. In Go, it’s like with You can start with something super simple and extend. So in this case, it’s totally true, but yeah, I think Go is totally different in terms of typing system. So it’s much more strict, let’s say. And while you are maybe building some smaller projects, it’s not a big deal, but when you are working on much more complex projects, when more people are working in, it’s much, much, much easier. Not mentioning about concurrency primitives that, in my opinion, in Node.js and JavaScript are awful. Sorry, in my opinion, promises and actually running everything on one thread is a deal breaker. In Go, it’s better. So you have an abstraction of GoRoutines that is very easy to understand and think about. And it’s also super fast because it can utilize multiple cores. Yeah, it has more threads. Hurray! And it has also nice primitives to talk between threads, let’s say. There are no threads because this is something special in GoRoutines, but communication between them is super nice, especially compared to Python.

Miłosz [1:33:33]: Easy to grasp, easy to read and understand.

Robert [1:33:37]: Yeah, I think it’s also interesting to actually compare to Python because, again, communication between threads, I have PTSD and I’ve asked about that, not mentioning global interpreter log. Come on.

Miłosz [1:33:50]: Yeah, so it’s hard to beat Go’s combination of easy of use and also how powerful it is.

Robert [1:33:58]: Yeah, and what’s also nice is how lightweight it is, because under the hood, one goroutine, so it’s not one thread. Without going to the details, it’s using one thread for multiple goroutines, so thanks to that, it’s super lightweight. So, for example, you can spawn like 1 million, 10 million goroutines, and it’s very lightweight, and you can run it easily on laptop.

Miłosz [1:34:22]: So there’s a follow-up. Yes, I meant the lack of structure. For example, in .NET Core, Spring Boot, you have an initial structure. So that’s about frameworks, I guess, more than languages. And we had an episode on frameworks as well, where we talked about this. I guess you can find some frameworks in Go too, where it will create some project structure for you. And maybe that’s why people are used to those structures, they start the project in Go, start with a single main Go and they are lost and they know what to do next.

Robert [1:35:00]: Okay, if you are looking for some inspiration, so on our blog we have articles about clean architecture, how to dig pragmatically, and there is also this wild workouts project that we created. We’ll add link to this project in the episode description and in the episode summary on our blog.

Miłosz [1:35:22]: All right, thank you for the questions and thank you for joining us.

Robert [1:35:27]: Thank you Miłosz.

Miłosz [1:35:27]: Leave a comment, let us know about your controversial opinions.

Robert [1:35:33]: In two weeks we’ll talk about Watermill as we promised a bit earlier. So Watermill, we mentioned this library already a couple times in this episode. So this is an event-driven, not only event-driven, but mostly event-driven library for building event-driven applications in Go. It seems that this is the most popular library for building this kind of architecture in Go, and I think it has now more than 8000 stars on GitHub, and I think it’s pretty popular and pretty successful and we’ll share some magic stories about watermills and why we believe that it’s successful.

Miłosz [1:36:14]: So if you plan to start an open source project, it might be interesting for you.

Robert [1:36:20]: And after that we’ll talk about sync versus async architecture. So connected to watermills.

Miłosz [1:36:30]: Where one even different architecture shines and when it’s better to keep to the standard synchronous operations.

Robert [1:36:38]: And it will be over-engineering. See you in two weeks, see you in four weeks. Thank you very much.

Miłosz [1:36:43]: See you then. Thanks, everyone. Bye-bye.

Robert [1:36:45]: Bye.

Learning Software Skills fast: what worked for us best in the last 15 years

Wed, 16 Apr 2025 16:00:00 +0000

Quick takeaways

Focus on applying what you learn - reading books or watching videos isn’t enough without practice
Build real projects that challenge you - trivial examples don’t expose you to the hidden complexities you’ll face in actual work
Expect and embrace frustration - feeling stuck often means you’re learning something valuable
Learn timeless concepts over framework-specific details - aim for universal software skills like modularization
Mix theory with practice in small chunks - read a bit and code a bit, rather than consuming large amounts of content at once

Introduction

In this episode, we discuss how to learn effectively as a software engineer. Why some people seem to learn faster than others? What are some practical ways to speed up your learning? Instead of promising magical shortcuts to becoming a principal engineer in months, we focus on a more balanced approach that helps you build skills by mixing theory with practice.

Notes

We mentioned our learning platform
Domain-Driven Design was referenced throughout - check “Implementing Domain-Driven Design” by Vaughn Vernon
The Pragmatic Programmer by Andrew Hunt and David Thomas is a great book for timeless software skills
Watermill - our open-source library mentioned as an example of a project that taught us while helping others: github.com/ThreeDotsLabs/watermill
Event-Driven Architecture traces back to 1950s, but was formalized about 20 years ago
The Repository pattern blog post

Quotes

Just going fast is a trap, because you can go fast in the wrong direction and spend years learning things that are useless in practice or maybe are useful in the short term but not helping you during your entire career.

Robert

Once you tackle building something more complex that you’ve never built before, or maybe never even seen before, that’s when things get interesting, and that’s where your skills are tested.

Miłosz

I learned the most with the hardest projects that I worked on. The projects that were easy, greenfield, and everything was known - okay, it was nice, fun, but I wouldn’t say that I learned that much at the end.

Robert

If you feel frustrated while learning, you are probably onto something. Because you are frustrated that you can’t make it work. And then it also motivates you to push through.

Miłosz

What helps to learn much faster is trying to focus on things that are timeless. You can get benefit from the compounding knowledge over time.

Robert

Until you start fighting the obstacles, struggling to make it work, you probably won’t learn much. The hard part is figuring out what to build that’s a bit more advanced and will test your skills. If you pick up an idea and think ‘I have no idea how to do it’ - that is probably a good direction.

Miłosz

What also helps is sometimes feeling as the dumbest person in the room. Work with people that are more experienced than you. And if you sometimes feel that you’re not the dumbest person in the room, maybe it’s time to try someplace where you will be.

Robert

If you work alone, you don’t need to care that much about code readability. If you can fit the entire context of what the application does in your memory, you can use a single file that’s 5000 lines long. But as you work with someone else, it starts to get more complicated.

Miłosz

Timestamps

Transcript

Robert [0:07]: Ever worked with someone who’s been coding for more than 10 years but still makes mistakes? Or have you seen a junior engineer write code that blows your mind? It’s not always about intelligence. The real secret sauce is how you learn. Learning the wrong things can feel productive, but it won’t get you far. In this episode, we’ll explore the art of fast, effective learning. I’m Robert.

Miłosz [0:27]: And I am Miłosz, and this is the No Silver Bullet live podcast, where we discuss mindful blackend engineering. We spent almost 20 years working together in different projects and teams. During that time, we learned that following advice like “always do X” or “never do Y” doesn’t work and can limit your growth. In this show, we share multiple perspectives that will help you make smart choices and grow.

Robert [0:55]: Yeah so if you have any questions please leave them on the chat we’ll try to address them on demand if not we’ll have Q&A session at the end of the live so we can answer other questions, anynthing to add?

Miłosz [1:11]: Good let’s go

Robert [1:12]: All right so I think it’s important to start with setting expectations because sometimes you may heard from many people that there are some magic way to learn faster you can be principal engineer in one month or maybe half year but well unfortunately it’s not that simple so everything what we’ll mention in this episode wouldn’t be advice like that it wouldn’t give you any magical power to be principal engineer in one year but definitely it can accelerate your car and it should definitely help you with not staying junior or mid engineer for four years because i think everybody probably met such person during working with software that as we mentioned at the beginning this was mid this mid engineer for forever yeah.

Miłosz [2:04]: And magic recipes usually don’t work, and often means someone wants to sell you something, so watch out for silver bullets.

Robert [2:16]: But still, sometimes there are people that learn really, really fast. And today we’ll share a couple hints that, from our observations, made them very effective learners. And again, it’s not about being smart, because from my experience, sometimes being smart can be even a trap, because you may try to be too smart and try to maybe dig too deep to some things, but it’s not always the best idea.

Miłosz [2:55]: Yeah, it depends on what you are smart about. But you can easily see this in different people you work with, especially in teams. Some people seem to learn much faster than others. And seems there must be some secret behind it or at least some ideas that can help you learn faster

Robert [3:22]: I’m not sure if you agree but from my experience it’s often not about what to do but actually the opposite so what you shouldn’t do to actually not be slower so I think we can probably compare it to going somewhere by the car so let’s imagine that you have Lamborghini and, slower car, but you don’t have a map when you’re going with Lamborghini. And in this case, you might get somewhere really fast, but not in the place that you need to go, actually.

Miłosz [3:52]: Right. So the destination matters, basically.

Robert [3:54]: Yes, yes. Because just going fast is a trap, because you can go fast to the wrong direction and spend years on learning things that are useless in practice or maybe are useful in short term. But they’re not helping you during the entire career because probably you work as an engineer for…

Miłosz [4:17]: Yeah, and it’s hard to differentiate because it’s easy to feel like you’re learning something. You can accommodate some trivia or ideas but you never put them in use. So I guess what we want to talk about today is learning skills.

Robert [4:36]: Something.

Miłosz [4:37]: You can put into practice you can use somehow in your work or just projects in general

Robert [4:45]: Yeah so probably it would make sense to start with some kind of maybe high level maybe not definition but idea what learning, means because you can do many activities that may sound like learning but it may not that’s useful for you to use in Rework.

Miłosz [5:13]: So maybe the fact that you can use a skill to build something in practice would be the primary one for me. So it’s not only about having the knowledge about some tool, library or programming language, but being able to use it in work to build something actually useful. And probably not a trivial to-do app or something most tutorials teach you. Right, so it’s quite easy to use any programming language or library to spin up a trivial web application, because there are tons of tutorials like this. And nowadays AI can generate this for you right away, because it could learn on so many materials on this. But once you tackle building something more complex that you’ve never built before, or maybe never even seen before, that’s when things get interesting, and that’s where your skills are tested.

Robert [6:25]: Yes, and I think maybe it may sound familiar for you, when, for example, you did the opposite. So, for example, you’ve been on a conference, you’ve seen some cool idea, or you read a book or article about this cool idea, and you’re trying to apply it later in work, and for some reason, it doesn’t work as it was promised in…

Miłosz [6:49]: Yeah, because you missed some hidden complexity or edge cases that were not mentioned there, or maybe the tutorial didn’t cover it, and now you’re stuck. That’s a pretty usual experience when starting to learn something new in software especially.

Robert [7:09]: I think we can probably compare it to playing an instrument or driving a car. Let’s imagine that you are trying to learn driving a car by just reading a book or watching some video tutorial on how it works. Okay, maybe it can give you some useful hints how to do that, but as long as you will not really do that and, put it into practice, you cannot really say that you can play an instrument or drive a car. Because in practice you discover a lot of hidden complexity that, okay, maybe on the movie it looks simple, but in practice there is much, much more and it’s harder than it looks.

Miłosz [7:54]: And there is some danger, because you can feel like you actually know how to drive or how to play a guitar, let’s say, after just reading about it. But then in practice you don’t have any experience, so you won’t be able to do it. And it’s quite similar to reading books maybe, on programming especially. And I think we read quite a lot, especially 15 years ago when there weren’t many online resources, videos and so on. The books were primary learning material for us. And sometimes the theory gives you nice ideas how to work with codes or mental models, but you have to actually start writing code to see if you learned anything.

Robert [8:48]: And it’s because you can read about many things, but often what’s described in the books it’s applied in some very specific context. And the project that you are working at this time may not really fit this, or it may fit, but it’s a bit different problem to solve. Because there are many similarities between multiple projects, but there are also some small things that are a bit different, but it can make problems much, much more different.

Miłosz [9:20]: And because books are focused on just one area, they can’t give you the entire context. And I think we felt this because we came up with many examples to teach people some concepts. And it’s very hard to come up with something that’s not this trivial to the application, that’s not complex at all, and come up with something that’s kind of like in real life, but also can be used to teach someone something. And same issue with books many books try to introduce this idea of a project or something like that But it’s quite hard to do it well. And to cover some real-life scenarios that someone can later put to use in their own projects.

Robert [10:09]: Yeah, and there is also one thing from our perspective of people that we’re creating online examples, articles, book. It’s also hard balance because, okay, in theory we can create an application that is as complex as a super complex application built by 100 people. But from the other side, it will be also a bit hard to work with that and learn from that. So it’s a hard balance that we are trying to achieve. The other thing that is also coming to my mind is that it’s connected to the thing that context in the book may be a bit different than yours. For example, when a couple years ago we were switching to Go programming language, we were working in some projects that required some more advanced tactics to write software like domain-driven design. The other side was that there were not all the materials. Actually, I would say that all the materials that were out there about domain-driven design was totally wrong.

Miłosz [11:10]: Yeah, it was a very long time ago. Yeah. It was very in the early stages back then.

Robert [11:17]: It still didn’t change that much now. There are some materials that are good, but still, most of them are still missing the point. But anyway, I think we spent probably half a year to figure it out, how to do it in a way that is compatible with Go programming language. And you can say that, OK, it’s just using it in a different programming language, but in practice, it requires time. So it’s not that easy.

Miłosz [11:42]: Yeah, especially if you do it on a live product. in a team where no one has experience like this. Thankfully, we could use some books, like implementing domain-driven design back then, which was helpful. But still, there was a lot of iteration, and I remember the first approaches were much different than what we ended up with. And probably not that great.

Robert [12:10]: Yes, probably we can summarize this first anti-pattern as this theory is not practice, and we’re taught to not be the person that is reading 50 books and think that, okay, I know how to solve many problems. Because just reading about that, it may be not enough. Because on paper, everything looks great, usually, but in practice, it’s totally different stories. And I know some people that were like that, that they were reading about many things, and when it comes to solving some problem, they were like, oh, let’s just do it like that. I wrote about it. It will be simple. But I know already that it’s a bit different in practice.

Miłosz [12:50]: Yeah, sometimes it’s fine to read just for fun if you are curious about something. But it doesn’t mean you have to use it in your day work. But it’s much different than learning skills, I would say.

Robert [13:05]: So it’s all about extremes. So don’t be a terrorist that is detached from reality. Don’t be like an Envoy Tower architect that knows everything best from theory but doesn’t touch this code and doesn’t need to write it.

Miłosz [13:18]: If reading doesn’t work, maybe video cursors. Do you like video lessons?

Robert [13:26]: No. No, because I think it has a pretty similar problem. So maybe it can give some more feeling like you’re looking like somebody is implementing something and it’s a bit closer maybe to feeling that you’re really implementing that. But it also has one risk that if you’re looking, if somebody more experienced is writing something, this person is probably skipping a lot of complexities that the person already know or just going straightly forward to the solution because, well, the person already know how to do that and doesn’t struggle about that.

Miłosz [14:04]: If they show you a big project on screen, you probably won’t be able to grasp the entire context just by watching the video. Although I think it’s a nice format if you want to give a quick overview of some code, or if you want someone to see how to do something exactly, like refactor something. What comes to my mind is live coding on presentations. That’s quite a nice format because it is like 5 or 10 minute video. Not video, it’s live on presentations of course but it’s a different format that can be useful but what I don’t like is a video course that I open and I see there’s 24 hours of video watching ahead of me and I am already bored I didn’t even start and I know there will be lots of filler content in there and yeah, that doesn’t make me want to learn.

Robert [15:14]: Yeah, so what are the solutions here?

Miłosz [15:18]: Probably build something.

Robert [15:20]: Oh no, I need to get my hands dirty. Touch the keyboard and implement stuff. That’s so bad.

Miłosz [15:29]: Yeah, it’s basically what we mentioned about this iteration while learning from a book. Until you start fighting the obstacles struggling to make it work you probably won’t learn much and probably the hard part here is figuring out what to build right because again you can pick up a tutorial and build some easy application some trivial application but it won’t teach you much so you have to pick up something that’s a bit more advanced and that probably will stress you a bit i mean test your skills so if you pick up an idea and you think like i have no idea how to do it that is probably a good direction i would say because you will need to figure it out yeah

Robert [16:27]: I think we can compare it to like playing instruments with planning to play a gig. It’s probably similar to how you should approach learning. For example, if you’d like to play a gig, you probably should start with something simple, play guitar yourself, later some rehearsals with your band, and later going to a gig and playing that. Obviously, it’s not easy, it requires a lot of time, but let’s imagine that instead you are watching a video on how to play guitar and you’re going strictly to directly to play a geek it will be probably not the best idea yeah and you’ll be super stressed and probably it will be a disaster but it’s similar thing it’s similar thing with learning programming skills so your geek is well the job that you’re working in and again you can try to learn something from just book from video and try to do your geek directly and okay, it will be a bit maybe faster, but from other side the result will be much worse.

Miłosz [17:35]: A parallel that comes to me is in software world it’s like your employer gives you a video course buys it for you you watch some 20 hour video and then you’re expected to apply it in the company’s project which you’re probably not ready to do it yet and you will experiment on a real

Miłosz [18:02]: product that’s used by real people. So there’s a risk that you will not apply it correctly and it won’t end up that well.

Robert [18:12]: And it’s another risk that I can see. So if you have some concept and the best place that you’ll see to apply that is the project that you’re working on your job, there’s a risk that you may end up with doing the CV-driven development. So basically, you are applying all patterns that you can see in your job project, and it will be probably a disaster for the problem.

Miłosz [18:39]: Of course, I applied the saga pattern in my previous year.

Robert [18:43]: Yeah, and probably somebody is applying it to the context that it’s not helping. So it’s making the project worse, and also you’re not relearning. Because if you are not applying those patterns for the right problems, you will also not learn how to do it properly, because you just apply it in the wrong context.

Miłosz [19:05]: So the ideal scenario, I think, is to have a pet project, but one that solves some real problem. Sometimes it’s hard to come up with something like this, but it doesn’t have to be super complex, but at least if it’s something that does something real, it gives you this kind of illusion of working on something more serious than a simple to-do app that’s free handlers and that’s it. It’s best if you can run in production and treat it as a production-grade deployment in general. So you have a domain and some servers or cloud or setup or whatever, set up some alerts, monitoring and so on. If you treat it like this, then you will start finding all the issues that can happen in a real-life project too. The hard part is finding this idea of what to build.

Robert [20:16]: But it’s probably one even harder problem, because you can test it on some pet projects, But I think it’s also essential to, at the end, apply it to some real project with some team. Because I think we’ve been trying multiple patterns in some smaller projects. But after we have used it with teams, it also changed how we’ve seen all that. Because if you’re working on a project alone, it’s a totally different dynamic, let’s say. Because it’s just much simpler. because if you overcomplicate something, you probably know how to go over it. But if you’re working with a team, and if you overcomplicate something, you’ll be not the only person that struggles. Your teammates will struggle much more, and they will also not be super happy about it.

Miłosz [21:10]: That’s a good point, since if you work alone, you don’t need to care that much about code readability, for example. If you can fit the entire context of what the application does in your memory you can use a single file that’s 5000 lines long and you don’t need many patterns there because you just know where everything is but as you work with someone else it starts to get more complicated it. So maybe that’s a good advice too, to seek someone else you can share a project with. I think that’s also how we started our more serious projects when we worked together.

Robert [22:03]: So there is a question on the chat that, like with playing guitar, so it’s good to have friends to play with, but what if you have no friend to play together or practice with? How you motivate yourself to learn guitar until you join a gig. So we’ll have some solution for that. So we and Milos are probably pretty lucky that we know each other and we’re building projects together for almost 20 years. But yeah, we know that it’s not the thing that everybody, not everybody has this luck, but we’ll have some hints that can help with that definitely.

Miłosz [22:41]: I guess the primary one would be you can find a community online and join someone with building a project but

Robert [22:51]: We’ll touch on that a bit later.

Miłosz [22:56]: Do you want to go over the two comments we have?

Robert [23:01]: Yes let me just check what from that we’ll cover later, I think we’ll cover so we can cover them a bit later when we’ll have something in those contexts so let’s wait for a while but yeah, we have in on our list.

Miłosz [23:27]: Martin mentions that he feels what I said about video courses and that they are like a torture yeah, I feel the same I think it’s interesting format for shorter things, showcases or screencasts, for example. But, yeah, like 20-hour video courses are…

Robert [23:51]: Or maybe for some, let’s say, more soft skill-related… Maybe not really soft skill, but not coding-related skills that are also important, but they’re not about writing the codes. And yeah going maybe into the problems that are there so finding the friends to go together or, maybe you don’t have a time because we know how we know how it’s changing with the time so when you are younger you can code over nights and over the weekends but if you’re getting a bit older you may have family, you can have multiple more duties. And it’s a bit harder because, there are multiple obstacles that you have. So the first one is that you need to find some project to work in. And we already mentioned that this is the hard part of it, because Because it’s sometimes hard to find a good problem to solve that is fitting the pattern that you’re trying to use.

Miłosz [25:03]: Not some artificial problem, but something that actually solves something.

Robert [25:08]: And it doesn’t oversimplify that. Because if you learn that, okay, get some simple problem and try to apply domain-driven design for that, and you will try to move it to real projects, it will just end up with some over-engineering stuff and it will be not the best. The other thing is probably also related to this torture with videos that you have super big chunk of knowledge to ingest, like 20 hours of video, you are watching that and later trying to apply that. And it’s super hard. And sometimes you maybe don’t have one or two hours to watch this video. It would be better to just have some smaller chunks to work in. And already I’ve seen on the chat that somebody mentioned that my main struggle when learning new skills is finding clear, consistent, practical and consistent material. I hate huge as boring text-only documentations and more visual and neat short tutorials. So this is basically it. It’s not easy to find this.

Miłosz [26:22]: Documentation especially. so it varies widely between projects. Some projects have really nice docs that you can get started easily, by the way. Give you some short examples and such. But some, yeah, not so great. You have to dig through the docs yourself and figure it out. So it’s definitely not easy.

Robert [26:47]: It’s also sometimes hard to find one place that is going from A to Z and explaining everything. And sometimes you need to go over 100 websites to find how to do that. Some alternative is Ask AI for that, but also you know that sometimes it can give you some hint, but it can lead you into the trouble.

Miłosz [27:09]: AI itself is a topic that suffers from this issue exactly. Because it evolves so fast, there’s probably no one place you can go and learn about how to use it. It just depends on which model you use and so on. So you have to go through many documentations and websites probably to learn it well.

Robert [27:35]: And there is actually a question. So what’s the AI impact for New West that skip the regular learning process? Do you recommend AI for using it? There are, starting learning.

Miłosz [27:48]: Software development?

Robert [27:49]: Yeah.

Miłosz [27:50]: So basically the question is how about using AI for learning?

Robert [27:56]: What do you think?

Miłosz [27:57]: I think there’s The same issue as with books on one hand, so it can teach you theory, and maybe even worse, sometimes it can give you ready answers, which you just copy paste or use directly in your IDE and you don’t touch the code, which can work, but you also probably won’t learn as much if you don’t struggle. And the other thing would be hallucinations, which is the primary reason I don’t use them much. For example, learning a library. I tried a few times to use AI to generate some use case for a new library I didn’t know, or an SDK. But very often it makes up function names, or just sort of invents something that doesn’t exist. And in the end, I just prefer to read the docs myself and do it from scratch, because it’s often confusing and I I don’t like the fact that I have to double-check everything it does for me.

Robert [29:21]: Yeah so i i think i have pretty similar point of view that i would say that it’s double-edged sword so it can help for many cases but you need to be really careful about what it’s saying and double check that okay because sometimes it’s helping to discover some thing that it may be harder to find somewhere in the internet or in the documentation and it’s helpful but it’s all about, verification that, okay, what it said, it makes sense. And I think it’s probably easier to use for more senior people that already know more context and more understand how, what’s the impact of the things that it’s generating. I can imagine that if I would be more junior developer, it would be the risk that, okay, I’m getting what it’s giving me and putting it to the project, And I’m spending ages to debug that and understand what’s happening. And if I would just follow the way that you mentioned that I would do it from scratch, it will be just much faster.

Miłosz [30:26]: If you want to learn the skill, I mean, you can argue that you won’t need the skill in the future because of AI. That’s a separate discussion. For now, no one knows for sure, so let’s leave it. But I also think there’s so many use cases for how to use AI for learning it’s very easy to give one answer because making it write the code for you is probably not the best idea if you want to learn some skills but I can imagine you could use it to give you some roadmap on what to learn you could prompt it with something like I know JavaScript and I want to learn Go recommend some concepts I should read about probably more helpful than just copy-pasting the code it gives you.

Robert [31:22]: Yeah, and I think there is one thing that with AI can be pretty helpful, and this is something that we’re actually working for last more or less one month. Because there’s also one problem when you’re learning something, you may be stuck on some stupid typo or some stupid mistakes, and sometimes AI I can help you with being unblocked. So actually, since one month, we are now working on implementing that kind of mentor functionality in our learning platform, because we noticed that some of you were stuck on some, again, some typos or some simple mistakes. And it can be actually pretty nice in finding those typos or mistakes and unblocking you. But from the other side, it can also say some stupid things. And I would say that this is the biggest challenge that we are fighting now, is ensuring that it would not give you some false advice and learn you something false, basically.

Miłosz [32:23]: And that’s something that’s very hard to understand if you are not an expert in the field. Because it will sound very confident. And if you have no experience with what it talks about, you want to know better. So I will be careful there.

Robert [32:41]: The good news is that we found some ideas what we can make to make it less wrong and pretty moderated on what it’s saying. But again, it will be never like 100% accurate. But let’s imagine that probably you remember some cases like you were building something for fun and you were stuck for 10 hours because of some stupid typo. So it will be just super useful to be unlocked because it’s not something productive let’s say to be stuck because of type of our 10 hours and it’s just frustrating.

Miłosz [33:13]: Yeah that’s a good point so also important to not be not go too far to the other extreme and say I won’t use AI ever because it’s not a good idea just use it as a tool as any other tool

Robert [33:32]: And I think it’s in general useful to making this tough process of learning by building stuff a bit nicer. Because we’ve been doing that and sometimes we’ve been really frustrated by the structure of building something. So basically we are trying to make this process more nice. So we are also, in our learning platform, giving some projects that you can work on. So you don’t need to find out the problems for the techniques that you are learning. And I think it’s also pretty cool because, again, in our life we built a lot of different projects and it took us some time to find some nice projects to learn stuff. But we know that it’s sometimes hard, especially when you don’t have that much free private time or sometimes you maybe would like to just have five minutes to work on something. And we’re trying to cut this learning to some small steps to give you ability to just spend five minutes a day and progress a bit on this small project that we’re giving.

Miłosz [34:48]: I like the fact that we recreated the way we used to work in the beginning. So you read some theory and then you solve some exercise, you write some code and you see it applied live.

Miłosz [35:07]: And still to this day, I think, is the best way I know of learning. You need some theory and then some practice. So this is what the other platform gives you as well.

Robert [35:19]: I think it’s also nice that it has some path with some proven techniques and you are just following those techniques. And you don’t need to look on hundreds of websites to find how to, for example, do event-driven architecture or something like that. But again, it’s not for everybody. So when we’ve been younger, we have just time to figure it out, go over internet and try to find all these techniques, try multiple different techniques, try to find a project to implement. But yeah, it’s also fun. Yeah, yeah, definitely.

Miłosz [35:53]: We can spend the time.

Robert [35:55]: It’s not for everybody, but again, if you are looking for some nice learning resources, go to our websites and it’s tab, trainings and courses. Again, it’s not for everybody. We’ve been not using such a platform because it didn’t exist, but again, it’s also not only way to learn, so you can easily replicate it yourself. Okay so the next thing about learning that i think i heard pretty often is that learning needs to be fun and interesting yeah and i think it’s also a bit risky because, it’s easy to start to optimize for fun in learning and in my opinion it’s not the best thing so if you compare it to school so well school is not about making so primary goal of school is not making to make learning fun it’s about learning if you would like if you are optimizing for fun not learning it’s kinder regarded, what do you think about yeah.

Miłosz [37:05]: And what comes to my mind is this gamification of you know different learning products so you get some prompts or badges and so on.

Robert [37:17]: Like Duolingo, for example. I’m not sure if you’ve been playing with Duolingo.

Miłosz [37:21]: A few years ago.

Robert [37:23]: I remember that it was actually quite cool that you have badges, you can have some frozen…

Miłosz [37:29]: You can invite friends and do it together. It seems like fun.

Robert [37:34]: And at some point you’re actually realizing that you’re not learning that much. I mean, you’re clicking some funny stuff every day and you have to take 100, days, it’s great, but it’s also like, okay, it’s very repetitive, and maybe it’s fun, but okay, I don’t feel that I went very far away.

Miłosz [37:55]: So I actually have a recent realization I had. So we were working on this AI stuff for our platform, integrating LLMs. And it’s a new technology, you know, there’s many things that have no standards yet, you have to discover many things and so on. And initially, I was very frustrated while working with it, because it’s a different mental model too. For example, you mentioned about hallucinations. We are used to programs being very strict and predictable. But with LLMs it’s just the other way. It’s a bit random at times. And it was initially frustrating. And then we figured out ways to make it work, basically. And it hit me that this frustrating part is usually the sign that you are learning something, you know.

Robert [39:07]: But I think it’s interesting because it’s often frustrating from one side, but it’s also a bit exciting. I’m not sure how to describe that, but…

Miłosz [39:16]: It is exciting after you struggle.

Robert [39:19]: Yeah, yeah, maybe, maybe. And at the end, you don’t remember that much this frustrating part.

Miłosz [39:24]: Yeah, yeah. So, I think that’s actually what you should be striving for, maybe? I’m not sure if that’s a good word to put it, but basically if you feel frustrated while learning, you are probably onto something. Because you are frustrated that you can’t make it work. That’s how I feel about it. And then it also motivates you maybe to push through, or maybe demotivates you to give up. But I think it’s a better sign than having fun. You can be having fun vibe coding. It’s a fun thing to do.

Robert [40:08]: Or making games. So for example, something that you recommend.

Miłosz [40:12]: Yeah, making games is fun too. But also can be frustrating if you don’t know how to do something. It’s a similar thing.

Robert [40:19]: And I think it’s similar when you’re working in some projects. So probably we worked sometimes in the same projects. And I’m not sure if you agree, but from my perspective, I learned the most with the hardest project that I worked. So the projects that were easy, greenfield, and everything was known. Okay, it was nice, fun, but I wouldn’t say that I learned at the end that much. I think I learned the most with the projects that, you know, it was sweat, blood, and at the end we succeeded to kind of overcome that. But there were times of this project that it was really hard to just go over it. And well it was tough it was frustrating but at the end we’ve been able to overcome that and i think it only works probably when to also be clear here so it’s not only about projects that it’s hard tough and it’s you’re not doing about anything about that because this is a death march it’s not learning but if you are in the hard project and you have some plan how to improve that and you are implementing this plan, you are learning a lot. For example, you have some super legacy application used by many people with super complex logic, and it will require a lot of skills on multiple levels to just overcome it.

Miłosz [41:41]: And you just have to figure it out. That’s the hard part and also the best learning moment. Like when you lose sleep because you fight with some issue you can’t figure out the solution for and then you the next day you wake up or you take a shower and you somehow grasp it unconsciously but that comes after this struggling part it’s not, you know, you are not just happy about it and having fun the entire day you think hard about how to solve this.

Robert [42:23]: And I think the opposite, I can sometimes see the opposite in people that think that are learning from social media, let’s say. So you on X are observing multiple programmers that are doing some cool stuff and you read that and you feel that, okay, I know how to overcome some problems that people shared. But I think it’s also dangerous because it’s worth mentioning and reminding that social media is full of shows, let’s say, making grass greener than it is, and a lot of populism.

Miłosz [43:06]: Yeah, so basically extremes.

Robert [43:09]: Yes, so what works on social media is writing about something that is controversial. Many people will disagree, comment on it, but unfortunately this is what works. And you may think that, okay, I’m reading those comments, ideas, and I’m learning a lot from that. But I think it’s not like this in practice. And if you have more experience, you see that a lot of that is just making grass greener and hiding the bad parts of some approaches.

Miłosz [43:44]: Yeah, it’s not really learning according to what we talked about initially. Because you don’t use the skills and you don’t learn any skills actually. Just learn some trivia maybe or some strong techs of some tech influencer.

Robert [44:03]: Especially those a bit less experienced that they don’t know that they doesn’t know and they think that, okay, you are overcomplicating stuff. You don’t need to use X. It’s not that, just go simple.

Miłosz [44:17]: Yeah, it may also seem like you contact only the extreme views on some topic. Because that’s what attracts most comments and trolls and discussion in general online. You are very extreme on one end of the spectrum about something. In reality, there are probably not so many easy answers, and it depends on the context. Our last episode about clean architecture is probably a good example of this, where you can find as many people who love it as those who don’t want to work with it ever again.

Robert [45:10]: But again, it’s a risk, because it’s easier to just scroll through social media and have something of learning, but again, it’s not really learning. It’s probably also similar to watching some fun programming streams and people doing some fun on coding. So it’s probably easier to ingest and it may be fun. But again, I would separate the fun from learning.

Miłosz [45:42]: It’s entertainment.

Robert [45:44]: Exactly.

Miłosz [45:45]: For example, my YouTube feed is now full of drummers doing some white tricks. But I’m not even playing drums. It’s something that’s cool to watch, but it’s not learning.

Robert [46:03]: So again, what’s our tactic here? I would say that entertainment is also important, but it’s also good to separate some time for some deep dive content that is not making grass greener and making ideal world in places where it’s not. It’s better to sometimes find some deep deep dive videos or articles, basically something that we believe that is teaching you a lot. And this is actually something that we’re doing on our blog. So for example, we have multiple blog posts that are super long and they’re probably not as easy to read as some five minutes articles. But from the other side, the alternative is reading some simplified articles and later struggling to implement it on production, because it’s usually how it’s ending up.

Miłosz [46:59]: And even if you read this deep dive article, you still need to apply it somewhere to truly grasp it. Right so it gives you a more more more tools more context but still it’s best if you just use it yourself

Robert [47:16]: Yeah and don’t take us wrong so it all doesn’t mean that learning doesn’t need to be fun it’s just about putting the priorities so the first is learning because if you spend time on learning wrong stuff you just lose a lot of time and for example if you’re prior to prioritize, fun but i would maybe say that you know you should not you should not optimize for fun but for excitement rather so find some exciting project that you can work on that will excite you to go forward and even if you’ll be frustrated this excitement will try to you.

Miłosz [47:58]: Want to see it live

Robert [47:59]: Yeah that’s the motivation.

Miłosz [48:01]: Here that’s very nice

Robert [48:03]: And i think that it’s for example what which helps me often to overcome this hard part.

Miłosz [48:12]: When the frustration hits, you need to use this excitement to look forward and push through.

Robert [48:19]: And I think it also helps at the end when you spend a lot of time on this frustrating project, but at the end somebody’s using that and you see that it’s useful for people and it’s like, oh, okay, it was better. And I think, for example, I have sometimes like this with Watermule, with the library that we created, that sometimes working on fixing some flaky tests or merging some PRs or spending time on doing some reviews, it’s a bit sometimes boring. Maybe it’s a bit sometimes tough to do. But at the end, when we see that we’re making a new release and people are happy about it and people are using this, it’s giving some kind of fuel to progress with it.

Miłosz [49:02]: And it also started as a small library that solved our specific use case maybe also a good example of how we can start working on something that is useful for someone else and can help you learn something because we also learned a lot about maintaining an open source project over the years thanks to it definitely Definitely.

Robert [49:30]: So, okay. Now we, I think, covered pretty deeply what is the good direction, what you should basically learn. So, let’s imagine that you have your map now, but let’s now discuss for a while how you can get faster to this place that map is showing you to go.

Robert [50:01]: So, the first thing that I have in mind in this is, again, what you should learn. Or, okay, actually what you should learn. So, I think what helps a lot is focusing on things that won’t get out of day pretty quickly. Because to take some example, let’s imagine that you are a front-end developer and you are some framework specialist. And the risk is that your knowledge may become obsolete much faster if we compare to learning it some universal skills. So in other words, I think what helps to learn much faster is trying to focus on things that are timeless. And basically, you can get benefit from the compounding knowledge over the time.

Miłosz [51:10]: Maybe also not stick too long to one technology, if you see something else that you could use. Which on the other extreme is also not that nice if you just keep jumping from one programming language to another. Even take frameworks, it may be useful to know more than one, even if you primarily use one, even to just learn some different concepts. Maybe that’s actually a good one about programming languages, but you can learn some just for for some different point of view even if you don’t do it on production But it can also teach you those timeless concepts you mentioned.

Robert [52:03]: That’s true. That’s true. But I think one help that can help you evaluate… Because you are totally right. So it’s important to not go into extremes like, okay, I’m not learning any framework, I’m just learning the things that are timeless. But it’s more about not going in an extreme and trying to mix it and not being a framework specialist. I think we already mentioned it in previous episodes multiple times. And in terms of AI, you will be never better than AI in learning some framework quirks. Because it’s just a limitation of your memory and it will be more updates than you can ingest. and you will be out of date sooner. But there are things that, well, AI will be never as good as you maybe. And like modularization or splitting bigger problems into smaller things like that. And it doesn’t matter actually what programming language or what stack you are using. For example, if you learn how to modularize things or split some problems into smaller. It will always help you, independently what technology you will use.

Miłosz [53:23]: It’s still relevant sometimes after decades since the invention.

Robert [53:29]: For example, I tried to check event-driven architecture that we are maybe not advocates, but we see many use cases where it’s useful. So from what I found, the first origins was in 1915, so pretty long time ago. Current form, it was formalized around 20 years ago, so a pretty long time ago, but again, the idea is pretty old. And the cool thing is that probably when you learned in this eight years ago something about it, it will be highly relevant still. And if you learn that many of the things that are universal and timeless, it will just help you over the entire career.

Miłosz [54:22]: Yeah, you can apply it to other languages or frameworks or whatever. Sometimes it’s fine to see how the solutions are reinvented in a different way. And this is like something fresh.

Robert [54:38]: Yeah, we’ve been touching this in an episode with clean architecture, I think, at times.

Miłosz [54:45]: Or, let’s say, like, Go channels, which seem quite fresh when coming from some language that doesn’t have these concurrency primitives built-in, but they are, too, not some new concepts, right? It’s just a concurrency model applied to a new programming language.

Robert [55:12]: I think it’s also pretty interesting that Vibe coding is a pretty hot topic, let’s say, now. And let’s even imagine, so I think it will not happen soon or ever, but let’s imagine a scenario when LLMs and Vibe coding totally replace programming. And still, in this world, skills like modularization or splitting problems into smaller, it will be still pretty useful. So you can try to hint this LLM a bit to how to split this problem into smaller, just to not have 5,000 lines of code, file, with everything. Because it will just struggle to do that. And it’s visible now, when you’re vibe-coding something and you’re trying to put everything into one file, it’s just a disaster. And it’s not that obvious how to split it into smaller chunks, let’s say. But if you have already this skill, it’s much easier because you can hint this model how to split that and its performance will be just much better.

Miłosz [56:17]: And modularization is probably a good example of this topic you can’t really learn that fast. And there are many extreme takes on it, like use microservices. And once you work with software for a while you know it’s not that simple and it’s also not something you can learn from a Twitter thread you learn in 30 seconds where someone says use this directory or whatever you just have to work through many different projects and learn some patterns learn some mental model that will help you later apply this approach.

Robert [57:02]: And I think it’s a similar case with testing. So, again, let’s imagine this work when we are no longer writing code, we are vibe-coding everything, fine. But it’s still useful to know what kind of tests you would like to have, and, how they are made. Because even if your model is great, it’s good to ensure that the next prompt will not do some regression in the logic that you had. And I would say that if this word will come, a bit won’t, but never say never. I think it will be critical to have proper tests for that, to ensure that does other prompt will not break things that some previous prompt did?

Miłosz [57:48]: Yeah, we can see it with the work we did recently. But similar testing principles still apply. Exactly. Even if you use new tools.

Robert [58:00]: Don’t be afraid. If AI will take your job, learn testing and modularization and techniques that you’ll feel that will be timeless.

Miłosz [58:13]: Maybe it’s a more general suggestion to have a diverse skill set. Probably everyone specializes in one area and it’s fine, but it’s useful to know a bit more about some fields similar to what you do. Like if you specialize in backend programming, you can still grasp some front-line ideas. So it makes it easier to communicate with teammates and solve more high-level problems than just looking at code.

Robert [58:50]: Yeah, and I think it’s something especially good for some more senior people, let’s say. So if you’re a junior, okay, let’s maybe try, you probably should try to master one area, let’s say. What more senior you are becoming, Link. More you should learn some other areas because i think what’s also useful is the fact that you can borrow from different areas let’s say so if you are even doing back-end you can borrow some ideas from front-end maybe from data science maybe from i don’t know ux or something like that so there are many nice ideas in different areas that can help you solve problems, or maybe even some areas that are totally not related to programming, like accounting, for example. You worked with some accounting domains. Actually, I think accountants have some cool ideas that you can apply to different domains, and it can help you to solve those problems nicely.

Miłosz [59:56]: Yeah, especially since the more senior you get, the more you work with people outside engineering. And you solve more high-level problems than just fixing bugs in the code. You have to figure out the entire process of a feature, let’s say. So the more you know about some other fields, the better you can do it.

Robert [1:00:21]: And what I think helps here is, again, doing some pet projects that… It’s probably our example. It’s not that pet project because now we’re doing it full-time, but before doing it full-time, we’ve been creating those pet projects and we needed to learn how to do front-end, how to do some basic data science, how to do some basic UX, and I think it helped us over the years when we were working on just back-end engineering, mostly in a daily job. And also there’s one thing that it’s sometimes good to, learn is about some management basics. So if you’re a software engineer, it’s sometimes nice to work for a while as a manager, because you can also understand the wider picture and maybe understand how the component dynamics are working. Because often we can go into some extremes like, why those stupid people want something from us? And if you work as a manager for a while, you know that there is sometimes some reason for that and it actually makes sense.

Miłosz [1:01:38]: The dangerous extreme here is thinking of yourself as, I’m just a developer. Tell me what to do and Don’t make me figure out your issues. So, yeah, that’s probably what you don’t want to do, because then you become, well, easy to replace, basically, because you are just typing code.

Robert [1:02:13]: All right so we’ve already said that uh from where you can to recap maybe from where you can learn so books so that this is the typical place from where you can uh learn from and it’s good direction but remember to put the things that you learned in practice because you will just forget you learn pretty quickly and it will not give you that much uh that you read this book basically.

Miłosz [1:02:44]: What I like about books here is if they stood the time of test of time and became classics like the Pragmatic Programmer for example it’s an easy way to figure out that there’s something valuable in them but you have to be careful because the quality varies there are some books that probably could be could have been blog posts but there are also some timeless classics and it can also teach you some theory that’s very nice to know.

Robert [1:03:17]: Yeah, so videos, as I said, we are not big fans of videos. There are some for programming knowledge. Maybe it’s nice for some soft skills, but as somebody mentioned, it’s sometimes a torture. And I think it’s also a hard process of learning like that, because you cannot spend five keynote on watching that, switching to code and later watching, I think it’s not working best, basically. Or for example, watching something for one hour and going to code.

Miłosz [1:03:50]: It’s easy to get distracted or bored.

Robert [1:03:54]: Yeah, so I think we more recommend the way that, for example, we are promoting in our learning platform that you are learning some small part of knowledge, applying that learning piece of knowledge and applying that. But you can also do often similar things with the book, basically. What’s also helped to remember stuff, at least for me, but I think probably for most people, is making the notes of the book. I think it may sound obvious, but sometimes you may not have enough time to do that. But I highly recommend that because without those notes, the chance that you forget what you just read is pretty big. What also works for me pretty nice is doing mind map for some more complex content for more complex or more complex concept content yes, because it’s also helping to navigate through them easier and basically.

Miłosz [1:04:56]: Writing down in your own words often helps you think differently about the concept and also teaching others is very useful. Because then you try to figure out how to describe this in a simple way. It’s often when I write about some topics on our blog that I start to get a deeper understanding of it. Because you have to construct this series of thoughts that someone can read and understand. So having your own blog or even writing in private is super useful for having a more deeper understanding of those ideas you learn. And you can go back to it later in the review. Even if you don’t, it’s worth the effort, I think, because it just helps you think better.

Robert [1:06:00]: And again, writing notes with your own words, I think it’s very important because it’s forcing you to do some thinking process and just remember it better.

Miłosz [1:06:14]: And something I really like to do is mixing theory with practice. We mentioned it before as well.

Miłosz [1:06:25]: So, you know, it’s easy to go to the extremes here. So on one hand, you can be the person who just reads the entire documentation first and then starts applying it, which can take a long time and probably you will forget some parts of it. Or you can just try to blindly write it yourself and never consult the docs. It’s also probably not a good idea because you can miss some concepts. So usually what I like to do is, let’s say I use some new technology or library or something, to get the example from the documentation and have it running on my computer. That’s the first contact you have with it. You can see it compiles, you see it running, you can play with it. That’s how you learn how it works. And then you will probably run into some edge cases you don’t understand. This is where the frustration begins. And this is when it’s time to read the manual. Just go back to the docs, find maybe some theory of how it works and you just go back and forth between practice and reading. I think that’s what works best.

Robert [1:07:52]: One thing that sometimes people are mentioning that it’s great to learn from mistakes and That’s true, but it’s also not the most optimal way. So often you can lose a lot of time going… Around some simple solution that you don’t know that exists and in perfect word it will be cool if you can have some mentor so sometimes you can have some mentor in the work that you’re working on if not find some communities so for example like our discord that we have or there are some communities around our trainings maybe reddit and x may be tricky part because a lot of online communities are extreme and if you are less experienced, it may be tough. Some asking AI, it can be also sometimes.

Miłosz [1:08:47]: Yeah, it’s a bit like this. If you don’t know what you don’t know yet, describing this to someone, maybe AI, maybe not, can be helpful. I remember how I was a junior programmer, it was super useful to have some seniors around that I could nag with questions during the first few months of work. I think it’s a super fast way to learn. If you have some problems at hand, you need to solve, but you don’t know how, so you ask someone who is an expert in this area, and they can explain why and how it works.

Robert [1:09:36]: Yeah, and I think from my experience, probably sometimes we missed this mentor, this person that could say us that just do it in this way, because we spend sometimes a lot of time to figure out some stuff, just to learn later that… This is the typical way of doing this, and if just somebody said to us, it would save a lot of time. On the other side, it’s also a nice process sometimes to find a way to solve some problem and, it was also useful.

Miłosz [1:10:09]: But i

Robert [1:10:10]: Think it’s important to have some balance basically here.

Miłosz [1:10:16]: Yeah but yeah maybe i i use for this can be a good idea sometimes because you not everyone can afford to have a more senior person sitting next to them and teaching them about some concepts so if you can have the the model explain some things or explain why something is done this way or another. Coming back to the previous discussion we had, if AI is good for learning, that might be one good use case.

Robert [1:10:47]: Watch out for that, because it can sometimes say some stupid stuff. What also helps? I think it’s also good to sometimes feel as the dumbest person in the So basically, work with people that are more experienced with you. And if you sometimes feel that you’re not the dumbest person in the room, and maybe it’s time to try someplace where you will be the best person,

Robert [1:11:22]: and you have some people to learn from. Because this can really help you to accelerate your learning.

Miłosz [1:11:35]: So maybe online communities are a good example of that. If you don’t get caught in a flame war.

Robert [1:11:44]: Yep. So watch out for that. All right. So something to add, Miłosz, or we can go to Q&A session.

Miłosz [1:11:51]: Yeah, I think we can wrap up.

Robert [1:11:54]: So yeah, just a reminder. So if you have any questions now, please just leave on the chat, and we can answer that. Okay. Okay, so there is a question about our event-driven training. So it’s one typical question that we have. So yeah, the plan about event-driven training is that, as we mentioned a bit earlier, so now we are working on our AI mentor that is integrated with our training platform. After that, we are planning to do big update to event-driven training before releasing that, and after that we will release that. So this is our plan. If somebody who is listening to that, if you already have event-driven if you already own go event-driven training, no worries you also have access to this because, as we promised you have lifetime access and lifetime updates of it basically this is the plan, we don’t have exact dates because what is the reality of estimation so.

Miłosz [1:13:00]: We received many suggestions and feedback from trainees on the training.

Robert [1:13:09]: We also had a lot of feedback like this is the best training that I ever had but we all are aiming even higher.

Miłosz [1:13:16]: We also received some issues that we could solve and we wanted to build another training first but we probably will first update, go event driven before the next sale. So you will hopefully be able to learn even faster and better with it.

Robert [1:13:39]: If you are not in our newsletter, the best way to know about this is obviously joining that, so you’ll be the first person to know. Again, we don’t want to commit on any date because we don’t know. When it’s done, it’s done, but, trust us it will be really cool update together with this uh ai mentor that we’re working now so we are at the last mile let’s say and it’s really really helpful with many cases when you are basically stuck or you need some hint what’s about what this exercise is it’s it’s cool okay the.

Miłosz [1:14:19]: Next one from ryan hi self-taught two years in career changer i know i need to build and learn as i go i’ve been studying tdd bdd any tips to navigate the design of my data modeling user stories i’m so noob don’t worry and so this is about the modeling of Yeah, modeling of the design of the application, I guess. It’s a bit like DDD, but maybe more general.

Robert [1:14:54]: Yeah, so I think we already have some blog posts about domain-driven design. They’re not that big about modeling, I think, on our blog. So you can have some hints, but this is definitely the topic that we’d like to explore deeper because it’s useful for more complex domains. And we had some experience with that, so we’ll be happy to share that. So we have somewhere the plans to write some new blog posts about that and have also a training that is around it but well as we said for the previous question so we have pretty long backlog before before that but.

Miłosz [1:15:31]: Maybe the follow-up is relevant because then royans says learning to deal with malleability of software and wanting to design things as from the beginning as possible is crippling me. So I think that’s the core issue. You need to let it go. This is something we talked about in the episode about architecture. And maybe in some others as well, that you can’t figure out the perfect design from the ground up. It never happens, because the software will change with time. So instead you need to adapt, try to start with something simple enough that works well, and see what the issues are, and then iterate.

Robert [1:16:18]: Yeah, and I think what’s helpful is the approach that model will be never perfect. It needs to be useful. That’s it. It’s a bit like a map. So when you have map, we’re talking about map multiple times during this episode, but when you have map, it’s not a picture from satellite or it’s not very, it’s just detailed enough to know how to go from point A to point B. But it doesn’t have all details, but again, it’s useful. So I think thinking about models in a similar way is useful. that you can try to make it perfect but it will be never perfect as you said you change over time or won’t.

Miłosz [1:16:58]: Be complete probably

Robert [1:17:00]: Yeah yeah but it also doesn’t mean that you should just do YOLO and you’ll improve it later because it’s also not the best approach because if you just, take the old approach it will probably hurt yeah.

Miłosz [1:17:15]: This is the the hard balance between making software that’s hard to extend and going to the other extreme where you try to foresee all possible future use cases and try to adapt your software to be universal so it’s it’s quite hard but it’s like i think it’s a good philosophy to start with that you know you want something good enough for now and something that is easy to change in the future and

Robert [1:17:44]: Make it as simple as possible but not simpler.

Miłosz [1:17:48]: Yeah that’s that’s hard to do i mean we touched on on this in the first episode about writing low quality code it is a bit similar topic um where to you know strive for this super high quality where everything is perfect but the reality is you probably won’t get it and I think we’ve seen it best in one system where we tried to model this accounting domain. And I remember, I anticipated this moment when we will finally cover all the edge cases and it will be finished. But then it turned out, even our stakeholders who were accountants didn’t know how to do some things. And we just kept iterating and struggling for months. Turns out that’s the reality of them.

Robert [1:18:46]: And don’t be afraid that it will be not perfect because it will be not to set expectations. Also, when we are building some models that are far from being perfect, they are sometimes wrong. But again, as long as they are useful, it’s good. And practice helps with time. So you’ll have more idea how to get this pass. But, well, practice, practice, practice.

Miłosz [1:19:15]: The next comment is about this, exactly. I switched six years ago, completely self-thought. Plus about two years of theory courses before the actual switch. The only way for me is to get as much real-life experience as possible.

Robert [1:19:31]: Yeah that’s it i see that also martin mentioned so yeah without.

Miłosz [1:19:37]: Practice no theory will stick

Robert [1:19:40]: Oh yeah i think i can confirm that it so you have the same feeling after 20 years thank.

Miłosz [1:19:46]: You for summarizing our episodes in one sentence that’s exactly

Robert [1:19:51]: This.

Miłosz [1:19:54]: And as soon as you feel comfortable at something, it’s time to learn something new. Good point. I’ve switched to Go plus DDD a few months ago, and it’s still frustrating to learn new things. I’ll buy it a bit faster. Yeah, that’s a good point as well. I shared before that I felt a bit like this, working with LLMs recently. Because for a long time we didn’t touch much new technology i think we just know because we we’ve seen that for many product issues very often technology is not the solution you can you can stick to very boring and stable technology and basically achieve what you need but yeah learning about something new is exactly this is exciting at the same time but also can be frustrating if you don’t know how to move forward. The good part is, once you do satisfaction and you actually know you learned something.

Robert [1:21:04]: I think it’s also really that something really new appears that can solve some new problems. I think LLMs are pretty different because it has some use cases that it wasn’t possible to cover earlier, but from another side also a lot of companies are trying to find a problem for the solution. And it’s also pretty problematic because it should be the opposite. So you have the problem and you’re finding a solution for that.

Miłosz [1:21:35]: Okay and ryan wrap up props up um i also go back and review material frequently when i can remember something without looking it up um yeah

Robert [1:21:48]: I remember that when we worked in one company we have the book about ddd on our desks all the time and we were all the time opening the book and looking for stuff and it was like ah yeah it was something like that. But I think we learned during the time a lot. So it was just, okay, we had something in mind that it was solving similar problem. We were looking to this block. Unfortunately, the example was in Java. So it was like, okay, this is how it’s done in Java. Let’s try to figure out how to do it in Go.

Miłosz [1:22:23]: Yeah, but it sure helps to have this material Or documentation. Another way to use AI maybe is to have it summarize some parts of it for you.

Robert [1:22:34]: Or maybe put everything to RAG and query.

Miłosz [1:22:39]: That’s another good use case probably. Because you won’t host it as much if you provide the full context to it.

Robert [1:22:47]: It’s a very fancy search, but it can do more things than search. Because it’s not text search, it’s more like meaning search.

Miłosz [1:22:55]: But when I think about it now, you know, going back to this project where we started using DDD, imagine we had an AI fed with this entire book and we could just ask questions about something specific or make it change the examples to go. That could be quite cool. But also it probably wouldn’t be exactly what we arrived at. So I’m not sure if that’s a good or bad.

Robert [1:23:23]: Yeah, yeah. But it was definitely a value in figuring it out yourself. And, well, I think it may be hard to be junior now, basically, because, as Samadhi mentioned, it’s probably trying to avoid LMs in some cases, but it’s not always possible, let’s say. Okay and the last question on the chat so well why how did you decide to teach it’s more rewarding for you than just programming i want to teach and do presentations but i feel like i have no enough experience how would i start uh so yeah um so how we decided to do that i don’t know how I.

Miłosz [1:24:20]: Remember that there was some New Year’s Eve party, and I told you I thought about writing a blog, and you said, yeah, me too. Okay, let’s do this. Sorry, created a blog.

Robert [1:24:36]: I don’t have that good memory, but it sounds like a good beginning story.

Miłosz [1:24:42]: There was no grand plan, I think. For me, writing about stuff is, on one hand, satisfying and on the other, it also helps me think better about the subject. So that’s what we mentioned about teaching others. So I think if you feel like it, definitely go do it.

Robert [1:25:04]: And I think you don’t need to be super experienced to do that. So i think earlier you begin then better and don’t be that much afraid that okay i will write something stupid or something like that it’s not that big deal to be honest i mean it’s hard to overcome that but at some point you will forget about that so if you start but so i think the good thing about is that it’s as you said it’s helping you to structurize the knowledge that you have and also i believe that skill of writing clearly it’s useful in multiple cases so as we mentioned a bit earlier so it’s helping you to convince for example your teammates to go in some direction write some design documents for some problems that you are doing and you can benefit in many fields from this skill, basically.

Miłosz [1:26:05]: And especially presentations are also not exactly about what you invented or discovered. There’s also how you present, how entertaining it is, and so on. So you don’t need to invent something great to be a good presenter or to be a good writer and yeah there’s probably still like space for for new articles good articles because especially now I think we are you know scattered with this AI-generated trivial posts on topics and spend time to write something great.

Robert [1:26:57]: And yeah, I think it’s again no silly bullet but what we recommend is creating the deep dive articles that are five times longer than average article in the topic and I think that this is some niche. And obviously, it requires more time to write that. Less people will read it. But I think in long term, it will just work nicely. And I think it’s also good to write about timeless things. So we mentioned those timeless things multiple times. But to give you a good example, so the blog post that we wrote about repository pattern six years ago, I think, probably last month we have like 50,000 visits because it was just reposted on Reddit and it went over the entire internet. And again, it’s probably since the start of the blog, I don’t know how many people read that, probably, I don’t know, 200,000 people or something like that. But it wasn’t in this one year. It was over six years and it’s still pretty valid and it will be valid probably in 10 years. So, but again, it took us some time to write that, but from other side, probably there are no other articles on repository pattern in Go that are that deep. So it’s some niche, obviously. And you also asked if it’s more rewarding than just programming.

Robert [1:28:22]: Oh yeah, definitely. And I think it’s also worth mentioning that probably you are still programming more than teaching. Well, we love to write code and I think it’s allowing us to have more ideas what we can teach, but it’s about some balance. And I think it’s just making it all more fun.

Miłosz [1:28:46]: Maybe we still do program what we teach through the platform. And maybe just one part of the story here is that we wrote a blog post, we had some example projects, and we wanted to recreate this idea. We talked about in this episode about being able to read some theory, write some code, and do it back and forth. To help people learn. That’s how it happened.

Robert [1:29:24]: So TLDR, I recommend it.

Miłosz [1:29:28]: And share with us once you do.

Robert [1:29:30]: Yeah, definitely. If you would like to also have some feedback, we can also help a bit. Right, so it seems it was the last question that we had. So last minute to… To ask us questions.

Robert [1:29:57]: Alright, and so while we are waiting for the last questions, just in case if they will arrive, so just a reminder. So if you are not part of our newsletter, I recommend to join that, because we recommend to subscribe us on all platforms where you are watching us, listening to us but newsletter is the best way because we cannot control magic algorithm of other platforms but newsletter you will be sure that will notify you about new episodes, so remember to do that uh if you have any comments questions leave us comment on youtube we’ll uh we are answering all the comments and we can also improve the and or touch it in the future episodes please don’t forget to leave us thumbs up five star review because again deadly argo algorithms are are ruling the world and without that more people will not hear about about our live podcast so yeah that would be it for today i think so thank you miłosz for today’s episode and see you in two weeks where we’ll talk about the popular Go opinions we have.

Miłosz [1:31:16]: Exactly. And why are we controversial?

Robert [1:31:18]: Hopefully.

Miłosz [1:31:19]: Can do it.

Robert [1:31:20]: And if you are not writing in Go, join us as well. Maybe you’ll learn something new.

Miłosz [1:31:26]: Or you can learn Go in two weeks using our learning platform.

Robert [1:31:30]: Or maybe in one evening.

Miłosz [1:31:33]: Okay. Thank you everyone for joining us today. Thank you, Robert. See you in two weeks.

Is Clean Architecture Overengineering?

Wed, 02 Apr 2025 16:00:00 +0000

Quick takeaways

Clean Architecture is most beneficial for complex projects with larger teams - for small teams or simple projects, it can become overengineering.
Separation of concerns is the core benefit - keeping domain logic separate from implementation details makes code more maintainable.
It’s easy to go too far - using too many interfaces or too many layers without a clear reason creates unnecessary complexity.
Start simple and evolve your architecture - don’t force Clean Architecture from the start if your project doesn’t need it yet.
Understanding the “why” behind the pattern is crucial - blindly following it without understanding leads to poor implementations.

Introduction

In this episode of the No Silver Bullet podcast, we discuss Clean Architecture and whether it’s overengineering or a best practice for organizing code. We talk about why the pattern is often controversial, when it makes sense to use it, and how to implement it effectively. We share our experiences using Clean Architecture across different projects and teams, including the common concerns developers have when first seeing it.

Notes

Clean Architecture: a software design pattern that separates the code into layers, each with its own responsibility.
- Introduced by Robert C. Martin (Uncle Bob)
Related patterns:
- Hexagonal Architecture / Ports and Adapters (Alistair Cockburn),
- Onion Architecture (Jeffrey Palermo).
- While slightly different, they share the core idea of separating domain from infrastructure.
The Dependency Inversion Principle: one of SOLID principles.
Blog post: Introducing Clean Architecture
Blog post: Microservices test architecture.
Blog post: Repository Pattern in Go
Blog post: Combining DDD, CQRS, and Clean Architecture in Go (Mentioned as available)
Example Go Project: Wild Workouts
go-cleanarch Linter

Quotes

In general, if you ask many people about clean architecture they will say that it’s overcomplicating things and making your project worse.

Robert

One of the main points of Clean Arch is to separate the interesting code from the implementation details.

Miłosz

The goal is not using any kind of architecture. But the goal is to use something that is useful for you. It supports your velocity because at the end of the day, every technique that we’re using, it should allow you to code faster and that’s it basically. So if you are using some architecture and it’s slowing your down, you cannot iterate faster, it’s probably the bad one.

Robert

If you have been in a team that spent hours discussing in which directory to put some code, then you might assume you don’t want to work with Clean Arch anymore because it’s just it’s too much hassle to keep discussing how to use it.

Miłosz

If you are building more complex applications and you are starting to mix all this stuff it’s super hard to maintain that. And this inversion of control helps you to cut this logic in multiple parts and make it easier to maintain.

Robert

Don’t do it because you want your code to be clean, but to better organize work in a complex project that will grow over time.

Miłosz

Clean architecture is not that complex pattern. You have three, four layers depending on the complexity of your application. You’re ensuring that layer from inside doesn’t know about layer outside, and you’re using interfaces to do inversion of control. That’s it.

Robert

The pattern is not exactly about project structure, which is also easy to fixate on. All the layers are more concepts, not exactly directories, although this is often the case.

Miłosz

Timestamps

Transcript

Miłosz [00:05]: Clean Architecture is an overengineering nightmare or the best way to organize projects. It all depends on who you ask. To make it work, you need to understand the “why” behind it and avoid the common anti-patterns. Today, we talk about why the pattern is so controversial and how it can help you in your projects. I’m Miłosz.

Robert [00:25]: And I’m Robert. And this is the No Silver Bullet live podcast where we discuss mindful backend engineering. We spent almost twenty years working together in different projects and teams. During that time, we learned that following advice like “always do X” or “never do Y” doesn’t work and can limit your growth. In this show, we share multiple perspectives that will help you make smart choices that will help you to grow into principal engineer level.

Miłosz [00:55]: If you have any follow-up questions, you can ask them in the chat. We’ll have a Q&A session at the end as usual. And we might also pick up your comments during the discussion.

Robert [01:07]: This time, we’ll try harder to look on the chat and pick up your questions, especially that I think the today’s topic is pretty controversial one. And, yeah, I think it would be great to see some different point of views on that and also hear your experience with that. So we can also learn something maybe. How do you see that? And we can also create better content.

Miłosz [01:32]: Alright. So the title says Clean Architecture for today’s topic but the pattern actually has a few other names like Hexagonal, Ports and Adapters, or Onion. They are not exactly the same thing. There are some differences in the implementation. But I think we can assume that the main idea, the core idea behind them is similar. Right? So similar that we can treat them as pretty much the same pattern. And we won’t dive deep into the differences today, And we’ll just stick to the clean architecture just because that’s how we started calling it and I think it’s quite popular anyway.

Robert [02:18]: Probably the most common term because, yeah, you could probably heard some terms like hexagonal architecture, ports and adapters, onion architecture. But, yeah, the idea on a high level is quite similar.

Miłosz [02:33]: Yeah, and while it’s popular I think everyone understands it a bit differently which is common with complex patterns.

Robert [02:41]: And probably this is the reason why there are also a lot of misunderstanding about that because many people actually are messing clean architecture with domain driven design. So some people don’t have a good idea how to connect that because, okay, in many clean architecture implementations you have the domain layer. So for some reason it’s like, okay, you have domain layer so we are doing domain driven design. Or some reason also some people also think that okay, clean architecture is some Java or it’s some enterprise. It’s not something that you can use.

Miłosz [03:14]: That’s only for enterprise super complex projects.

Robert [03:17]: Yeah, yeah. And in general, if you ask many people about clean architecture they will say that it’s overcomplicating things and making your project worse.

Miłosz [03:29]: Which is problematic because if you don’t understand something, chances are you won’t implement it properly in your project. So that’s risky. And I think often it’s also a very controversial pattern. If you mention it to someone and they had bad experiences from the past usually they won’t react well, like, they won’t be happy about you using the pattern, let’s say.

Robert [03:58]: Yeah, and I think it’s interesting that we worked in multiple teams and usually we have some people that were not happy about using clean architecture but at the end we’re kind of able to convince them that it’s nice. Probably we can have one person confirm that. So Firmino said that he’s missing working with Milosz. So it’s a good sign. So at least one person is missing working with you. Maybe clean architecture helped a bit.

Miłosz [04:28]: Thank you Firmino, at least one person misses me. You can mention if you like clean architecture or not. So it might be one example for today’s discussion. Maybe it’s not that bad.

Robert [04:41]: What I think is interesting about that the thing that I already mentioned that you know, we were leading, multiple teams in our lives and it wasn’t uncommon for us to have some people coming from other projects. And, well, we had pretty good experience with clean architecture and they were coming to our project and most of the people were like, oh, like, okay, it’s cool. Please tell me how it’s working. But some people, I think usually some people who have already done have more experience or thought, oh, no, clean architecture or over engineering stuff. But in most cases, it was like, okay, it took some time, like, it wasn’t weeks. It was like sitting with this person for a half hour showing how everything works. And at the end, most of it people were saying, yeah, it’s actually cool. I’m using it in my projects after the job. Actually, I remember one person even said that he was building a game and he used the architecture in that because he could exchange, I think, engine or something like that. It was quite cool.

Miłosz [05:43]: So does it mean we should use it everywhere? Would you say it’s a silver bullet?

Robert [05:52]: No. But, you know, I think pet projects after work, it’s probably a pretty good place to experiment and do some crazy experiments. But, yeah, long story short, I think we’re pretty surprised to later hear that it’s not everywhere like that, like in our team. So there are people that try to clean architecture and at the end, they were saying that it’s always bad. And we started to wonder what is the reason why people hate architecture in all projects.

Miłosz [06:24]: Maybe we can answer it today. But maybe let’s start with a very short definition. Let’s try to boil it down to the very core ideas because there are many flavors and, you know, even if you use one of the flavors, you can use it in a different way than someone else. So let’s just try to agree on what’s behind the idea. For me, the first one is separation of concerns, and it often comes with the layers idea in those architectures. But after years of using it, I think you could just summarize it as, you know, you agree that your- project has code of two categories, The boring code, which is the implementation details that you don’t care that much about.

Robert [07:17]: Or you should not care. Because, unfortunately, often it’s the opposite.

Miłosz [07:21]: Yeah, like the SQL queries you use or what database engine you use. It is important, of course, but probably doesn’t make your business win automatically. And on the other hand, there is the the interesting code, the domain code, the logic of what your application does, sometimes called business logic. But I think this is a bit unfortunate term because it, you know, it again has those enterprise vibes. And I think people writing usual web applications maybe don’t think of business code when they write the logic they do. But the fact is it is some kind of domain code anyway. Right?

Robert [08:05]: Yeah. But it’s worth mentioning that it depends heavily on what kind of application you’re working on. So just a kind reminder. So this is no silver bullet podcast, live podcast, and, yeah, it can suggest that it depends on different projects. And so as you said, so sometimes you have business logic, but sometimes it’s as simple that you can have it in controllers mixed with SQL queries. But sometimes it’s not that simple. I mean, you can go this way, but and we did it in our lives but it’s a nightmare a bit later.

Miłosz [08:37]: But I think we can agree that the main point of one of the main points of CleanArch is to separate those two, right? To separate the interesting code from the implementation details.

Robert [08:51]: Yes, and probably it’s worth mentioning how it’s done. So it’s all about the well, I think it’s not the first time when we’re mentioning that there are some old patterns that are still valid and the same is for, inversion of control. And the idea is that you can implement, you can implement so you can define interface on some behaviors that you expect for example, from the database, from external providers. So and thanks to having interface for that your implementation doesn’t need to know really how it’s implemented under the hood. And, again, if you’re creating a simple application it’s not a big deal because the logic is not that complicated needed there. But if you are building more complex applications and you are starting to mix all this stuff it’s super hard to maintain that. And this inversion of control helps you to cut this logic in multiple parts and, well, make it easier to maintain.

Miłosz [09:52]: And it’s not that complicated, right? Inversion of control sounds a bit complex but in reality this is accept interfaces instead of implementations.

Robert [10:03]: Yes. But we’ll come into that because, well, like always the idea is simple but in practice people unfortunately are overcomplicating that and there are multiple ways on how you can overcomplicate that but we’ll touch on that in a while.

Miłosz [10:19]: So we have, separation of concerns of the domain and implementation details and second inversion of control of dependencies. Often there is some concept of layers, which are basically groups of code you use that are dedicated to something. So often we have the Adapters, which are for, you know, external communication of external things like databases or API clients or even for entry points to your applications which we like to separate often.

Robert [10:58]: So in other words, it can be calling other microservice or other module. So it it can be database, it can be external API, external by not your company but well, if you are building real microservices, you know, other microservice should be kind of like separate, something totally separate for you.

Miłosz [11:17]: And this layer is quite boring usually. It just maps your code onto some external code and that’s it. It should be boring. Then there is some kind of application or use cases or controllers maybe, layer where the logic is or orchestration of your application. And sometimes also domain layer which is this pure business logic layer, which does not always have to be.

Robert [11:49]: So the second layer is so again we’ll use multiple names for that because as we said that at the beginning so there are multiple names for let’s say similar idea like clean architecture, ports and adapters etc. So we’ll just use multiple names to be compatible with more. So the second layer is interfaces or entry points or sometimes ports. And the idea is that this is the place where you’re defining your HTTP handlers, your gRPC handlers, your message handlers, even CLI. So any entry point to your applications. The idea is that any operation that is coming into your application, it should go through this layer. And what’s super cool about this, layer is that it’s allowing you to have different entry point for the same operation. So, for example, let’s imagine that you have application that allows you to create a customer and for some reason you need to create customers through message and also through HTTP API. It’s nice because at the end you can do the same operation share a bit less code and support multiple different ways of interacting with your application from external world.

Miłosz [13:00]: I really like this approach because, you know, when when you take a look at the project, it’s very easy to see what are the entry points. So it’s very clear where is the API if you want to interact with it.

Robert [13:13]: And it works pretty nicely with the pattern that we touched in the previous episode. So MVC. So this is basically your controller from your MVC.

Miłosz [13:24]: But, given all those layers, it’s also important to keep in mind that the pattern is not exactly about project structure, which is also easy to, you know, fixate on. So all the layers are more concepts, not exactly doesn’t have to be directories, although this is often the case. But we probably will touch on it a bit later.

Robert [13:57]: So the next layer. I think this is pretty problematic layer because I think many people don’t understand what this layer is but this is application layer. So this is the layer that is doing the orchestration of everything. So I think it’s not probably the best place to describe how it can look in practice. At the end, we’ll give some materials that we believe are covering how to implement application layer nicely. But, yeah, long story short, this is the orchestration of what you are doing in your application. And this should be called by your ports.

Miłosz [14:31]: Ports or interfaces or entry points.

Robert [14:35]: And it should call your domain layer if you have this layer, and your adapter’s layer, long story short.

Miłosz [14:44]: So that’s the parts of the puzzle, I guess. The separation of concerns, and inversion of dependencies and layers, right?

Robert [14:57]: And I think we forgot about one layer or maybe you mentioned it. So the domain layer. The optional one because I think it’s worth mentioning that it’s sometimes useful but not always. And but it yeah. It’s the layer with the logic. So the layer that should be free of any database logic of any adapters, it should be, relevant. Again, it’s probably something for I think it’s better to read code than talk about that to show how domain layer works. But again, we’ll give you some material.

Miłosz [15:28]: It’s probably also not the place to go into details on application versus domain, but you may find some of it on our blog.

Robert [15:37]: Let’s maybe take a look on the chat so I’ve seen a lot of messages there so we can catch on it. So, yeah, there are some voices that, yeah, I like it often and use it often and like it. So, yeah, we have some positive things about clean architecture. The second comment, I love that clean architecture and your articles helped me to show how to code is not about language but about structure and functionality. Thank you.

Robert [16:05]: We will later link to some articles that we have on our blog. So, yeah, I think it will be just explaining better how those layers work because, again, it’s pretty hard probably to just talk about that without showing the code. What else? Nice topic. I believe that we shouldn’t face architectural choices in dogmatic sense. Definitely. We’ll touch on that a bit later.

Miłosz [16:28]: Yeah, we’ll touch on it.

Robert [16:30]: Oh, another person missing working with Miłosz.

Miłosz [16:34]: Even though we didn’t work together. Yeah, maybe in the future Oleg, who knows.

Robert [16:40]: Greetings. Yes, so there’s one comment about overengineering. I see when using interfaces a lot. A lot of projects are created with a single interface each struct. We’ll touch on that also.

Miłosz [16:57]: Yeah, I think that’s a very misunderstood part of the pattern like how to use interfaces.

Robert [17:04]: And it’s actually, I think, pretty easy to overcome and also to have some way of understanding when needed problem. But no worries, we’ll touch on that. It’s like using abstraction without being abstract.

Miłosz [17:19]: Yeah, that’s a good point.

Robert [17:20]: I think it’s exactly that. But again, we’ll go into that. As I understand clean architecture, you can’t depend on layers that are deeper one level. So you have to duplicate structures every time that makes me sad. Yes, but not always. I think we’ll also touch on it. Okay, so hi, I work at a big e-commerce company with 10k+ IT area with 4,000 microservices. That’s the scale. Okay, to give some context, there is an area which is a clean architecture for most of their microservices. So this is probably around what we said a bit earlier so Or maybe not.

Miłosz [18:12]: And there is a follow-up, I think you need to include this.

Robert [18:15]: You can touch on it.

Miłosz [18:18]: Yeah, it’s implemented across apps and libs, all Java of microservices. This fact implies that all the areas have an important learning curve to understand and add code here. Yeah. So you mean that there’s, bigger learning curve to understand the code because of it? Which can happen, and we will also cover this.

Robert [18:51]: But I think it depends a lot on how you’re implementing that. So again, I would say that it should be some hint like if it’s taking a week to explain a person how can architecture clean architecture work in your company probably something went wrong and it’s maybe not really clean architecture but something around. Because, again, explaining clean architecture and showing it in the code, it should be quite quick.

Miłosz [19:14]: Should be, yeah, but not always. I think people always start with this big chart of different circles and shapes and it can be confusing at first. Yeah. I think we will talk about controversies in a bit. So let’s start there.

Robert [19:35]: So maybe let’s finish with the comments. So, people write the interfaces next to concrete type when it should belong to the client code. Only including the methods itself depends on common mistake.

Miłosz [19:48]: So this is exactly this misuse of interfaces. Right? So if you do it this way, then there is no inversion of control, basically. Or maybe there is but gives you nothing in return.

Robert [20:06]: So another thing that we’ll also cover, so no worries, is starting a new project with this architecture. We’ll cover that. How to explain to my colleague that it’s worth using this approach? I think we’ll also cover that. And again, later I will also recommend checking the articles that we’ll link a bit later. So we are also in those articles giving some context where when it works and why it’s, makes sense. Okay. And to catch on the last comment so far, I’m really proud of you guys. Oh, thank you. Because you have decided to run your own company as your main. Now, we’re also pretty happy about that. Alright, cool. So, now, we’re finished with the comments so far. Again, if you have anything to comment in, we’ll be happy to go over it. Alright. So layers. We talked about layers. I already mentioned that it’s not necessary to always use all layers and I think it’s one reason of reason of one common mistake that people are always using the all layers and because of that it’s making it more complicated. So somebody in the chat mentioned that it sometimes hurts because you need to copy structs between layers and it’s a waste of time and it’s true. But it’s mostly true in cases when it’s not needed because you’re just using all layers when its project is not complex enough.

Miłosz [21:39]: You start with the big idea instead of seeing where the project leads you. Especially if you start small. Probably, it’s better to let the architecture evolve over time than to force some complex ideas in the beginning.

Robert [21:55]: I think we’re always mentioning that that, you know, you need to be pragmatic at the end of the day. So if you see that pattern is creating more mess and require more effort that it’s giving you back probably it’s something wrong because at the end of the day it should help you and, well, that’s it. And it’s connected to the dogmatic approach of that. I think dogmatic approach is pretty often the case why it’s over complicated because often often there are some people that are saying no we need to do it in this way. It needs to be consistent with everything else.

Miłosz [22:29]: We like consistency. And I think the topic that we talk about in every episode just keeps coming back.

Robert [22:36]: But, if you didn’t have chance to check our previous episode about the low quality code highly recommended because it’s also connected to that. Right, but let’s go back to the architecture.

Miłosz [22:52]: So maybe just to summarize, I think you can it’s very easy to to say what will be the the opposite of the architecture. I think that will be having all your code in an HTTP handler. Your logic, your validation, and, your database code is all mixed in the same handler or controller. I think that will be the opposite, basically.

Robert [23:20]: And it’s a better approach? No.

Miłosz [23:23]: It depends.

Robert [23:23]: We’ve been doing that. Sometimes it was good enough. Sometimes, well, I hope nobody needs to maintain this codebase any longer.

Miłosz [23:37]: Somehow clean architecture is quite controversial, right? We already mentioned this. And I think for us it was quite surprising because when we started using it, we were quite happy about it and people in our teams were also quite happy. We even heard of them spreading the pattern to other companies as they left our team. Yeah. So why do you think it happens?

Robert [24:10]: I have one theory. So I think that the theory may be that we are trying to be extremely programmatic. So I heard that I heard it multiple times. So if we have seen that somewhere it was making our lives harder than more helping us it was probably we’re not using this pattern in this case for you and we’re not blindly following that and I know that it’s opposite compared to other companies.

Miłosz [24:39]: You mean if someone forces you to use it? Or maybe someone someone has seen it implemented in a way that wasn’t the best to work with?

Robert [24:50]: Or even worse, somebody is forcing you to use that without understanding it. And this is the ultimate combo.

Miłosz [24:57]: For the sake of consistency, for example. All microservices in our company will follow Clean Architecture. So, yeah, that might be controversial and that might scare people, I guess. And for some reason, this is tends to divide people. And often I hear something like, you must be Uncle Bob cultist because you use those patterns. So it starts to get almost political maybe, I don’t know. It’s funny.

Robert [25:38]: But for what I can see in the chat? I can see that some people actually also see it pretty pragmatic. So, I don’t consider hexagonal architecture over engineering at all. When building a bigger than just couple of weeks project, first, it allows to register stubs for our own implementations, details, and tests. Yep, it’s true. The second comment saying that I believe clean architectural learning is not that hard. Yep, that’s true. And we believe that it’s true. I have seen people moving to similar way of development without knowing those concepts exist. Yeah. But I think we can compare that. And actually, one team will have ultimate combo because we’ve been building a project in Go that and that time Go wasn’t that popular language and 90% of the team was learning Go and Clean architecture at the same moment and it wasn’t an issue. Maybe we’ve just had a great team. It was probably also important reason there and very open minded and great team. But, yeah, at the end, it was cool.

Miłosz [26:57]: I think you skipped one. I think you skipped one comment.

Robert [27:02]: Did I? Yeah. Hi folks, I’m wondering how well clean architecture matches the lang’s initial idea. From the little that I’ve read about it, I had feeling that Go apps were supposed to be a little simpler. Oh, yeah.

Miłosz [27:24]: My favorite topic of keeping it simple and we will touch on this and on Go later. So stay with us, please.

Robert [27:36]: Right.

Miłosz [27:41]: Those comments are very much about the architecture not being that complex in the first place. I think maybe it is clear when when you learn it. But I think if someone doesn’t know it yet and sees a codebase, then it can become quite confusing. Right? I remember once when our team was using it, but other team in the company was not, and people there were not aware how it works. And for some reason, they had to debug something in our codebase one night. And that was quite confusing for them because if you don’t know the rules, know where the dependencies are, and so on, it can be quite confusing at first.

Robert [28:31]: Yeah. But I think important part of this is being kind of open minded because if somebody is like, oh, clean architecture is bad over engineering. I don’t want to look on that. So obviously, if somebody have approach like that, well, it will not work. But if somebody’s open minded and he’s open to learn how it works, it helps a lot. Yeah. But And I think that that was part of the problem that other teams were like, no, I don’t like this idea and why we are doing that?

Miłosz [29:03]: We like consistency, as we said. So, you know, if we use the simple Go mindset, for example, let’s say, it can be it can add to this controversy of using it. I mean, that might be one of the reasons.

Robert [29:19]: And it’s also worth mentioning that often within a company you have multiple teams solving different problems. So some of the teams may be solving some simpler problems that doesn’t require some advanced techniques and some teams in the company may be working on very complicated domains. And it’s not a matter of using any tool that it’s complicated. It’s just complicated by design and you can help to tackle that, but it’s just helping you to make it simpler. But it has some base complexity, let’s say, that you can try to tackle, but it will be always complex because just logic is super complex.

Miłosz [29:56]: So should we talk about some common issues maybe? How it fails basically?

Robert [30:08]: Yeah. Because we were discussing this topic a bit earlier with people on some social media or in real life. Yeah, it was a very common topic to understand why it’s not working in multiple projects. So we gather multiple reasons that we know that it’s leading to making the architecture overcomplicated or even project failing. By the way, if you have any, any stories from you, also please leave it on the chat so it will be also cool to hear about that. Right. So the first thing is, over engineering architecture. So as we said multiple times and it was also mentioned multiple times on the chat, clean architecture is not that complex pattern. So you’ll have three, four layers. It depends on the complexity of your application. You’re ensuring that layer from inside doesn’t know about layer of in the outside, and you’re using interfaces to do inversion of control. That’s it. But, unfortunately, people are going sometimes a bit too far with that.

Miłosz [31:14]: Yes. Because what you said doesn’t sound like proper architecture. It reminds me of this, what we discussed in previous episodes about this belief that we can figure out the architecture now, and in the future, we will just cookie cut the services or applications. It sounds a bit similar to me that we try to figure out the perfect architecture first, all the rows in place and layers and so on, and then just apply it. It sounds much much better than just following some simple rules of using interfaces. And it’s easy to go too far, right, and become a purist and just force this idea everywhere.

Robert [32:06]: So, overconsistency, let’s say.

Miłosz [32:09]: Yeah. This is the hard part, when you want to be consistent and when there is too much consistency.

Robert [32:19]: It’s always a cost. so the second thing that I have in mind about clean architecture and I think this is probably the most common one so it’s and I think somebody mentioned it on the chat so it’s using too many interfaces. So in general, the interfaces are needed if you are kind of communicating between layers so those layers inside will not know how what’s implemented then. But that’s it. But very often I’ve seen that within the layers before putting more interfaces. For example, you have domain layer and under the hood it has multiple interfaces. And what’s even more funny often it’s under the hood have only one implementation. And it’s pretty problematic because one, it’s harder to inject that because your dependency injection is starting to be super hard. And this is something that is also probably common thing common thing between between multiple episodes because people were saying that they have complex of complex problem with complex dependency injection and from my experience, it’s often the problem is that your dependency tree is too complex. It’s not a problem with some dependency injection framework or whatever you are using. So if you’re not able to inject your dependencies by hand, probably something over complicated.

Miłosz [33:37]: It’s part of the over engineering, I guess. You know, just forcing the ideas because there’s a rule that says accept interfaces because you might want to change them with something else, and then you create some domain service, let’s say, that accepts some some structure. You might you might go and think, oh, maybe I will use an interface here because I am supposed to use interfaces, right? Maybe in the future we will want to change the implementation. And it’s exactly this trap that you take the base ideas and just use them everywhere even when it where it doesn’t make sense. So maybe I no. A good a good a good heuristic here is to think of, what what you want to use mocks for in your tests. That’s probably what you want to use an interface for.

Robert [34:30]: Yeah, and obviously between layers but this is a different case. And I think it’s very important to mention that, okay, if you are mocking things you should probably mock only things that are IO in general. So it’s database, some external API. You should mock not mock your domain logic because you have no benefit of that. So your test will be worse because you will be not checking the, the real code plus well, it will not speed up your tests basically. So mocking makes sense in most cases when you need to speed up your tests or or simulate some external logic. In your domain code, you should be able to do it through your public API.

Miłosz [35:13]: Exactly. You can tell you went too far if you’re, you know, if you use too many mocks and all your testers do are checking some function calls on the mocks you used. It probably means you went too far and you don’t need to mock that much.

Robert [35:30]: Yeah, but there are some exceptions. So there’s no silver bullet. So exceptions are very very very complex domains. So sometimes you may have some complex domain and within the domain layer, you may have interfaces to hide this complexity somewhere. But you never should start with that because it will go in the wrong way. So I guess, I don’t know, within one year probably it will be not possible to have such the complex domain code to, need to have really layers within layers. A couple years old codebase that is super complex, maybe, but I would watch out for that.

Miłosz [36:07]: Layers within layers actually sound like the Great idea. Yeah. Like the You’re so smart. The next pitfall, right? It’s again over engineering in the layers area. I remember this Reddit post. You know which one? The other day at the Golang sub subreddit there was a Reddit post where the OP said something like, Don’t use pin architecture, it’s over engineering. And of course it, you know, invited me to click. And in the content they said something like, you should have at most three layers in clean architecture. And I thought, yes, of course, what’s the problem? Again, this is how different the implementations can look like. I guess what this person found in the code was, I don’t know, six layers? I can’t even imagine what was there. But that’s another issue, you know, if you just take the idea of layers too far.

Robert [37:24]: Another surprise. I mean, if I would work in projects where you have six layers at least, I would also hate using that and would love to go with a simple approach, obviously.

Miłosz [37:36]: Yeah. So maybe it’s good to start with the basics. So you want to separate the implementation details from the logic, from the domain code. And that’s two layers for starters. It doesn’t have to be much more. You can separate the entry points if you want. That’s kind of a hard layer, but I would call it part of the adapters as well. It’s a bit similar. I’m not sure why would you use something beyond the domain layer. What’s the fifth one? I don’t know.

Robert [38:16]: Maybe you know somebody is using layers from hexagonal and onion architecture.

Miłosz [38:23]: Yeah, maybe. So if you see more than four, I would be very, very careful. I would ask some questions.

Robert [38:33]: Definitely. The next thing that I have in mind about the common problems about clean architecture is using it everywhere. So one point that we’d like to, you know, give you with this live podcast is that there’s no one single way to use something as we’re mentioning, in the intro. So if somebody is saying you always do x or never do y, It’s pretty big chance that this person just work in some specific type of projects, but it’s not universal advice.

Miłosz [39:09]: Or they want to sell you something.

Robert [39:11]: Or this. Some shortcuts as usual. But what is the message? So the message is that, you know, clean architecture is nice, but it doesn’t matter that you need to use it in an entire company. So the thing that I mentioned a bit earlier. So often in company, you have teams and services that are complex and simple and using the same approach for everything is just crazy because in some places it will be over engineering and sometimes it will just suffer from from the complexity that is not handled properly and it will be also the problem.

Miłosz [39:51]: Sometimes even in the same project in different modules, it also has some trivial domain that doesn’t need it. Just what we have in the White Workouts, our example project, the Users module or service is basically a Scrub domain with no layers at all. Yeah, and it’s different for your past projects because if

Robert [40:14]: You are working on something after work, it doesn’t matter that much. I think it’s worth trying multiple approaches to have your opinion on how it works. But in a job, it well, it’s worth to be pretty mindful. And if your project is not complex maybe clean architecture may be not needed but sometimes it may be worth with starting with some simple one without domain layer, for example. It doesn’t cost that much. But if you know that your team is bigger, then it can make sense to start with that.

Miłosz [40:46]: Yeah. And and some apps are are may most of them are not be a good fit or maybe if you create a library, for example, maybe it’s not the best idea because the users might not be familiar with it or for things like CLI apps maybe or something infrastructure related.

Robert [41:06]: And you if you think about that, so libraries are usually living in inside of some other layers. They are not covering multiple of them. So usually, it’s probably adapters or ports. Like, for example, Watermill library that we’re maintainers are also It’s living within ports and it’s not going to other layers basically.

Miłosz [41:26]: Yeah, exactly. So you don’t want to tie it with your application logic or the domain. A similar one is Project Structure, which we mentioned before. It also keeps coming up in our previous episodes that it’s easy to focus too much on the project structure in the beginning. Again, it’s this idea of, you know, let’s prepare this great environment, great architecture, and then we can just keep adding features. Most of the time it’s not that easy and the project will evolve anyway, in some way you don’t know yet. And you can always refactor, right, if it changes. So I wouldn’t start with thinking of the, you know, what’s the perfect project structure or what’s the perfect number of layers. Just start with the basics, something simple and then refactor as you go.

Robert [42:40]: And assume that you may be wrong. I mean, we have almost twenty years experience with coding in general. Also pretty much years, you know, with architecture and still we are not always right. But I I think it’s important to know how you can be not right and how to handle that and not go too deep into the wrong architecture, basically.

Miłosz [43:05]: It’s probably not possible to invent the perfect structure in the beginning anyway.

Robert [43:16]: If somebody is saying that he can do that, don’t trust this person.

Miłosz [43:21]: They want to sell you something.

Robert [43:25]: And overall I think it’s if you’re thinking about some pitfalls of architecture or problems with that. So the most important thing that I believe made our teams happy about using this is keeping in mind that it should be helpful for you. So the goal is not using any kind of architecture.

Miłosz [43:44]: Oh, it’s a big one.

Robert [43:47]: But the goal is to use something that is useful for you. It supports your velocity because at the end of the day, every technique that we’re using, it should allow you to code faster and that’s it basically. So if you are using some architecture and it’s slowing your down, you cannot iterate faster, it’s probably the bad one.

Miłosz [44:12]: I mean, it’s very easy to start being dogmatic about it, to become like a purist. And it is tricky because we have a bigger team and it helps to have this consistency around the project. So Clean Architecture gives you this consistency in supports. But then there might be places or features where it’s not a great fit. But if you want to break the rules, then someone needs to decide if it’s fine to break the rules or not. And it often starts those endless discussions about it doesn’t follow clean architecture or it breaks.

Robert [44:57]: Which layer it should be in? Is it domain logic or it should be in domain or application layer?

Miłosz [45:03]: Okay. Let’s call it a meeting. A meeting to discuss is super important. And sometimes it is important but sometimes it’s not. It’s not. It’s just, you know, a detail.

Robert [45:14]: And, unfortunately, this is the topic that we cannot give cannot give you a simple ask for like always do x. The best way, unfortunately, here is to just do it over time and learn by doing and probably this is the best advice that we can give here. But, well, it’s hard to have any shortcuts here because there are just many different kinds of applications. And, again, it’s important to be open minded. Also listen to your teammates because it’s also easy to be, you know, the architect or the person that gives architecture. And if you are going to holidays and going back, you see that it was all removed. It’s probably the sign that.

Miłosz [45:54]: Have to change everything.

Robert [45:55]: So, you know, it’s also probably the sign that, okay, maybe something that you invented is not that helpful for people because if it would be if it would be helpful for them, they would not remove it during your holidays.

Miłosz [46:08]: I think this this factor is for sure part of the controversy around it. Because if you if you have been in a team that, you know, spent hours discussing in which directory to put some some codes, then you you might assume you don’t want to work with Cleanarch anymore because it’s just it’s too much hassle to to keep discussing how to use it instead of this right code. Yeah. And especially if you didn’t have the proper onboarding for new people

Robert [46:39]: In your team because often, you know, people are changing teams, changing companies. And this is also important thing to ensure that, okay, the new person is joining the team, the person will know how it works in your company and how to navigate about that. Because if you will not do this, the person may be not happy about it.

Miłosz [47:00]: Yeah, it’s probably best to show them instead of having some documentation because I think it’s easier to show it as an example maybe than reading about something. Definitely. But on the other hand, I think clean atch also enables some nice onboarding because you can have those you can have new joiners implement something quite simple because of the separation of layers. Like, you can give them an HTTP handler to implement and it will be a quite simple task. They can probably do it in one day and get started with the codebase.

Robert [47:41]: And again, we’ve mentioned it already a couple of times but if explaining to some person your clean architecture takes more than half hour, probably it’s something wrong there. So we can say probably it’s some not very accurate metric but more or less like this.

Miłosz [48:06]: So overall, don’t ignore developer experience. Think this would be the summary. Yes. I think it’s very important because it’s basically what everyone will be working with daily. So you just don’t make a religion out of it at the end of the day and don’t try to use arguments like this is not clean. It’s the worst worst kind of discussions you want to have. You don’t want to have them.

Robert [48:39]: It’s probably worth mentioning to also set some expectations. So if you have everything in your HTTP handler your dependency injection can be as simple as it doesn’t exist. If you have clean architecture, it’s more complex because you need to inject more things. But again, it should be manageable by doing it by hand and okay, it depends on the language a lot. So with Milosz, we are mostly working with Go where there is it’s compiled language. So it’s a bit harder to do some magical dependency injection. But it’s actually quite good because because you need to do it. So in our opinion, it’s easiest to do it by hand. And if you’re in the position when it’s too complex to do it by hand, probably it’s showing you the problem with your maybe clean architecture, maybe architecture in general, but it should be doable by hand. In other languages, we know that it depends. So we also worked a lot in PHP where it was working automatically. Sometimes it was nice but it was harder to debug with more complex dependency tree.

Miłosz [49:48]: There are some libraries usually. I’ve seen someone in chat mentioned fx. We haven’t used this one, but we used wire from Google. It was okay. But currently, we just wired wired it up manually in in the main function, basically. And it it grows. It’s big. But most of the time, that’s not an issue, really. You just fix it and if it compiles you’re good to go.

Robert [50:18]: And I think it’s one risk maybe that some people maybe don’t see at the first sight that but, you know, it’s something making your it’s something making your super complex dependency tree easier to handle. It’s actually opening some kind of gate of hell because you can do we can make this dependency tree more complex if something is helping you with that. Are going deeper and deeper into.

Miłosz [50:42]: Those layers within layers. Interfaces within interfaces.

Robert [50:46]: If you are doing it by hand probably you sooner see that something is wrong there.

Miłosz [50:53]: It’s also a good general rule to start fixing the pain once you start feeling it because if there is no issue maybe you don’t need all these solutions.

Robert [51:03]: And if you are starting to blame dependency injection framework probably the problem may be not in the dependency injection framework but between laptop and wall.

Miłosz [51:17]: Tough pill to swallow, but sometimes, yeah.

Robert [51:20]: Yes. Firmino is actually mentioning that even in Java, dependency injection can be easily done by hand but developers could use using auto wired from Sprint. Yep. That’s true. But again, we’re also not that against using such frameworks but again, you need to watch out. It’s like with every tool. Okay, not every tool but there are some tools that are encouraging you to do bad things and probably dependency injection framework is on this list. But again, it all have its use cases like ORMs also, I would say. So many people also doesn’t like ORMs because it can also lead to some issues. But, in my opinion, it’s like with knife in the kitchen. I mean, you can hurt yourself with a sharp knife. But from other side, it’s also pretty useful. I like sharp knives. I think you also like You need to just watch out. So that’s it. It’s much better than using blunt knife. And, yeah, the last thing that can lead to problems with clean architecture and we have some solution for that fortunately is using the wrong resources. So many resources that we’ve seen in the internet are unfortunately missing the point of clean architecture and it’s encouraging to use it in the wrong way that is far from being pragmatic. Probably doesn’t help that many of those clear architecture hexagonal onion architecture they don’t have super strict rules on how to implement that and it’s nice because it’s giving flexibility but it’s not nice because it’s giving flexibility on what you

Miłosz [52:58]: Have to decide for yourself. And it’s exactly where this starts getting difficult because you’re just starting out and there are no, definitive rules. And this is where you start thinking, okay, what’s my perfect project structure? Which can be a trap. And I think many of those open source examples often miss the point or they show some very trivial domains like, you know, to do apps that have two entities and two endpoints. And if you take a look at this as such a simple domain implemented in the architecture, it will look like over engineering, of course, because it makes no sense if you can fit this into files to split it across the project. So you have to be careful of this.

Robert [53:53]: But there is also one, tricky part about this because so we have run repository in Go where we are showing how to you can connect everything. And I think it’s, you know, fully worked application. It’s not that complex still because it’s still some more pet project even if it’s fully functional. But also, if project is too complicated, it’s also hard to grasp the most important parts of that. So it’s it’s all about the balance. So as mentioned multiple times, at the end of the we’ll give you links to the useful resources. But, yeah, it’s important to just use those resources.

Miłosz [54:35]: Yeah. I see an interesting question in the chat. So does clean architecture fit in projects with lots of tiny and simple microservices or better fits in a monolith application?

Robert [54:51]: It depends. There is no silver bullet.

Miłosz [54:54]: Yeah, what’s a simple microservice? I guess that’s that’s the question. I I hope if you have, a monolith, it is split into modules just like those microservices, and then it’s pretty much the same answer. If if the module or service is complex enough, to benefit from Cleanark, maybe a good idea to implement it. If it’s truly trivial and crowd domain, probably not.

Robert [55:24]: Yeah. But I think there is one risk that I’ve seen multiple in my life. You’ve seen it as well. So because microservices are cool for splitting complex problems into smaller but the problem is that often people are splitting it to granular. So you have multiple so instead of having one service that can handle some, use case you have five, ten microservices that are handling that. It’s okay. Maybe every service is pretty simple and you don’t need to have clean architecture inside of that. But each time when you’re implementing a new feature it takes changes in five or 10 microservices.

Miłosz [56:03]: Yeah, because because it’s different entities, I guess. Yes. Yes. My favorite from the past is the Avatars microservice. Stored Avatars.

Robert [56:14]: And, you know, if you have sub granular microservices, okay, and then you split at your complexity but if implementing one functionality takes changes in 10 microservices, I’m quite sure that it will be much easier to implement it within one service that is not maybe microservice, whatever microservice is. But doing it in one service will be easier. And okay, this one service will be more complex but Clean Architecture can help you with, handling this complexity because you don’t need to split this complexity in multiple services. You can instead split it by layers. And it’s making your so this is the thing that we’ve done multiple times. So it was 10 microservicization operations. So moving multiple microservices into one service and obviously this one service was more complex but clean architecture was helpful to handle this complexity because we didn’t have one big sum of everything but it was data in multiple layers and at the end implementing features was much easier. So, yeah, I hope it’s useful.

Miłosz [57:21]: Yeah. Looking at the chat, I’m gonna read the next one. So, Tony says, For me, the most difficult part of clean architecture is when implemented in a team and different people diverge in some details and the mess appears when little by little the project becomes less consistent. This is exactly the developer experience issues we talked about previously. I think it’s very hard and probably what can help is having some team leader in place who can just cut the discussions and make a decision. Because otherwise, yeah, you can take long hours discussing, and it’s natural that some people in your team will be like consistency more. There are many developers who strive for this consistency and like to have everything neat and tidy. And if someone starts doing something in a different way they will be annoyed and you want it to be the same way. And I think it’s probably worth mentioning that,

Robert [58:33]: At least, I believe that democracy doesn’t work in programming teams, unfortunately.

Miłosz [58:38]: Yeah, unless you have infinite time and can spend days discussing and voting.

Robert [58:42]: So probably sometimes usually something closer to totalitarianism works a bit better. But better to not overuse that obviously. But it sometimes helps with these discussions.

Miłosz [58:55]: It’s a tough balance, I would say. You want some discussion with the team but you also need someone to say, ‘Guys, let’s just focus on important stuff.’ But it’s very hard to make those know, exceptions because if everything is consistent, why would you use a different way in this in this case? You know, maybe it’s just more easy this way. Maybe it’s fine. But if you do advanced then, no, people exactly what what the comment says people might think that, oh, okay. It doesn’t need to be consistent so let’s just do whatever.

Robert [59:38]: And then also, if you’re not a team leader but team leader is doing a bit of totalitarianism when you are going too deep into some discussions also don’t be mad at this person because we’ve been in this position and, you know, it’s also not that hard. But well, after you are in this position, sometimes you see that it’s easy to overdo some discussions and that’s it. So sometimes at the end of the day, it’s better to have something that works, but it’s not perfect, but it works because sometimes you can throw it out. Probably you can listen more in the previous episode about code quality.

Miłosz [01:00:13]: Yeah. I think it’s one of the most difficult parts of implementing clean arch. We don’t know definitive answers here.

Robert [01:00:24]: Yeah, there’s also a nice comment about frameworks. Frameworks like Rails that define a clean directory structure seems to be harder to apply that architecture and I would say it’s still possible. So I don’t have that much experience with Ruby on Rails but I worked more in PHP with PHP frameworks like Symfony or whatever and those frameworks usually okay, it requires some effort but usually you can hide them in some layers. So mostly those frameworks are about your ports but it’s also sometimes a bit application layer but it’s doable and it’s often worth doing but often it’s also not worth doing. So it’s getting probably hard to give some universal advice here but you can do that in general. And sometimes it may be beneficial for more complex domains because those frameworks are nice and we’re discussing it in the previous episode also about frameworks. But long story short is that they can give you some velocity at the beginning but later it may be harder to do some more context scenarios. And if you went with this framework structure all in, it’s much harder to customize that as and sometimes, you know, the product after a couple years, it’s still within the framework and we work sometimes in such companies and it’s even impossible to get rid of this framework because it’s so integrated with your code. And if somebody will split that in the screen architecture way, it will be maybe possible to replace that and maybe just not use it at all. And it will be possible to develop this, product further. But often, unfortunately, it’s something like that. And it’s super hard to develop the product further because it’s so limited. But it’s probably worth mentioning that often those projects were successful because at the beginning they’re they’re managed to have free fast velocity and win the market.

Miłosz [01:02:22]: Yeah. That’s that’s the benefit of using them. But if you can introduce even some, you know, thin domain layer or something that’s separate from from the framework, that might be helpful if you want to move on. Not to use everything it gives you. But, of course, then then it’s kind of, you know, defeats the purpose of of using the framework in the first place.

Robert [01:02:52]: Right. So we talked about, what are the common issues of clean architecture, what are the contraversions, but I think there are also some upsides that we’ve seen multiple projects and some reasons why this maybe was invented because it’s like with many techniques like it wasn’t invented for no reason.

Miłosz [01:03:14]: Yeah, exactly. So the why behind it is important. So maybe let’s start like we did, yeah, with the burning it down with separation of concerns. And, you know, it’s the reason we we want to separate those concerns is this this logic independent of implementation details. Right? And, basically, the what it gives you is you’re you’re able to easily replace one implementation with another, which we talked about about mocking. That’s one use case. Super useful. But also you can replace it with something completely different like changing the database.

Robert [01:04:16]: It’s quite interesting because you can do that, but how often are you doing that?

Miłosz [01:04:19]: Yeah, that’s a good question. It’s often a counterargument I hear about clean architecture or interfaces in general. Like, yeah, yeah, you talk about how it’s easy to change from one database to another, but in reality, it never happens. But it happened to us recently, so that’s one example when it was useful. So we switched from Firestore to Postgres, and using Zaprouted was quite easy. Another nice example we did, in the past was starting with in memory implementation. So creating a domain domain code with some very simple, in memory implementation of the database. And then later when we know this is what we want we can just replace it with SQL or whatever implementation. It’s also quite nice.

Robert [01:05:22]: Yes, but I think it’s also worth mentioning that this is not the end of the architecture. So the end goal so it’s designed. So if you can replace your database or port easily. It means that, okay, you did a good job with implementing architecture, but it’s only showing you that it’s done properly. But it’s not the end goal in general.

Miłosz [01:05:51]: Yeah. In reality, you’re not going to be changing the database every few weeks. So it’s it’s not why you do it. But I think but it gives you this nice separation of, you know, the implementation details from from the interesting code in your in your project.

Robert [01:06:11]: Yeah. And why it matters? So it’s mostly because of, cognitive overload. So as I mentioned a bit earlier, so when you have multiple microservices that were too granular, you are joining it into one service. If you would be all in your handlers probably it will be very complicated. But if you’re splitting it in multiple layers, it’s allowing you to so as long as you know how the clean architecture works, it’s it’s making you a bit much easier to reason about the code base because you see what are the entry points of your applications, what use cases are supported, and how you’re interacting with the database. If it’s everything here in your HTTP controllers, for example, it’s super hard to understand that because you can have some query that is called in some weird way, but it’s hard to backtrack how it’s called.

Miłosz [01:07:03]: Yeah. And and usually it spills over somehow to your to your logic.

Robert [01:07:07]: And it’s nice if you have some transaction, but often it’s not.

Miłosz [01:07:15]: And if you don’t have the separation of any kind, it’s also very easy to lock yourself in with some framework or library because, you know, you just scatter the library method calls all over your domain code. And if you want to migrate later you have all those places you need to change which is much more complex.

Robert [01:07:39]: And there is one interesting comment about abstracting the database layer, that it makes sense when you need to support different databases. So you need to choose what to deploy. And it’s a very good point. And once in our life, we actually did something one level higher. Long story short, we worked, with the product that needed to be deployed in two ways, as microservices and as monolith. And it wasn’t some it wasn’t some crazy architectural idea. It was a pragmatic thing. So it would have, two deployments. So one in physical, servers and physical device and one in the cloud with the kind of simulated device. And because of that, it needed to be deployed in this way. And architecture was quite cool because of that because it was a matter of having two set of ports and adapters. One for the cloud deployment and one for the more bare metal deployment. And it worked Mhmm. Pretty nicely. And it thanks to that, it was manageable to do this.

Miłosz [01:08:39]: Yeah. That’s quite quite a specific approach, but that’s a good fit for for clean architecture in general.

Robert [01:08:46]: Don’t do that. You probably don’t need to do that, but, yeah, sometimes it’s useful. Alright, so the first thing was the cognitive overload. But the second thing is that help is much easier with clean architecture is testing. And like cognitive overload which is important for a more complex project with bigger team, etcetera etcetera. The same goes for testing because when your, project service or whatever is small, probably doesn’t matter what you test. I mean, you can probably just test it end to end and you can do that. But if you are working in the application that have thousand of use thousands of use cases, a lot of complex interaction with your database, testing it end to end is impossible because probably most of the use cases you cannot trigger with the public interfaces and it will be super super slow. It’s especially, true for applications where the domain layer is very complicated because you can have even thousands of use cases and test cases and running it with the database, it will be just super slow. So mocking the extra level is the must because in other, I mean, obviously, you can wait a week for running test suite and I know that such application exists but probably it’s not something that you would like to do.

Miłosz [01:10:09]: Yeah, and often having parallel tests in the same database is not often trivial to do. You can have conflicts between the use cases and so on.

Robert [01:10:19]: At some point it will be slow. I mean, obviously you can scale it horizontally but Yeah.

Miłosz [01:10:24]: There’s often this argument that you can you have Docker, right, or test containers. You can just spin it and run all on this infrastructure. And it’s true. That tests are so critical, you know, path of local development and CI that you probably want them the fastest you can.

Robert [01:10:45]: Exactly. So for example, if you’re changing some important functionality, you don’t want to then wait ten minutes every time when you change and change that. You would like to wait at most one second and see. Okay. It works. Oh, no. And thousands of unit tests are running and checking that. But it’s the thing that it’s again important for bigger projects, not all projects. If your project can just have end to end tests, it’s fine. I mean, we worked with some projects. The main project that we’re working on now in Tridos Labs, the Academy, is at the at the beginning we just have end to end tests for a pretty long time. It was just enough. So

Miłosz [01:11:25]: Yeah. I mean, if we were if we had unit tests, it would be even better. But sometimes you have to optimize for delivery speed. Exactly. So that’s what we did. Yeah. Speaking about tests, maybe we can just quickly talk about how we speed them. If you have multiple layers, you you probably don’t want the same testing approach for for all of them. So, for example, as we mentioned, for the domain layer, unit tests are are great fit because it is purely logic. There is no usually little dependencies that you can mock out, and you can have many many test cases that run fast using just viewing tests. That’s that’s pretty nice. And on the other hand, the adapters usually are a good fit for integration tests. So this is where we run some Docker database, for example, and just test the repository or the client or whatever it is in in separation from the domain. Right. And for the entry points and the application layer, we usually have little tests here, if any, because if all it does is translates one layer to the other and orchestrates things, it’s probably not no big benefit in testing it.

Robert [01:12:52]: And I think it’s actually one common issue that we’ve seen multiple times and I think we forgot about mentioning that. But, yeah, if you have domain layer and application layer, running a lot of unique tests for your application layer probably doesn’t make sense, especially if you’re mocking everything around because it’s

Miłosz [01:13:10]: Yeah. There are all kinds of tests. Especially if you have those mocking libraries that let you assert whether a function has been called, then your your test becomes become basically an application of your code. And every time you change anything, you have to go to a test and adjust it and basically gives you zero value.

Robert [01:13:31]: Yeah. I think it’s actually great symptom, let’s say, of why you have the bad tests. So when you are copying your logic of your logic in application from application error, for example, literally to the test. It’s it means that it’s something wrong with your tests. Yeah. And especially if everything around is mocked. But I think there is one nice, type of tests that are solving this problem nicely.

Miłosz [01:14:01]: Which is?

Robert [01:14:02]: Components tests that I think at least from my experience not many people knows of but long story short, idea of component tests is that you’re mocking everything from external word and usually you’re using so you’re running your application in for example, Docker, you’re mocking all external words or external APIs, everything. You have just your database in the Docker and maybe and everything else is mocked. And you’re calling public APIs of your application and that’s it actually. So, it’s different from end to end tests because you’re not integrating entire application. You are just having actually stops for the external words and it’s nice balance because you don’t need to, have super big environment to integrate everything with the downside that obviously you may mock some wrongs in the wrong way. But this is this standard problem with mocks but it doesn’t mean that you shouldn’t have end to end tests that are integrating everything. But you can cover much more cases than in end to end tests because you can define behavior of, the external APIs plus you’re also checking the integration of everything. So it’s covering a bit.

Miłosz [01:15:18]: Yeah. Probably the biggest benefit, Maybe the benefit, but the characteristic is that you don’t call other services in inside those tests. So you have to somehow stop them or use numerators or something like that. Because you you want to test this one service end to end basically basically not the entire infrastructure around it. So it’s also quite fast thanks to this.

Robert [01:15:46]: Yes. And it’s much easier to also run local because if you are running end to end tests with entire infrastructure, it’s not possible to run it on your laptop. And we’re big fans of being able to run as much as you can locally and test against that because it’s much easier to and faster to iterate over that.

Miłosz [01:16:05]: But no one enjoys, like, another team service breaking the test URL locally. It’s the worst experience. It’s not productive. So if you want to learn more about, testing this architecture, we have also a blog post about it, and we will link it in the description. Maybe another, like, like, less technical benefit I like about clean arch is, how it changes your mental model over time. So it gives you this kind of consistency in the project, so it’s easy to find what you look for. For example, if you if you want to add a new API entry point, API method, or message handler, you know, to which directory to, you know, to go and which files to edit.

Robert [01:17:10]: And this is actually the often the counterpoint of many people that’s, yeah, I’m looking on this clean architecture project and I have no idea where everything is. Come on, spend fifteen minutes to understand where it is.

Miłosz [01:17:23]: Yeah, if the project is big enough and you work with multiple people on it, and this investment in learning, you know, the structure probably pays off, especially if it’s consistent with how you do other projects in the team. And I think it also gives you this if this mental model spreads among everyone in the team, it gives you this nice vocabulary to to use. So, you know, when you discuss how to implement on some feature and someone says, okay. We need SQL adapter. Then everyone knows what is this, and it’s quite clear how to implement it. So it’s it’s very easy to, to discuss the the project.

Robert [01:18:14]: And then what’s useful in this because I know that it may take some time what layer you can import to what layer and works forbidden. The good, the good news is that there are leaders that can do that usually. So, for Go, there is go clean arch linter I written a couple years ago and it seems that it’s pretty useful for many people. I guess it have probably around thousand stars in GitHub so probably it’s useful. Often you have also linters I think in Java like, ArchLint or something like that that you can define some, I think it’s in Java and PHP. So you can define what are the modules interaction. So, yeah, it’s recommended to check for that, and it’s pretty useful to because you can ensure that the dependency between, layers is properly handled.

Miłosz [01:19:10]: Yeah. In in Go, it’s especially useful to have this clear direction because you you can have import cycles. Your code won’t compile otherwise. So if you have some, you know, some clear way to to import between packages, it becomes just easier to stick to one way and don’t worry about it again. So that’s that’s quite nice. Maybe maybe the next one would be what I also like about cleanarch is how it becomes easy to extend the application over time. And it’s a it’s a bit similar to this idea of having, you know, great architecture from the start. But I think it’s a bit different because you don’t really need to have all the use cases. But thanks to this clear structure, for example, it’s easy to to add new SQL implementation or or or, for example, to to change one database query with another. That’s that’s faster. And the impact is very low because I mean, it’s the the plus radius is small because you you don’t touch multiple files and areas. You just change this one little part and it works. So you don’t need to think of all use cases in the beginning, you just adapt over time with new adapters, with more application use cases, And this way you avoid side effects. In contrast, if you have this huge HTTP handler with everything in it and you make some changes to it, you don’t really know what can happen. That’s, you know, the worst scenario you can have when you work with a project you don’t know well, and you see this huge method of thousands of lines, let’s say, and you make some edits but, yeah, who knows what happens next. We don’t really know this. It can affect many, many areas.

Robert [01:21:25]: And, yeah, the last upside that it’s coming into my mind is the one that I think many people doesn’t think about, but we used it pretty heavy in previous teams. So it’s how how it helps you to split work.

Miłosz [01:21:41]: Oh, yeah.

Robert [01:21:42]: Basically, the idea is that you can plan some work so you have some feature to implement. You know that you need to implement it pretty quickly as usual. It will be great to parallelize your work as much as possible. And how we’re usually approaching that was, agreeing on the interfaces between layers and just splitting the work. So, for example, we had some feature to implement. So we agreed that, okay, for example, we need such interface to integrate interact with database. This is more or less what we need to have in the domain layer and we could split this work for four people, for example, and each person was responsible of implementing one layer and it was quite simple after they agreed on one interface. And thanks to that, implementing one feature instead of taking, I don’t know, three weeks, it was possible to finish it within one week including deploying that. And it was most important for features that we needed to just implement quickly. And this created some nice separation for each person and they were sure that we’re not, not conflicting that much. And, obviously, sometimes there were some changes into the interfaces because, well, you will not be always sure always right from the beginning. But assume that you are talking to each other.

Miłosz [01:22:58]: Yeah. But I can just it gives you this nice vocabulary to use. So, for example, when you when you split, the feature into smaller tasks, someone can say, okay, I will take care of the HTTP layer. And it’s super easy for them to, you know, create this API definition that will just do nothing in the beginning but will be merged, can even be deployed, and everyone else can integrate with it.

Robert [01:23:24]: And you can also split by complexity often. So, for example, you have some junior person, some principal person, and some mid developer. And more or less you know how much complexity will be in each layer. So usually, for example, your HTTP layer shouldn’t be that complex usually because sometimes you have. Sometimes your domain logic will be very complex. Sometimes it will be quite simple. But for example, adapters will be complex or vice versa. And it’s also very helpful to kind of assign work properly.

Miłosz [01:23:56]: Yeah. And it helps helps you create smaller tasks and PRs which is also a great thing. If you have, you know, a one file change in SQL adapter, I can review it in two minutes probably or something like that versus this whole feature where you need to focus on everything else. I think it’s quite unique overall, this approach. Mhmm. Because I think in most teams, you would expect to see something like, Okay, Robert, you’re supposed to work on this feature now. So see you in two weeks and you will show us what you have. And then you will come up with a PR that’s, like, I don’t know, 4,000 lines long. And then everyone spends reviewing it two days.

Robert [01:24:47]: And you have 100 comments like why you did like that or I don’t like this. Exactly. I think it’s even a good situation because often you will also struggle to find somebody who will do your reviews because this is the problem often when you are not working as a team but a set of individual stuff. Yeah. Your, goal is to your goal is to implement feature. It’s not a team goal. And if you are working at it as a team of four, for example, or five, you have always people to review that and work is just going much, much, much faster. Yeah. But important disclaimer again, it’s for, as we mentioned earlier, it’s for more complex projects with bigger team. Again, for a full team, it’s not a big team but it’s not two or one person team, obviously. And we assume that it the features are complex enough to take one week of four people. So Yeah. Also important to keep in mind because, you know, sometimes there are features that one person can implement within one week. And using four people to implement that with green architecture, it will be just over engineering probably, and it will be just waste of time.

Miłosz [01:25:55]: Yeah. But it’s good to keep in mind it’s possible to to speed work, actually. I think it’s not the the default for sure in the industry and somehow we accept that, you know, it’s normal to just have someone sit on a picture for two weeks and then review what they did. Or other two weeks? Yeah. It’s really not that productive. And sometimes if you have the layers especially, you can have three people or four or two work on the same thing and you know, do it in parallel on different areas. It’s super effective because you have those small small PRs, smaller, bus radios of of the changes. You can deploy early often and all those good things.

Robert [01:26:42]: Yeah. And I’m not sure if you remember but I remember this feeling like just work flowing.

Miłosz [01:26:48]: Yeah.

Robert [01:26:50]: Yeah. It was just pleasure because it was super effective and even very complex features were able to, you know, deploy it within one week. So it was cool.

Miłosz [01:27:01]: Another benefit that’s kind of not relevant to Clean Arch, but you have those small subtasks in each feature and you can see how it moves on the board every day, you don’t really need stuff like estimations maybe because, you know, whoever is responsible for the product or any stakeholder can just take a look at the board and see, okay, five out of eight subtasks are done in two days. So probably in two days they will be done with this. Okay.

Robert [01:27:33]: It requires the team to have a pretty big, let’s say, engineering culture to achieve that.

Miłosz [01:27:39]: Yeah.

Robert [01:27:41]: It’s very highly recommended to at some day work in such a team.

Miłosz [01:27:46]: Yeah. I think that goes a bit beyond clean arch. But I think clean arch is what really enables it. If you don’t have this clean separation in the code of different layers, then it’s very hard to split work because you will just keep getting, you know, in like, merge conflicts are the probably the least issue, but just conflict of who who works and what and how it impacts each other’s work.

Robert [01:28:19]: Yeah. Yeah. Probably it’s a topic for a separate episode how to do this splitting because it’s also not obvious how to agree on this, let’s say, one week work and everything like that. But definitely we’ll touch on it at some point.

Miłosz [01:28:34]: So we talked about the benefits, talk about talk about the issues. The the question that’s asked very often when someone asks about architecture is Can we just do it simple?

Robert [01:28:49]: And I think it was even mentioned on the chat.

Miłosz [01:28:51]: Yeah. Especially on on the chat, I think it was related to Go, which is a very good point because Go is seen as this simple language, you know, in contrast to Java, I guess, or C++ or whatever you want. And this is true. But does it mean that clean architecture has no place in Go? So, I think, you know, believing that if you keep things simple, it will save you is kind of lazy thinking, and it can work well if the problems you solve are simple. But if you start modeling something complex, it can it can be worse than having some more complex architecture in place.

Robert [01:29:47]: Yeah, and I think we’re lucky enough to work in different kind of projects and I think it was nice to try different approaches. Try to do as simple as possible as over complicated as we can to try some techniques and, yeah, I think it’s nice to showing that this balance is not simple, but it’s not like you can always do simple way. It would be great always to go with the simple way. But, well, unfortunately, most of the applications are okay. Maybe not most, but many applications are not that easy. To make it even worse, I think probably we may end up in the situation when those simplest applications can be handled by AI because it’s not complex. I mean, it it can already handle some simple scenarios, but the deeper problems, well, it’s a bit hard to reason let’s say reason AI, but you know what I mean.

Miłosz [01:30:47]: And it’s hard to define what simple code is too. And, of course, everyone wants to create simple code. I I don’t think anyone will tell you that they don’t want to do it. But, for example, in Go, in the beginning especially, it was, I think, a seamless Flat structure of a project was considered simple. So probably you don’t want to create any packages at all or just a few and then use these files in them. Works well for a standard library. That’s a good idea if you write a standard library. But good luck managing this project with 10 people. When everyone creates files they want, I don’t know. How do you create some consistency then? That’s the hard part.

Robert [01:31:38]: How do you manage conflicts? So when you have more people working on one service and you have everything in one package, it’s a much bigger chance that you will conflict on something.

Miłosz [01:31:48]: Yeah. So it’s very, very hard to not be naive about simplicity. But if your team is two people, we are just alone, maybe three people, manageable. Or if you can just fit the entire code in your head.

Robert [01:32:05]: Or you have 10 times more microservices than people in your team. You laugh but probably you remember some projects where it was the case.

Miłosz [01:32:16]: I see the solution. You just need to have one file microservices and then it’s not an issue anymore.

Robert [01:32:23]: Very recommend checking but, yeah, long story short, unfortunately, it’s it can give you some impression that it’s simplifying things. But again, implementing one change in 10 services, it’s no way that you will be able to work efficiently.

Miłosz [01:32:42]: So basically, complex projects needs need proper modeling and can be approached with this primitive structure. But of course if you use some very simple domain then it makes sense to keep it super simple.

Robert [01:32:59]: Yeah, and I also noticed that many people think about clean architecture in terms of product structure like you have the structure, multiple directories, but I think it’s not the most important part about clean architecture. So it’s more about the mental model. So I would say that interior you can probably have clean architecture with one package because it’s more it’s harder to manage obviously because you probably will have more conflicts. It’s harder to ensure that you are not interacting between the layers. But, well, in theory, probably, you can do that. But I don’t see a really point in doing that. But, again, it’s not about the structure of the product project. It’s more about how you’re splitting, functionalities in your application. And I think it’s also pretty consistent with the Go philosophy where you should have one package with one responsibility. And in some way, it’s compatible because you can have HTTP package that is responsible for handling HTTP. You have domain layer that is responsible for the domain layer. You have adapters that are interacting with the outside world. So I would say that it’s very compatible with the approach of the language.

Miłosz [01:34:15]: eah. Yeah. If you put it like that, it’s clearly this. Maybe it’s the issue of being dogmatic but the other way around. So if you are too dogmatic about being simple, it can also hurt you.

Robert [01:34:33]: It’s quite ironic. I mean, both sides of the barricade are doing the same issue. So they’re dogmatic but just in the opposite direction. But they’re making exactly the same issue.

Miłosz [01:34:50]: So it’s probably good to remember the the basics and to understand why you do it in the first place. So don’t don’t do it because you want your code to be clean, but to, you know, better organize work in in a complex project that will grow over time will be there in a few months or years, and you have many people working on it. That’s probably good good idea to use in architecture or some flavor.

Robert [01:35:21]: Yeah. But if your team are two people, it’s easy to go in the wrong direction of the spectrum and over original things and make it just too complicated. So not see any benefits of clean architecture and just pay the price. That is usually not that big if you will not over engineer that because, again, it’s not adding that much overhead if you know what you are doing, but still worth worth keeping in mind.

Miłosz [01:35:52]: Similar to the, question of should I write high quality or low quality code that we discussed in the first episode. Right? So if you if you have two person team, not, you know, developers who work together for a long time, and they trust each other, know how they work, and so on, you probably don’t need that many roles in place. But if you have a fresh team that just started working together of 10 people, you probably want more more guidelines. So there is some consistency in the project, so they can just work better together. So pick also depending on the team and how how much mature it is. What about some alternatives to clean architecture?

Robert [01:36:45]: Well, so we already touched a bit on microservices. So as we talked earlier, clean architecture is useful with the handling complexity, making testing things easier. So it sounds like what microservices are promising you. But please keep in mind one thing that we touched multiple times because this is very often the issue that we have seen that people just went too far with microservices and you have more microservices than, people in your team much more.

Miłosz [01:37:19]: And it’s usually Pretty common.

Robert [01:37:21]: Yeah. Yeah. And usually, it’s ending up in the situation when implementing one functionality takes changes in 10 microservices. So I’m not saying that microservices are bad, but they are bad if you are splitting them too granularity. And I think it’s one of the sign that it’s done with how many services you need to touch to implement our functionality or, for example, how many services you need to test one entire functionality. So, for example, if testing one use case that is handled requires 10 microservices, it probably may also mean that, there are two granular.

Miłosz [01:37:59]: Mhmm. Yeah. And there’s also the vertical slices architecture, but we didn’t use it as much yet. We probably have nothing smart to say about it. Yeah. So maybe next time, the next episodes.

Robert [01:38:15]: Or maybe somebody has some experience with that, so you can write in the chat.

Miłosz [01:38:26]: Okay, since we work with Go most of the time maybe we can just give some hints about Go specifically using clean arch. I think my favorite part is the implicit interfaces in Go. If you don’t use Go, in contrast to languages like Java, for example, you don’t need to specify that your your structure implements an interface. You just need to define the methods with the same signature as the interface, has. And while some approaches like ports and adapters, such as these separate ports layers for interfaces because you have to keep them somewhere to implement them, in Go you can implement them just close to where they are used, which is, I think, super convenient and also in line with the idea. So you can just define an interface next to the constructor with the dependency it needs. And basically, it’s the only place that knows about the interface in your entire project.

Robert [01:39:42]: And I think it’s also nice that you can make it work with interface segregation principles. So you can just use some subset of the interface that you’re implementing in the consumer, let’s say. So you don’t need to always use entire interface everywhere. You can just use some subsets of the metal. And it’s useful when you are doing some refactoring and, for example, changing some external adapter and you it’s nicely visible where it’s needed.

Miłosz [01:40:12]: I think someone mentioned it in the in the chat before. But this anti-pattern of creating, interface with exactly the same methods as the implementation. So you often see something like, you know, interface and then an implementation called the name of interface implementation, something like that. It makes no sense. And on the other hand, if you have the smaller interfaces, many things become easier like testing because your mock just becomes one method instead of 10. That’s a super nice feature of Go that plays well with clean arch.

Robert [01:40:51]: The second problem that is pretty high a big problem is the fact that many sources are a missing point of clean architecture. It probably doesn’t help that it’s not defined that strictly, let’s say. So it’s adding a big, big, interpretation area. Let’s see. So, yeah, look for good sources. So if you’re writing in Go or you would like to write in Go, we recommend checking our blog obviously because we covered it already a couple years ago and it’s still totally relevant. I think even if you’re writing in a different language, it’s the many ideas are common so it might be worth checking, what we’re writing there. But remember that different languages have a bit different, let’s say, idioms and some things are done a bit differently. So we know it from Go because many people try to port, clean architecture or other techniques directly to go from Java, for example, and it was terrible idea because of what you mentioned a bit earlier. Not, for example, in Go, we have duck typing. And because of that, it’s implemented a bit differently. And it’s worth keeping it in mind.

Miłosz [01:42:11]: Yeah. And probably the biggest anti pattern you can see in the examples are the single single model used for everything. So So you can have a structure that’s used for, you know, the domain logic, but also it it has database and JSON tags on them, which probably means, you know, it’s also used for storage and for API responses.

Robert [01:42:35]: And it probably means that we’re getting into this problem that implementing an architecture in such a way will lead to over engineering. Because if you are able to use the same model everywhere, it means that probably you don’t need such architecture and you could just use it everywhere and you don’t need to have all this strange mapping, etcetera. You can just simplify it a lot. But if so if you don’t see benefit so you’ll not see benefit from using this approach. But if you have more complex models and your domain model is not exactly what you’re returning from the API that works in multiple domains like, for example, financial, domain. The benefit of having different models in different places is is there.

Miłosz [01:43:29]: I think it’s time to summarize and Yeah. Go to the Q&A. So remember about the basics. Right? You want to separate domain logic, whatever it means for your app, from implementation details. That’s the that’s the primary idea of of clean arch or and similar patterns. And try to focus on the on this idea, on the why behind it, not on the how or or directory structure for for starters.

Robert [01:44:10]: Yeah. And there are a couple things that the architecture should give you and if those things are not met, probably it may mean that it’s not maybe you don’t need to use Clear Architecture like managing complexity with the bigger project, easier to test and write faster tests, you can parallelize your development, you can insulate your changes, and improve navigation. If you don’t have those problems, maybe you don’t need clean architecture, maybe your application is not that complex. Maybe, you know, just having everything in your HTTP controller it’s fine. It’s totally fine. We’re doing it in some cases when because even within one application you can have Twin architecture in some places and doesn’t have it in other places and it’s totally, totally fine. And No use approach. Will save you a lot of time.

Miłosz [01:45:07]: Yeah. On the other hand, there are some pitfalls. So it might be confusing at first for people who don’t know it yet. So take time to explain it to your colleagues so that you are on the same page. And watch out for common issues like using too many interfaces or too many layers or just over engineering the the basic idea, and watch out for using it everywhere, as Robert just said. So if you if you use modular monolith, I hope you do, not all not all modules need to follow the same architectural patterns. You can use something different in different modules. It’s fine.

Robert [01:45:52]: And, yeah, to analyze your benefits and pitfalls of your application, it’s worth putting to this equation the team size, your experience, and, well, watching out to not be too dogmatic when you’re doing that.

Miłosz [01:46:09]: A good tip for whatever you do. So we’ll link some materials in the description. Check out our blog post on clean architecture, on testing strategies as well. And there’s one about integrating clean architecture with DDD and CQRS. In Go. And also the one about the repository pattern is probably quite relevant to clean architecture in general. And all those posts are just a series that describe the white workers repository, the example project. So you can check it out too.

Robert [01:46:51]: Let’s take a look on comments. So we already covered some of them but let’s check what we have missed. Where are comments? Oh here, okay, so let’s start from the beginning. Yeah, so again a couple comments that people are missing working with me, Osh. Good sign. So usually we’re using clean architecture so, you

Miłosz [01:47:28]: Know. Yeah. I think it’s the only pattern that I’ve heard of people using beyond, the teams we worked on. So, you know, we met one year later and they told us, oh, you know, I spread it in my team and now we also follow Clean Architecture’.

Robert [01:47:47]: Or I missed the project that we worked together. The current one is so.

Miłosz [01:47:53]: But I mean, the the architecture is a pattern I’ve heard mentioned multiple times, but it’s probably also the the only one mentioned so often. So that must be a sign. Right? Looking at the bottom, there’s a question about how clean architecture differs from hexagonal. And I would boil it down to naming with a different approach. I would say we use a combination of both. So it’s not important. I would say it’s implementation details.

Robert [01:48:36]: But, yeah, I think it’s all about you have some inner layers and outside layers and inner layers doesn’t know how outside layers are implemented.

Miłosz [01:48:47]: Yeah. And all those variants call them a bit differently but the core core idea is the same.

Robert [01:48:56]: Yeah. And, you know, my theory about this is that, you know, the multiple techniques are reinvented every 15 years because, you know, there is a hype cycle of every technique that at the beginning it’s, hype time. Oh, yeah. It’s solving all of our problems. So everybody is going into that without understanding how to do it. Multiple teams are implementing that. Some people in very dogmatic way. So people are starting to hate this technique and the word is spreading that, this is a bad technique. Don’t use it ever. It’s super bad. It’s overcomplicating stuff, etcetera. And less and less people are using that. And after some time, somebody is thinking, oh, okay. It’s so nice idea but let’s it’s the name is already so spoiled that we need and we need to name it a bit differently and so people will be ederabad about that. And, you know, it sounds absurd, but it works.

Miłosz [01:49:58]: So are are you suggesting we need a new name? Yes. And your standards too?

Robert [01:50:04]: Yeah. So if you have any idea how to name next, iteration of clean architecture, listed on the chat.

Miłosz [01:50:16]: There’s a question about tests. We kind of mentioned this before. So do you ever add tests on the application layer and how do you go about it? Mocks or fakes in memory by implementing the interfaces? So it depends how many logic do we have there. If we have a separate domain layer, we usually have almost no test in the application layer because it becomes mostly orchestration of things. Yeah.

Robert [01:50:45]: And it should be covered by the component tests, basically. So you are calling the component tests and it’s going over all layers and checking that everything is integrated with that problem.

Miłosz [01:50:55]: But sometimes it makes sense if there’s some some logic you can test.

Robert [01:50:59]: Especially when you don’t have domain layer, for example, because sometimes you may not have enough logic to have a separate domain layer. So you may have more logic in application layer. But in this case, what will for example, what you can do can you can have functions that you can just, test independently without entire words.

Miłosz [01:51:19]: Yeah. And in this case, in memory fakes work well. And this this is where having small interfaces is super important.

Robert [01:51:27]: Yeah, and what’s important there is ensuring that you are mocking this IO stuff that I mentioned a bit earlier. So don’t mock your, I don’t know, logic, let’s say, because it your test will be just mocking some subset of logic and they will not give you a lot. So, you know, mock your database, mock your mock your external dependencies, don’t mock your logic.

Miłosz [01:51:52]: And we are big fans of writing the fakes, manually. I know many people use mocking libraries but we find them usually not that helpful.

Robert [01:52:04]: At least in Go. I think it may be, let’s say, problem of Go because of its compiled language and often those libraries are using a lot of reflection. For example, I remember that in PHP there were some nice libraries for testing but in PHP it’s a bit easier due to not being compiled language but, well, upsides and downsides.

Miłosz [01:52:29]: And sometimes it also creates, like, brittle tests. If you you start testing the function calls, for example, then you often need to change them if you change anything in the code.

Robert [01:52:46]: And, yeah, the problem with mocking libraries so it’s another thing in the bucket of things opening the gate of hell so it’s making easier to mock too much. In other words, it should be painful to mock things because you can mock too much.

Miłosz [01:53:03]: Yeah. That’s what Todd in the comments mentioned before that if you need a mocking library, you are mocking too much.

Robert [01:53:09]: Exactly.

Miłosz [01:53:10]: Yeah. That’s kind of the idea. I think that Yeah. And Firmino had some experiences with architect and his Gomock lib. I don’t remember which one gomock is. Is it the one using reflection or the one using generation?

Robert [01:53:29]: I guess so. I already forgot it intentionally.

Miłosz [01:53:36]: Yeah. I just found another one from Todd. The first step I use when trying to introduce layers to people who don’t use them is to explain that the application shouldn’t be tightly coupled to the API. So, yeah, it’s a great approach, and also shouldn’t be coupled to the adapters, the database, for example. I guess the hard part of explaining it is explaining someone why it is good to decouple them.

Robert [01:54:05]: And if you don’t see any reason again maybe you don’t need the architecture because if you have a CRUD application and you are mapping your CRUD structure before your at least at least four layers, in worst case six layers or how many, it’s just a waste of time.

Miłosz [01:54:28]: Yeah. And sometimes it’s also not that trivial to explain the benefits because you can use the typical reasons like if you ever want to change the database it will be easy. But it’s easy to counter this with. We won’t. Which is also true.

Robert [01:54:54]: Don’t use this argument.

Miłosz [01:54:57]: Yeah, I mean, it’s it’s hard to have a discussion that that it doesn’t start to be about this dogmatic approach. Our code needs to be clean. Our code needs to be simple. So watch out for that. Yeah, but this is definitely a nice idea.

Robert [01:55:17]: It’s also probably worth mentioning that, you know, you won’t be able to always work in the teams following this approach, which is fine, but also don’t be a rebel that is just using clear architecture in your small project. I mean, if you’re working the team that, okay, you’re not following this, don’t be the rebel. I mean, sometimes disagree and commit, it’s sad, but well, maybe it’s not sad. It’s something that you need to use in you should work in a team, not alone.

Miłosz [01:55:48]: It, I guess, depends on how well the project is doing. You know, if I know it’s hard to, you know, disagree and commit if you see all the issues that following some pattern could solve. That’s that’s the hard part, you know.

Robert [01:56:05]: But you will be not always able to just convince everybody. It’s different.

Miłosz [01:56:10]: Yeah. Just you need to know why you you want to use some pattern. Right? We discussed this in the first episode that we had this one project where everyone was super happy to work with because it it used, many cool patterns, and we kept talking about it years after it. And the project was shut down because it had no paying users.

Robert [01:56:40]: Well, at least it was great.

Miłosz [01:56:42]: So that’s the hard part. You need to be productive and avoid getting into this big ball of mud issue. But sometimes it doesn’t matter that much as well.

Robert [01:57:00]: Yeah. So at the beginning, we have a couple of comments that somebody is missing working with you. We have also one comment that please come to my team and make team follow architecture and solid. I use elixir, and I don’t know if it’s the language or the community, but, oh my, everything is coupled.

Miłosz [01:57:21]: I don’t think it’s the language. I think it’s just it is what it is.

Robert [01:57:27]: I think it’s also something connected. I mean, the mindset and language. I will not mention language names started with r or

Miłosz [01:57:38]: But Yeah. But, you know, even if in Go, which is supposed to be a simple language and, I don’t know, somehow make it magically better or your applications easier to develop, We know the reality is it’s just as easy to create something coupled in Go than in other languages. Maybe with less magic. That’s the good part.

Robert [01:58:01]: That’s true.

Miłosz [01:58:02]: But, for example, you know, the struct tags in Go make it super easy to couple layers and it’s not that easy to see if you don’t know what you’re doing.

Robert [01:58:12]: But I think it’s something in that, you know, a lot of Go people are very open minded because it’s, you know, fresher language and in general, you you know, if somebody was eager to learn some new language and try something different, it’s at least from my experience, it many cases, it was real open minded person and it was somewhere there. But, yeah, about about the team, you know, if you are not able to change the team, change the team. It’s sometimes like it and it’s just sometimes not possible to change what you’re doing in the team. So, you know, it might be worth, you know, leveling you up in your skill set and trying to find a better place. Unfortunately, there are a lot of great places and hard bad places to work in. It’s also a bit unfortunate that we know that it’s hard to join good teams that are, you know, following some good practices. But from other side, you know, also grass in your neighbor’s garden is always, more greener than yours. But yeah. It you can always find a better team. But for that, you often need to

Miłosz [01:59:20]: There’s also there’s a risk there that you start thinking that the team using clean architecture is better than yours. Yeah. Who knows? Maybe they over engineer it so much you would hate it even more.

Robert [01:59:34]: Yes. Yes. But yeah, about the quality again, I recommend checking the episode about how to write bad code qualities. We’re also touching a bit on how to set expectations because well, none of the team is perfect. There are always problems and but, yeah, there are better and worse teams.

Miłosz [01:59:57]: Okay. Are we out of questions?

Robert [02:00:01]: I think so.

Miłosz [02:00:04]: I think we are just on the two hour mark.

Robert [02:00:08]: Perfect.

Miłosz [02:00:14]: Cool. So thank you everyone for joining us. If you want to not miss the next episode, you can subscribe to our newsletter on our blog, and we will let you know. You can, of course, subscribe to YouTube, our YouTube channel, but we’re not sure if they will notify you, so our newsletter is the best one, the best place to get notified. And you will also not miss posts, our notifications of our our our posts, so that’s another reason. Yeah. Leave us any comments. Tell us if you liked it and what you can improve. If you are listening to this after the live happened, you can just comment under the video. And if you listen to this on any podcast application, give us five star rating.

Robert [02:01:17]: Yeah. Unfortunately, we’re ruled by the algorithm so your, thumb up or flight style review can help us a lot and I think this is the best way how you can support us and well, help us to record more episodes.

Miłosz [02:01:37]: Which, will happen in two weeks.

Robert [02:01:39]: Yeah. About what we are talking in two weeks? I don’t remember.

Miłosz [02:01:42]: I think how to learn quickly.

Robert [02:01:45]: So I think it will be another episode that is nicely connected to what we were talking earlier. And, yeah, if, Nicholas, you were mentioning that you would like to join your team, We are not looking for a job, but what can help is, well, learn more and being able to join better teams. So this is the tactic that we definitely recommend.

Miłosz [02:02:09]: Or spread some ideas within your your team.

Robert [02:02:13]: Yep. That’s true. So, yeah, the next episode is about that. Hope you see.

Miłosz [02:02:19]: Thank you everyone for joining us today. Thank you, Robert.

Robert [02:02:22]: Thank you, Miłosz. And see you in two weeks. Bye bye.

Miłosz [02:02:25]: Bye bye.

When you shouldn’t use frameworks in Go

Wed, 19 Mar 2025 17:00:00 +0000

Quick takeaways

Frameworks promise productivity but often lead to issues as projects get larger and more complex.
The Go community prefers small, focused libraries over frameworks due to Go’s design philosophy influenced by Unix principles.
Watch out for risks using frameworks like vendor lock-in, deprecation, and costly migrations that can take months.
Explicit code is more maintainable than magic framework abstractions.
Choose your approach based on project size and maturity - frameworks might work for prototypes, while modular libraries are better for long-term projects.

Introduction

In this episode of the No Silver Bullet podcast, we discuss frameworks in Go and when they’re useful or problematic. We talk about why the Go community generally avoids frameworks compared to other languages, and how small, modular libraries are often preferred in Go development.

We share our experiences with frameworks across different projects, including tradeoffs between productivity and long-term maintenance.

Notes

Model-View-Controller (MVC): Pattern first described in the 1970s for Smalltalk, still widely used today.
Unix Philosophy: design concept created by Ken Thompson (also a Go creator) promoting small programs that do one thing well and work together.
When to avoid DRY in Go
Watermill: Our event-driven library for Go designed to not be a framework.
Repository Pattern: Our blog post that is still relevant and frequently referenced.
tdl and pq - the CLI tools we mentioned.
Clean Architecture: The topic of our next podcast episode, a design approach that helps maintain separation of concerns.
Wild Workouts : Our example Go codebase demonstrating clean architecture.
The Best Go framework: no framework?

Quotes

The happy path is easy enough, but the happy path is usually not the hard part of software. We often overvalue how much effort the boilerplate requires.

Miłosz

Framework knowledge tends to become out of date. You can spend days or weeks learning something about a framework, but it can be outdated. And if you switch to another programming language or company, a lot of effort that you spent to learn stuff will be just wasted.

Robert

It’s more important to learn even-driven architecture because you learn the theory behind it and how it works in general - it transfers better to whatever you will do later. Focus on timeless skills like how to split modules in your application, how to make it decoupled, how to write business logic so it’s easy to read and modify.

Miłosz

The Go language is heavily influenced by Unix philosophy - write programs that do one thing and do it well, write programs that work together. It’s visible in Go’s standard library. This is why Go promotes building independent components that you can connect together.

Robert

You need to be careful not to go too far with foundations. It’s better to start with some modular libraries, have some reasonable setup in place, but don’t go too crazy with it. Most of the time you’ll need to refactor the project anyway, whatever you do, because it can change drastically.

Miłosz

One big decision at the beginning may cost you six months of work later. Understanding if something is tightly coupled to your application is simple - just think about how easy it would be to remove it.

Robert

Timestamps

Transcript

Robert [00:00]: What if framework you choose today costs your team months extra work next year? Choosing a framework at the beginning of the project can greatly impact the project in the future. We have been in projects where relatively small decisions in the beginning led to many months of work a couple works later. Have you ever considered why the Go community does not favor using frameworks in go? Does it mean that using frameworks is always bad in Go? I’m Robert.

Miłosz [00:31]: And I’m Miłosz And this is No Silver Bullet podcast where we discuss mindful backend engineering. We spent almost 20 years working together in different projects and teams, and during that time we’ve seen that following advice like “always do X” or “never do Y” rarely works and can limit your growth. And in this show, we share multiple perspectives that will help you make smart choices and level up into the principal engineer level.

Robert [01:01]: If you have any follow up questions, leave them on the chat. So we will try to answer them at the end or maybe even during discussion if we will have enough time.

Miłosz [01:14]: So we talk about frameworks today in Go, but also in general, maybe let’s start with what we consider a framework. So we are all on the same level. First of all we focus on backend frameworks. So we don’t talk about the frontend world today. Just the backend side of things. And in this episode we specifically focus on the Model View Controller framework or MVC, which is usually what most people consider a framework during the discussion of “should I use a framework at all?”, or “which framework do I choose?” So maybe, Robert, could you give us a TL;DR what MVC is?

Robert [02:02]: So MVC. So it’s an interesting pattern that it’s already pretty old, but well, I’m kind of fan of some old patterns because I think often they’re still pretty relevant. And the same thing is with MVC. I think for the first time MVC was let’s say invented or written down in 1970s. So yeah, it’s more than 50 years ago. So pretty long time, but it’s still pretty valid.

Miłosz [02:29]: Like SQL.

Robert [02:31]: Yep, yep. And I think it was for the first time in Smalltalk language. That was also pretty interesting one. But we’ll not go today about talk about Smalltalk. But yeah, and MVC. So it’s acronym for Model View Controller. And the interesting thing is that the chance that you are using MVC is pretty high. If you are building an application that has a database and a user interface, and you are allowing to modify this data. So it’s pretty high chance that you are using MVC. And it’s because it’s the industry standard on how you are building applications now. And very often it’s actually MVC pattern. And the idea here in MVC is to split your model, view and controller. So model, this is your domain logic. Your entities that are handling data. View is we can say the user interface often and controller is responsible for orchestrating everything between this. I’m not sure if you agree Miłosz, but probably it may be obvious for some single page applications, but I guess we can also say that even if you are building some react application, it’s some form of MVC also, even if it’s a bit different than how it was built.

Miłosz [03:56]: On a different level, but a similar idea. With the one model that is the source of truth of data, and then some parts of the application that react to it in some way. So that’s similar concept I guess.

Robert [04:13]: And yeah, today we’ll discuss MVC frameworks. So if you think about frameworks like Ruby on Rails, Django, Laravel, Symfony, Spring, all of them are MVC and are MVC frameworks because they have building blocks for building MVC parts. So you have some building blocks for models. Usually you have some building block for controller and you have some building block for view. So the idea is that they should simplify your life when you are building such kind of application.

Miłosz [04:45]: And there are some other libraries that we probably shouldn’t consider MVC frameworks for this discussion. Like router libraries and in Go there is many of them, but you can also think of something like maybe flask in Python or I guess chi or echo in Go. So they give you the tools for working with HTTP messages and you get the views, but there is no model. Or you could do it yourself with some ORM framework maybe, but it’s yeah, it’s not out of the box compared to, for example, Django that gives you all of it.

Robert [05:31]: Django or anything what I mentioned earlier, like Ruby on Rails or Laravel, Symfony, etc. etc.

Miłosz [05:38]: From what you said so far, frameworks seem like a good idea, like the separation between model, view and controller seems like a separation of concerns. What we want most of the time and they do many things for us, they come with batteries included. So usually if something sounds so great, there must be some hidden cost, right?

Robert [06:09]: Yep. And frameworks are no different. So I went over websites of couple of frameworks to figure out actually what they are promising. And I have a list actually about things that are usually mentioned there like authentication and authorization, logging, observability, queues, database access or some sort of ORMs, dependency injection, sending emails, payment provider integration. And the idea is that it’s all integrated. So for example, if you are using any component, it should be integrated with observability with queues. And the idea is that it should work together out of the box. I also checked some promo materials for those frameworks and some sound pretty nice. Like a “complete ecosystem for web artisans” or “build and ship software with tools crafted for productivity”. Sounds good. This one I like pretty much “compress the complexity of modern web applications”, or the last one “optimized for happiness”. It sounds a bit like some religious cult, you can say, but to be fair, well, spoiler alert it’s not false. I mean, in some contexts it’s meeting those requirements moments. Maybe happiness sometimes.

Miłosz [07:34]: It sounds good on paper. I think especially if you just start, if you bootstrap a project, especially before we had AI tools to bootstrap projects with. It could be quite cool to just call a couple of functions and have a website ready.

Robert [07:57]: And it’s kind of “wow” effect. So you just spend five minutes on something and it’s working. You can show it to your mom, “Mom, look what I built, oh”.

Miłosz [08:08]: Especially if you are a beginner, right?

Robert [08:12]: Yeah. So you don’t need to have all this knowledge about how to connect stuff. And you can focus on things that matter at the end and are the most important. So building the useful things. And you are productive when you are doing that. So that’s cool.

Miłosz [08:29]: And what I also like about it is that you can focus on writing the core domain parts of your software because for many web apps, there are probably, I don’t know, 80% of the code that is written is very generic, right? It’s what all web applications use, like database handling or HTTP routing and stuff like that. So it makes a lot of sense to focus on what’s different in your software and not waste time on writing what someone solved already some time ago. But I think there’s also this trap here that I’ve seen many companies and projects have this dream of, you know, we will build all the foundations first. So we have this super custom framework or platform or whatever this means, and then we can just create new applications or microservices or whatever else, like a cookie cutter. So it virtually should take no effort in theory. And it sounds nice, but I think it’s also a trap because it ignores the fact that most of the complexity in software is in the details, in the weird edge cases, which is, by the way, also probably the reason why AI tools are not that great yet. I mean, they are very, very nice for the generic cases, maybe for similar stuff for what frameworks are nice. But when it comes to those weird edge cases you need to handle in the real world, yeah, that’s a bit harder.

Robert [10:18]: It’s actually quite interesting that we can say in some way that AI is competition for frameworks, because, yeah, that’s true.

Miłosz [10:26]: They all fall into this category of this dream of “code with no code” or “software, with no code” to create applications with no code like UML was supposed to be at some point maybe.

Robert [10:42]: But let’s go back for a second to Go, because the title of this live podcast is “When you shouldn’t use frameworks in Go”. So let’s go for a second for Go, because now we covered a bit how it looks like in the different technologies. But it’s worth mentioning that in Go it’s quite different situation. So like in different languages that we mentioned you have some kind of well-established frameworks, Go is different. So there is no leading framework. And generally the community is against using frameworks. And this is interesting situation and I think one of the reasons for this situation is that well, Go is pretty new technology compared to other technologies. And well, building one big framework, it requires much more effort. If you compare it to just building small libraries that do one thing and yeah, it’s just faster. So it may be probably one reason for that.

Miłosz [11:46]: And maybe also the, the libraries that we have now evolved over years. I remember like when we started eight years ago, there weren’t that many libraries for anything but they started appearing over time and they grew and became more polished. But there was nothing that connected them all. Well, there were some attempts for sure, but I think there is no one standard framework right now that everyone uses. Maybe it’s because also of how the standard library looks like and the Go philosophy of small packages that do one thing well. That kind of resonates if you think about the current ecosystem of open source libraries in Go. There are many smaller libraries that focus on doing just one thing.

Robert [12:44]: I think a good reason for that may be also the design of the language, so how the language was designed. So I think, I’m not sure if anybody mentioned it explicitly, but I think it’s visible that it’s heavily influenced by Unix philosophy. So I suspect that for at least one reason that one of the creators of Go language is Ken Thompson, who is also author of Unix Philosophy and Unix Philosophy tells us to write programs that do one thing and do it well. To write programs that work together. And it’s visible in Go standard library. If you look on how, for example, HTTP package is built, how io.Reader or io.Writer are built. So, you know, io.Reader or io.Writer is an excellent example of a small program, let’s say, or interface doing one thing and you don’t really care about how it’s working under the hood. It’s pretty similar. So if I would like to explain Unix philosophy for you, probably the best way will be mentioning how your terminal works. So you have things like cat, you have things like grep and you can connect it together and pipe that. And this is similar. This is actually this idea. And this is moved to Go heavily. And yeah, I think it may be also the reason why Go is kind of promoting the building independent components that you can connect together. And it’s nice because for example, if you have http package, you can have multiple libraries that can understand HTTP request or HTTP response writer and interact with that. So it’s promoting this, creating independent libraries. And for example you can have a library that is just giving you one HTTP middleware, and you can integrate it with the rest of the system.

Miłosz [14:38]: It’s a great API design rule overall, no matter the programming language really. And it’s nice that Go gives you those interfaces out of the box. So most libraries try to stick to them, which makes it easy to connect them together.

Robert [14:53]: And it’s not something that I’ve seen in different languages. So usually in different languages you have library that is doing HTTP, and standard library is not giving you that much. Or sometimes those frameworks or libraries are doing something totally incompatible. So it’s just hard to have multiple libraries that are working together. And in Go giving these interfaces, it makes this much easier.

Miłosz [15:15]: That’s actually a good point. I think Go is the first language I’ve used that gives you this really powerful HTTP server. That’s yeah, it wasn’t ideal from the start, now it’s a bit better. We usually use routing libraries anyway, but it’s quite solid if you want to start with anything. I think most other languages, at least at the time Go was released, weren’t that great. You had to just, you know, include some custom library. So why not use a framework if you have to do it anyway? And there were some attempts at creating Go frameworks, I think. And one risk of using them is that if there is no one big default framework like let’s say Spring in Java, you have this risk of this framework becoming deprecated, and then you are stuck with some project that uses a framework that’s hard to migrate out of. And I think there was go-kit. We’ve seen used some projects.

Robert [16:28]: No, it was go-micro.

Miłosz [16:29]: Oh, true. Yeah. And I think there was also a very popular Gorilla router library that also became deprecated over time. So you can still use this of course, but it doesn’t receive updates. So there is some risk that you might need to migrate out of it in the future. And you know, migrations are fun.

Robert [16:55]: And you know, if you are using frameworks, you have this framework that is integrating all MVC layers. So model, view and controller, it’s much harder to migrate out because if you just have a library that is handling routing for you, it’s kind of simple to get rid of this library. But if it’s integrating everything more, I think model is probably the worst part, because if your model is integrated with views and controllers, it’s very hard to get it out of your application. And yeah, and okay, somebody may say, okay, it’s not happening, that libraries or frameworks are deprecated. You can check history of them, that some of them are quite dead, we can say. And there is also one example of licensing change. So go-micro from what I checked actually changed license and is no longer open-source license. So it’s business source license. So the same license as Terraform is using and it’s not open source. So you can read the code, but you can cannot use it like open source thing.

Miłosz [18:02]: Time to fork.

Robert [18:04]: Yeah. But remember fork the tag before changing license. So the thing what happened with with Terraform. But it’s risky obviously. And in some cases it may just cost you a costly migration. And this is just one example.

Miłosz [18:20]: So someone could say it can happen with any library. Right. So why bother? But I think the difference is the smaller libraries are easier to to replace if it impacts just one area of your code. Using a different library here is quite easy, probably. Depends on how you do it, but hopefully. But if you do it as a core of your project, like a framework, then it becomes much, much harder to do it.

Robert [18:47]: Definitely. And if you go back for a while to discuss the promises of frameworks because, you know, we’ve been there also a couple of years ago when we were starting programming and, and yeah, at this time we’re working, to give you some context. So it was, I think almost 20 years ago. And at this point, the landscape of programming looked totally different. So, for example, if you know Laravel framework from PHP, it didn’t exist yet. jQuery, that already sounds like super old technology, it was almost not adopted yet. And if you are talking about other PHP frameworks. So it was Symfony, but it was also not widely adopted. It was something pretty new. And when I remember that we use Symfony for the first time, it was actually quite great at the beginning because it made a lot of things that we were doing earlier by hand, and it just allowed us to save a lot of time. So I already mentioned authorization, logging, nice observability. It just worked of the box, and it was great. It also generated for us database models. So again, we were able to deliver stuff faster. But again we were also a bit less experienced. So for us it was like magic at that point. And I remember that before that we were just crafting some SQL queries by hand and it was super dirty.

Miłosz [20:22]: But this magic was also a bit unexpected for me when I first started using it. So I remember you tried to onboard me to Symfony. And you know, I was looking through the project and I couldn’t find a SQL migration files and I asked you, where are the SQL files? You told me what? You don’t use it. You just you just use models in PHP. And that felt weird for me at the time because you had to change the mindset. You have to just trust that framework, that it will do its job well. You don’t really. I mean, you can dump the schema, of course, and inspect it, but usually just accept that there’s some annotation you have to do and that’s it. So if you would like to know how it works, the internals, that’s a bit of a mindset change you have to do. And also, on the other hand, if you have someone who is used to working with Spring, let’s say, we had developers who were Java developers and joined us to write in Go, and I remember they were confused about how we don’t have a dependency injection library, and they were just mind blown similarity of, you know, how why do we do it manually? Why not just put a single annotation and it works? So it’s a bit different concepts. You have to adjust.

Robert [22:07]: But I think it’s also one downside that, you know, it’s hiding this complexity from you. And sometimes those people were coming from Spring. They struggle because, okay, they need to write some query by hand. And it’s like writing a SQL query is so hard. I didn’t write any SQL query in my life earlier.

Miłosz [22:29]: That’s maybe why, you know, “no silver bullet” is so difficult because we tend to adopt to one way of doing things when we started.

Robert [22:42]: But I think it’s also showing that it’s, you know, often frameworks are showing you that, okay, you’re doing it in one way and people are not really thinking about doing some exceptions because for many cases, it’s just enough to have generated model that does queries for you and ORM is doing for you, but probably 10, 20% of queries can be written by hand and it can be much more optimal. But people are often going in the direction that, okay, we need to do it with ORM and are going with some crazy stuff and also to ensure that it’s properly shown somewhere with the view layer.

Miłosz [23:18]: Yeah, we like consistency. What can I tell you? I had the same experience as I just just mentioned. It’s nice to work in a way you already know. And if you have to change it’s problematic.

Robert [23:34]: But probably we can recap that as okay, frameworks obviously they have some downsides but well they are giving us a lot. So you may ask us okay why we are no longer PHP developers writing in Symfony. So it may be some reason that, well, it wasn’t the best technology in our life and now we are a Go developers instead. So it’s a question what went wrong. So I already mentioned that, you know, 20, 10 years ago the landscape was totally different. So it was much less open source tooling like for observability, logging, tracing and routing so you didn’t have much choice because you could use Symfony or some magic PHP frameworks, or Spring maybe, or maybe some Python tools. So actually frameworks were giving this a lot for free, but now it’s quite different because you have much more libraries, especially in Go that are open source. And they are using some open standards.

Miłosz [24:38]: Or even standard library like with HTTP in Go, you don’t need a separate library for this and many other similar for stuff like, you know, it wasn’t possible back in PHP. You know the standard library there was quite poor at the time.

Robert [24:55]: Oh yeah, especially compared to the Go one.

Miłosz [24:57]: So basically we learned that there is no free lunch. There is no silver bullet, you have to consider what you need to give up to use a framework. And I think it’s more relevant if you think about the project in the long term. Because if you just start the project, as we said, you know, we just throw in a couple of function calls and you have a working website that’s pretty cool. It feels like magic initially. But over time it becomes less cool because you start getting into those edge cases that the framework doesn’t handle for some reason. And maybe what you mentioned about SQL queries is a good example of this. So often it’s it just makes more sense to use a custom query to get some data you want, some complex query, instead of trying to use the models. I especially like how you know in Go the ORMs often have a separate language in the struct tags. So you have to learn a separate syntax that is not validated by the compiler. And I started to wonder, you know, why is it really that that helpful to use it.

Robert [26:34]: And you need to also do a PhD from how those magical notation works and it’s not great.

Miłosz [26:42]: Yeah, exactly. So at some point it just becomes less flexible. It’s easy to start. But then you need to add some extensions, maybe, you know, even in several components that use the framework and it’s starts to get a bit tricky.

Robert [27:01]: Model is one thing, the flexibility of models. And if you are talking about MVC frameworks because, as we said at the beginning. So in most cases when we are talking about frameworks, it’s MVC frameworks. So if it’s MVC, it needs to support not only model and controller, but also view. And from my experience, very often the biggest problem is in the view because okay, if you are starting to build your simple application, just having a form and a form with some fields there with simple inputs, it’s enough. But probably very soon you will find out that what frameworks are giving you out of the box, it’s not really allowing you to express, let’s say, every UX and requirement that you are receiving earlier. So yeah, it’s also obviously often you can do workarounds or for example, not use the view part from framework. Just return your models from Rest API or from other site, you are losing a lot of advantages that the frameworks is giving you.

Miłosz [28:10]: You need to find this balance. Because if you find yourself spending more time on extending this, those missing edge cases and writing those workarounds, it’s probably a good idea to do something else. Those queries for example. Right. So if you start fighting with those with this weird syntax of an ORM or a library, and maybe just give up and use a standard SQL query and should be good enough.

Robert [28:44]: And I think it’s especially a problem when your project is becoming more successful and team is bigger, because if you are two people, team whatever, and your product is not that successful or widely used, it probably doesn’t matter. Actually, you can take shortcuts and it will be everything fine. But well, if your product is becoming successful, all the problems are starting to pop up there.

Miłosz [29:08]: As usual, the happy path is usually easy enough, but the happy path is usually not the hard part of software. And we often overvalue how much effort the boilerplate requires. Like, it often it just seems boring to to write all this, you know, routing and database database queries and all this stuff that you do in every application. It’s not really different. It doesn’t differentiate you from other applications, but it’s not really a big effort. Right? It’s just boring. And now LLMs will happily do it for you. So it’s even better.

Robert [30:02]: But from one side, writing it by hand is boring. But I would not say that it’s super interesting to do workarounds for that. And I would say that it’s much, much worse than.

Miłosz [30:14]: Oh, but sometimes, sometimes it’s actually fun to do it. I have a story maybe I can share. So you remember in one project we had this entities stored in a database in, and we used one model for them in Go. And yeah, we had this issue that we didn’t want to return the exact values we stored in the database to the front end side. So we had one model and we wanted something different in the views than just the database model. And we could just create a different model and map those fields between them. But that would be the boring part, right? You don’t want to write this mapping.

Robert [31:11]: We are smarter.

Miłosz [31:13]: Yeah. So it’s the usual question we get asked, right? Like, okay, I did clean architecture and I separated my models between database and HTTP, let’s say. But what now? How do I automatically map those structures. Yeah. You don’t. Like in this project. We exactly thought we are too smart to do it by hand. So we wrote a library that could do it for us using reflect and custom struct tags. And please don’t do it.

Robert [31:13]: It was great.

Miłosz [31:52]: And since then, my rule is if you use a statically typed language and use reflection think twice. Is this really what you want to be doing?

Robert [32:05]: Yeah, probably second outcome is that don’t be smart. Usually being smart as programmer, it’s not the best way because you may become too smart. And your solution may be that smart that nobody will understand that. And it will take a lot of time to maintain it.

Miłosz [32:26]: This is actually my favorite part of this story is the why behind it. We just thought we are smarter than writing boring code and that led to more trouble. So you have to be careful around this.

Robert [32:48]: All right. One important thing. So I think in general, the discussion about frameworks that we have today, it’s nicely connected to the previous episode that we had two weeks ago. So if you didn’t have a chance to listen to that, I recommend it because in general it’s often well, choosing framework is big architectural decision. So often if you do it at the beginning, it’s quite hard to remove it later because as we said earlier. So it’s a matter of coupling your models to rest of the application. And it’s not like replacing your HTTP router or something like that. It’s usually very tightly coupled, and it’s just hard to remove that. And yeah, in previous episode we were discussing pretty deeply that it depends a lot in which stage of project you are or if you think this project will be successful in long term, because probably if you are doing some pet project or you have some small team and it’s proof of concept, probably doesn’t matter that much. But if you see that your project is taking traction and team will be bigger, it may be problematic to go out of this decision.

Miłosz [34:20]: So this assumes someone would need to remove a framework, right. Which usually doesn’t happen if it’s a big and proven framework that many people use. But with smaller frameworks it’s a bit more risky. So some time ago we worked with this go-micro framework. You remember, for microservices.

Robert [34:46]: I remember.

Miłosz [34:47]: Across many teams, tens of microservices, and there was some issue with the protobuf version there. There was something custom.

Robert [34:57]: Yes, I think it was some custom implementation of protobuf that supported something extra, but at some point it was no longer maintained.

Miłosz [35:06]: Was it related to gogo/proto or not? Doesn’t matter. But somehow there was a decision to remove it. And that’s a long story short.

Robert [35:18]: Yeah, I think so. It wasn’t like, okay, we need to remove that. I think it was leftover. Somebody decided to use it a couple years earlier. A year and couple years later we were in the project and it was some leftovers from that. And I think it was not maintained and it didn’t support something. And it also have some weird edge cases. At some point we needed to get rid of it because it blocked something, something else.

Miłosz [35:44]: Possibly also back then Go modules weren’t as great as is it now. And I think there were also some issues with this, but basically there was decision to get it removed and it went exactly how you would expect a framework removal to go.

Robert [36:04]: It to be exact. So it wasn’t removing entire framework, it was just leftover from something that was there since a couple of years.

Miłosz [36:13]: I think it took man-months of, I don’t know, half a year, probably in total effort from people from different teams because this spanned across teams, because it was related to communication between services. So obviously it’s what connects the project together. Yeah. I remember when we finally got rid of it, there was a big maybe not event, but I remember celebration where everyone was super happy. I mean, I were too, but then you start to think about how dumb effort this was, you know, spending time on removing some obscure library that caused some issues, but, you know, shouldn’t be there in the first place probably.

Robert [37:07]: I think it’s an excellent example that some decision that happened a couple years earlier that have that big impact later. And yeah, it’s not the only time when we’ve seen that. So obviously it’s hard to do all the decisions fine all the time. But yeah, the thing that I mentioned. So it’s worth thinking like, okay, I’m using this framework library or something like that, just in case if it will be deprecated, how I can remove that. And if the answer is okay, It will be super hard. Maybe it’s worth thinking. Okay, maybe we should choose something different or how we can mitigate that. Or again, maybe again. It’s a bit more in the previous episode about that, but maybe it’s fine because maybe it’s proof of concept.

Miłosz [37:58]: Probably hard to decide what is a good, you know, or maybe maybe a different way. What is what are some red flags that tell you stay away of this framework? Because I guess if someone uses a proven framework like Spring or Ruby on Rails or Symfony or whatever…

Robert [38:20]: Probably it would be not deprecated.

Miłosz [38:22]: Probably. Yeah. Because the community is big enough. That’s one thing. If the company behind it would drop it, I guess you could expect the community to just fork it and maintain it over time.

Robert [38:36]: Or maybe not, because it’s just not adopted enough and because I think it’s the thing in Go. So in Go we have pretty, it’s pretty big segregation of frameworks and there is no one leading framework. And maybe some people will do some updates, for example in the fork. But I would not expect that it will be super big updates. So it’s risky in Go, I would say.

Miłosz [39:04]: Especially if the framework is created by one person or a small team.

Robert [39:09]: Or for example, some other risk that people are often not thinking. So if it’s created by some VC backed company. So, you know VC capital is not free, it’s not charity. So probably they will need to make money at some point. And we’ve seen recently when it was big shift in trying to get money from multiple products and some licenses changed, like with Terraform, for example, and it’s definitely a risk. Again, especially with things like framework that it’s just coupling everything and it’s hard to remove. So I can imagine that in many cases it’s like, okay, let’s better pay for that and forget.

Miłosz [39:58]: Or maybe let’s just pay for the first year and then we’ll migrate. And you know, this migration task sits at the bottom of your backlog, and one month before the next invoice, you realize, oh, we should migrate.

Robert [40:13]: Task: remove framework.

Miłosz [40:15]: Oh, let’s do it next year. And then the next year, the company shuts down because they run out of funding. So yeah, there’s a risk.

Robert [40:29]: If you are thinking about framework, it’s also worth looking on the features list and assess if you need everything from this list. Because frameworks are usually providing many things out of the box, but very often you don’t need to have everything out of that. For example, do you really need, I don’t know, payment provider integration? Do you need ORM or many things from that? And it’s a question if it’s worth paying the cost of coupling everything. Or maybe it’s just better to use something independent.

Miłosz [41:08]: And often frameworks have to be that, have to cover all those edge use cases because otherwise they are not useful because people use a framework and they have all kinds of requirements. I think it’s even for authentication is a good example. I think there is this authboss in Go for authentication kind of framework. I don’t know if maybe it’s more a library, but I think it nicely shows how we can have so many approaches to authentication. If you want to create a library that’s popular, you have to support all of them. Otherwise people will just skip it because it is not supported. But as a user, in contrast, you only probably want 10% of what it offers.

Robert [42:02]: And I think it’s also making the complexity rising exponentially, because when you have multiple things like database models, logging, observability, payments, view controller and you need to have everything integrated. Adding one case, it requires you to add it very often everywhere. So obviously it’s much harder to make it flexible because you are often focused on providing one path that works. But if you need some flexibility around it, just maybe not compatible with the rest of the components.

Miłosz [42:39]: And this one is often very, very useful. If you have a proof of concept app or just a trivial CRUD domain. Then having this happy path scenario and you know, one model for everything that’s actually quite useful because you don’t waste time on this mapping and so on.

Robert [42:57]: That’s true. But from other sides. Well, experience tells me that often applications are not CRUD. I mean, I like to think about that. Okay. If your application is simple CRUD, maybe spreadsheet is enough.

Miłosz [43:13]: Exactly. That’s that’s the usual case, right? Like, why would you build custom software if it’s that trivial? For stuff like this, this no-code solutions should be ideal.

Robert [43:27]: Or spreadsheets. I love spreadsheets.

Miłosz [43:29]: Or some of the more, more complex spreadsheets that are now quite popular.

Robert [43:36]: Oh, come on, you’ve seen what you can do with Google spreadsheets and that entire startups can live on Google Spreadsheets, and it’s fine for a while.

Miłosz [43:46]: Yeah, I think the biggest trap is if you think you have a CRUD domain, but you have not, and you start with this single model for everything, and then you realize it when it’s too late. And we have a lot of articles about this on our blog, so you can take a look. There’s this one to watch out for the DRY, don’t repeat yourself principle, which is about this issue. So we will link this in the summary later. So you can take a look. And it’s also about the mapping issue I mentioned before. And I see that Grzegorz in the chat asked about this mapping. So there is a mapping library in Java mapstruct and it is common to use it. So why can’t we do it in Go. And how did we overcome the mapping problem. So the issue with the magic mapping we created was that it was just hard to maintain because there was a lot of reflect code, which is hard to grasp and test. And, you know, long term it will be just it would be easier to just map the structs manually. And that’s what we did later in later projects. So that’s how we overcame the problem. We didn’t. We just kept separate structs and mapped one to the other when we had to. And I don’t know, maybe it sounds boring or bad. It is quite boring, but there is a big plus of boring code. It’s trivial to understand and test and change.

Robert [45:40]: And it’s also compile time checked for that. So if you break something, you written it properly. Compiler will tell you that it’s something wrong here.

Miłosz [45:51]: And today we have so many, you know, copilot tools that can generate it for you that it’s even better than it was a few years ago. It’s basically free. So that’s you know, that’s our approach right now. We prefer explicit to magic.

Robert [46:10]: And I think what I can recommend is just trying it because again, it may sound boring, but yeah, I would recommend trying that and just feel that it’s not that scary. And actually it works quite nicely. And again, don’t be afraid of boring code or boilerplate because again, we tried different languages also where it’s more promoting some magic, some elegant code, but at the end later it’s super hard to understand. Like Python is probably a great example that okay, it’s in general allowing you to create nice interfaces that look really nice, like some decorators, but at the end you have no idea what’s happening under the hood, and in Go, you just cannot do that. And it’s very explicit. And you can navigate in the code up to system calls to understand what’s happening. And it’s very linear I would say. So there are just navigating and you see what’s happening.

Miłosz [47:12]: But maybe there’s also a good idea now to remind of the no silver bullet principle that now what we just said about this manual mapping and all of this is especially nice if you have if you work with a team in a bigger project and you know, we have you have to have some consistency and maintainability for the long term. And we talk about this in the previous episode. But again, if you have a proof of concept application or something you work on just yourself or want to try to build some MVP, then of course just doing this single model might be a better idea, maybe just simpler and easier in the short run.

Robert [48:02]: Yeah, I think it’s a really good point. And usually where the problem is, it’s when people are going into extremes, like always do it like this and it’s it’s usually the bad idea. And yes, you said so. It’s also not like where you always using separate model for everything. No actually we often using one model for multiple things. But it’s in case when the model is the same everywhere. So for example, or sometimes and sometimes we’re also wrong. I mean, we’re creating some two models for something. And we later see that okay, we’re always mapping it 1 to 1. Maybe let’s just make it one model and whatever. So it’s important to be kind of mindful probably in that. And if you see that you’re doing some stupid mapping and it’s always the same. Because what we just said a couple seconds ago about this mapping manually, etc., it’s good for the models that are different. But if you are always have the same fields in two models, maybe you don’t need to have two models. Maybe it’s fine to have one model. And it’s also a matter of encapsulation, because sometimes you need to have DTO when you are doing an encapsulation like in some of our projects that. So for example in Wild Workouts project that we have. So in this case it’s we are introducing some transport DTOs to have encapsulation in domain layer, but again, it’s not always needed. If you sometimes have some pretty simple models encapsulation, whatever you have, small team, whatever, you have 100 people team, you know, it’s totally different thing. And spoiler alert. So I think it’s also in the topic of clean architecture that also a lot of people are saying like, oh, clean architecture is overengineering or whatever, but okay, how complex is the project that you are mentioning? How many people were there? Was the person that was using clean architecture aware how to do that? Maybe it was overengineered, but maybe it was not the proper implementation.

Miłosz [50:02]: We’ll talk about it in two weeks. About clean architecture. I think what helps is knowing both extremes, because then you can just consciously choose one method. It’s much worse if you don’t know about it. Just stick to one and use it for everything. But what’s also tricky is consistency, especially in a team. If you map structs for the entire project and then someone wants to skip it in one place, it usually rises long discussions about should we do it? Should we not do it? But I think if you work in a team, it helps to have more strict guidelines. So you just get rid of those discussions or you know, someone has to decide.

Robert [50:53]: All right. So time to go back to frameworks. So we already know where more or less we should avoid that. Let’s name it vendor lock in. So we should watch out for frameworks that can become deprecated. Or there are some problems with flexibility. And especially in Go when we don’t have the frameworks that are standard one. So we may be more afraid that something will be deprecated or maybe non-flexible. So what do you think are the alternatives in in this case.

Miłosz [51:30]: What worked very well for me before is just modular libraries, you know, just connecting libraries together. And it can also mean internal libraries in your company or in your project, the so-called common package. That’s also considered bad practice sometime. But I think you need some consistency usually. Or you want some consistency across things like, you know API communication, messaging, maybe database as well across projects, especially in a team or across company, usually makes sense to have some standards around this. So you don’t have just wild west, everyone doing whatever they like. But the difference is, instead of this one framework that you have to fit into the project, you just you use the libraries for where they make sense and sort of like building blocks or puzzles. You just pick what you need and leave the rest. And if something changes or breaks or you want to remove one thing, it’s quite easy to do. And I think what’s also very nice is you hide the complexity from like whoever uses the API. I had this experience of setting up a code base in a company in Go, and I took this approach. And for example, we used Watermill for messaging, and I hid it behind internal library. And there were all the bootstrapping, all the settings and so on, which can be quite complicated if you use some Pub/Subs. And the nice part is, you know, my colleagues didn’t need to know exactly how it worked, although it was documented and quite easy to grasp. But as a user of a library, you usually want very small API, right? So for example, if you use an HTTP router, you don’t want to, you know, work with the internals of the protocol. You just want to say /users GET and that’s my function. And that’s it. You don’t want to configure it further unless you want to do something more specific. So then you can. But I think it’s a nice design philosophy in general to have small APIs.

Robert [54:16]: Yeah, and I think in practice it’s not that hard to usually do that. Again, we’re mostly talking about Go now, but I think often it may sound like rocket science. And I think a lot of people are really promoting frameworks like it’s helping me to create stuff so fast. And yeah, you can also if you go to the frameworks websites, you can also see some big logos using those frameworks. But you know, don’t be fooled by that that much because I’m not sure if you know how often getting some logos on your website work, but you know, if you have some colleague working at some big company, you can ask him to use your library or framework in this company, and later you can put this logo. So, you know, it’s often more like that instead of, you know, I don’t know, some Airbnb using some big framework, so you know, it can be just used in some back office application used by couple people in the company, and it’s fine.

Miłosz [55:20]: Sometimes if you read those stories, it seems like the framework is the success behind the company or something. And when I read this I feel envy because, you know, I would like to be as a successful as they are. So maybe I will use this framework. But most of the time it just gives you this boring boilerplate. That’s quite nice, but also not life changing usually.

Robert [55:51]: And sometimes even opposite from my experience. I mean, I’ve seen some projects where again, it wasn’t that it was couple months to remove the framework. It was rather impossible to do that even. I think I would find some projects that I built. I use a framework and sorry for everybody who is maintaining it now, but I was young and I was optimistic, but I’m pretty sure that if they was able to get rid of that, that’s great, but I’m pretty sure that they didn’t. And it’s probably a mess to.

Miłosz [56:25]: What we didn’t cover yet is in-house frameworks, which can also be a trap because then you have no one maintaining it if you don’t do it.

Robert [56:38]: So I think it’s probably some balance between some internal libraries and that are using something that is pre-built versus frameworks.

Miłosz [56:46]: I really like the internal libraries approach, and especially to hide some open source libraries behind it, behind some API. And one example might be observability. So I think for a logger, you know, it’s I like to expose my own logger type as just a small interface and then use some implementation behind it. And I think you replaced logrus recently with slog in our project. And we used the same approach. And it was quite easy.

Robert [57:25]: Yeah definitely. And probably the same goes for metrics and for tracing. So usually metrics and tracing the interface is not very big. So most of the interface is matter of setup. And but the function to do trace to report metrics. It’s pretty small. So you can and well it looks that those libraries are also changing pretty often. And if you look in the longer time span it it may change again.

Miłosz [57:55]: And the metrics and tracing calls will be scattered all over your your code base. So if you stick to the. You just import the library directly everywhere. You will need to redo it everywhere the same way. And maybe not a big deal. You know you can just do some search and replace or whatever, but it’s much easier if you stick to your own interface and then just replace the implementation behind it. And that’s quite a nice outcome. And you don’t need a framework for this separate package.

Robert [58:31]: And what’s also nice is that you can also hide a lot of flexibility that those libraries are often supporting. So you can expose a simple interface without a lot of complexity flexibility. And it can also standardize how, for example, you are doing metrics.

Miłosz [58:50]: And maybe one counterexample for frameworks or we can have from our own is Watermill. Right. So for people who don’t know Watermill is our event driven library, and we especially had this idea in mind in the beginning to make it not frameworky at all. So the idea is that you should be able to replace it with anything else you want. And of course, it depends on how tight you couple your application with it. Hopefully you don’t. But yeah, that was the idea from the beginning. And I think it it works pretty well right now in our projects we could just, you know, throw it away and use something custom.

Robert [59:42]: And yes, I think it’s nicely answered the question that I mentioned earlier. So how that you can understand if something is coupled to your application by understanding how easy you can remove that. And for example, if Watermill. Well, you can do it pretty easily because you can still keep some application using Watermill and consume messages by any programming language practically. And it will be just compatible because under the hood you have messages on some message broker of your choice and it’s compatible with everything and vice versa. For example, if you have some event driven system that is already there, you can implement Watermill into that because Watermill is not doing anything specific basically.

Miłosz [01:00:27]: Yeah.

Robert [01:00:32]: One more thing that is also interesting about if you think about frameworks and I think it’s visible in the different languages, maybe you heard sometimes the let’s say title like Spring developer, Symfony developer or whatever. And it’s actually kind of makes sense because there are people that are let’s say experts in all this framework works and how to use that.

Miłosz [01:01:01]: I remember like ten years ago, there was a big part of software houses were using Ruby on Rails, for example. I bet they hired experts. So it made sense. Or maybe still makes sense sometime. But…

Robert [01:01:21]: I think it’s one big downside because you know, those framework knowledge how it tends to become out of date. So you know you can spend days weeks or learning something about framework, but it can be outdated. And if you switch to go to other programming language or to other company, a lot of effort that you spend to learn stuff, it will be just wasted. Because this is not, let’s say, universal knowledge that you can put in other projects. So I think it’s also worth mentioning that I would not really recommend to specialize in some frameworks, because you can use your time more wisely.

Miłosz [01:02:08]: And you can learn some patterns that are unique to the framework. And for example, if it supports messaging, you can you could learn how to send a message and receive it, but you don’t learn how it works underneath. So maybe it’s a better idea to learn even driven architecture, because you know the theory behind it and how it works in general, because it’s better transfers to whatever you will do later. And maybe in general, some more timeless skills like how to split modules in your application, how to make it decoupled, how to write business logic so it’s easy to read and modify.

Robert [01:02:55]: Yeah. And there are a lot of discussions now like will AI replace us or whatever. So I would say that if somebody will be replaced. So probably as framework developer, I would be afraid the most because this actually something in what AI is pretty good. So about having the knowledge about all the quirks about frameworks, how to use some strange use cases and how to connect everything. So it’s an example. So I don’t like to write front end, but AI is actually pretty great in that. So if I’m for example, using react and there are some magical things that I have no idea that they existed, I knows that and can use it pretty nicely. And I can imagine that there are people that have all this knowledge how it works. But well, I would be a bit afraid that I may have less, let’s say, job opportunities. And again, it’s also the risk that it will be out of date and from other side. LLMs are not that great in things like design, modularization and you know, splitting bigger problems into smaller. It’s still something that you can do better than it.

Miłosz [01:04:10]: And you need to have the knowledge to verify if hat the AI spit out is valid, which is also important. If you do it blindly, it might not end well. So, you know, there are some skills that are here to stay probably whatever you do, like those timeless architecture patterns and so on, which probably make more sense to learn than just one framework and be done with it.

Robert [01:04:39]: And I think it’s also something what we promote, and we also see that it works for us, because I think one of the good examples is the repository An article about repository that we have on our blog. So I think we have written it five years ago, and I don’t remember how many times it was on Reddit on /r/golang, but I think it was again there for one week ago with more than 100 upvotes. And I don’t know, probably, I don’t know, 20,000 visits again. So, you know, we written an article about some knowledge that is universal and it was valid five years ago, and I’m pretty sure it will be valid in next five years.

Miłosz [01:05:21]: So yeah, even if you use AI heavily to, you know, to update your code, I guarantee if you use repository pattern, it will be much easier to work for the AI than to just have raw queries.

Robert [01:05:34]: You know what to suggest to it. Definitely. And so yeah, this this is the approach that we recommend and we see how it works for us. Because you know we’ve been learning from PHP frameworks 15 years ago. And, you know, it’s not helping us that much now. I mean, it was nice because we learned that, okay, things can be built in this way. Some things are possible. But we also from other side was focusing on the things that are universal. And we can just use for longer.

Robert [01:06:13]: All right. So time to summarize what we said today. So the thing that I already mentioned. So I think it’s not like it’s always bad to use frameworks. And especially if you are a beginner, it’s pretty useful because it can make you more productive. And it can help you to focus on things that matters. And later you can try to reproduce that what you seen in frameworks and create your own simple libraries that are doing that.

Miłosz [01:06:47]: Yeah, but don’t become a framework developer. Just focus on more timeless skills and universal knowledge because the framework specific stuff will probably get outdated soon. And if you learn the universal concepts, it’s easier to join a team that needs something similar, but maybe not exactly. In your in your domain. It’s easier to pick up new skills then.

Robert [01:07:15]: Also watch out because I think there is a lot of promotion, let’s say on social media about frameworks. Maybe some is paid, maybe some not. But it’s often focuses on the bright side that okay, it’s making you productive. It’s simplifying stuff. But I think it often misses the long term implications that you see. You will see when you use this for 2 or 3 years, and team will become bigger, and you may being the trouble. And it’s even harder after a time, because you’re spending more and more time on workarounds. And when you spend weeks or months on doing workarounds, it’s pretty hard to just remove them because of skunk. False sunk cost fallacy.

Miłosz [01:08:00]: But it’s useful to have some experience with frameworks. So at least you know how they look like, how they work. Especially with stuff like code organization or modules or observability. So you can maybe recreate it with modular libraries later. And you know, seeing a project suffer from framework use may also be useful because you see where the issues are. And, you know, you can often see it in in products, even if you don’t work there. I think, you know, products where it seems like initially they iterate so fast and deliver to market, and then you log in one year later and it looks the same just because there were some some nice initial speed but then lost over time. And yeah, if you join a project like this as a developer it’s a nice chance to learn some anti-patterns as well, which will help you.

Robert [01:09:08]: And again, if you join such company and in the first week, say, okay, we need to remove this framework and let’s try to do that. Let’s listen to the previous episode because it’s also not if product is in the stage that okay, it’s all coupled and you cannot maintain that. It also requires some proper approach on refactoring that. And yeah, remember that Go is also pretty specific in terms of frameworks, while in other languages there are some well-established frameworks in Go it’s not. And I’m pretty sure that the situation will not change. So in Go in general, we don’t recommend using frameworks as long as maybe there is some proof of concept or maybe some your pet project. So it may be nice to see okay how some stuff can look like, but for bigger project with more people in general were not recommended in long term.

Miłosz [01:10:08]: I can recommend module libraries approach like internal libraries for your company. That usually works works well. You have some consistency. You can do stuff one way, but then easier to throw some things out if you need. You don’t need to stick to it.

Robert [01:10:29]: All right, so is it time for Q&A? So let’s see what’s happening in the chat. Where is chat? Hello. So Go is a great choice to build a LLM powered applications, but it is visible to use Golang for developing LLM models. Viable. Yes. So I think I can agree with that. Go is pretty nice language for building applications that are using LLMs because. So if I would for example compare it to Python. So for LLMs it’s often nice to do some gathering of the data for example asynchronously. And good luck with Python.

Miłosz [01:11:29]: I actually work on this currently for our platform. And yeah, Go concurrency model works is great for this. Yeah, but for developing models, I don’t think I can say anything. We don’t have experience with this.

Robert [01:11:47]: Obviously, but I think, you know, most probably 99.99% people are not developing models. And it’s fine. Like you are not building your own CPU so you can use something that other people did already. I think you can build almost anything in Go, mate. Yeah.

Miłosz [01:12:13]: You can, but as as the next comment says, as Konstantin says yeah. I think Python is the tool for now for most deep learning. So yeah, we don’t have any experience with deep learning in Go. So we can’t really answer this.

Robert [01:12:32]: But yeah probably Python will be the choice. But from some recent news it seems that you can build TypeScript compiler in Go.

Miłosz [01:12:40]: Oh yeah.

Robert [01:12:41]: So yeah, it seems that Rust is not the only option. Actually, it was pretty fun to see how many people were pissed because they didn’t use Rust for that. I actually wondered if any of them used write something in Rust. Or maybe it’s some cult. But let’s leave it. Yeah. Daniel mentioned that I just signed your platform guys three days ago. I want to grasp skills in Go. Yeah, I hope you like that.

Miłosz [01:13:11]: Welcome on board.

Robert [01:13:12]: And yeah, we should also have some actually AI related stuff in our platform. So we should have AI mentor soon. So if you are starting to learn, Go. I think it will be also pretty helpful to with some more complex exercises that we have there. So stay tuned. Okay. The next message. When I think of frameworks in Go, the first one that comes to mind is GoFr, which is super useful for smart and concise microservices with all powers like observability, gRPC, HTTP and queues. I didn’t hear about this one. I’m not sure if you did.

Miłosz [01:14:01]: No, not really. Maybe we just stopped looking at new frameworks in Go? Yeah. I think like what we mentioned before applies. Basically it’s I’m sure it works well. Just be careful if you have a bigger project that will be there for a few years. Just keep in mind maybe you will want to migrate out of it.

Robert [01:14:27]: And updating five microservices is easy, but 50? It’s just much, much, much harder.

Miłosz [01:14:35]: I’m sure it works for many people, for many projects just, you know, keep in mind those cons of it and consider what would you do if you wanted to move out?

Robert [01:14:49]: So thought experiment. Can I remove that easily.

Miłosz [01:14:52]: If it’s just, you know, gRPC, HTTP and queues then it sounds like just inputs and outputs of your application. Maybe that’s if you keep it separate from the business logic. Maybe that’s fine.

Robert [01:15:04]: Yeah. So in other words it’s not MVC framework. So it’s not coupling everything. So it should be easier.

Miłosz [01:15:14]: Okay. The next one how to how do you set up an API project. You expect to be big. Do you follow clean architecture? Do you use a package for dependency injection? Yeah. If we expect the project to be big, then clean arch or sure is what we will use and we will cover this in the next episode in two weeks.

Robert [01:15:37]: But watch out because I think everybody expects that project will be big. Reality is often different.

Miłosz [01:15:44]: So but this is also something we talked about a bit in the previous episode about when to write high quality code because this is similar similar question. Right. I expect this, this product to be to be big, and you have to be careful not to go too much too far with the foundations. So, you know, be wary of thinking like, I need to set up all this foundations now. So later I will just add features and it will be super quick. Most of the time it’s not like this. You will need to refactor the project anyway, whatever you do, because it can change drastically. So it’s better to start with some modular libraries, maybe have some reasonable setup in place, but don’t go too crazy.

Robert [01:16:47]: Yeah, and remember 90% of startups fails. So if it’s startup, you know it will be good to not fail because of you. Spend too much time on making your product future proof. So.

Miłosz [01:17:03]: Yeah, dependency injection. We used the wire from Google before it was sort of okay, but also had some issues. I think later we switched to just doing manually.

Robert [01:17:17]: Yeah, yeah. So wire was nice because it’s code generation based. So I think it’s the best what we used so far, but also it’s kind of become very complex at the end. So we found out that it’s just better to inject by hand because you see what’s happening there. And at the point when you feel that, okay, it’s super hard to inject everything. It may be not a problem of DI, it may be a problem of your code, let’s say. I mean, maybe your dependency graph is too complicated, and maybe this is the problem because I think it’s a good example. DI is a good example that it can surface other problems. And some people may think that, oh, okay, it’s a problem with the DI framework, but in most cases it’s not a problem with DI framework. It’s a problem with your code and dependency graph that you just have too many dependencies, too many interfaces. So the most often we think that, you know, there were interfaces for things that have one implementation and it was injected and it was multiple layers of injection. And, you know, just having it one level in the AI and simplifying it, it just made things simpler.

Miłosz [01:18:35]: So yeah, I think we need to talk about it in the next episode as well. Combined with clean architecture because it’s related. But it’s also nice concept of because if you start on a project now, you expect to be big. I think dependency injection is something you shouldn’t be thinking about. because that’s this kind of overengineering maybe like just, you know, start with some something simple. Now start with manual dependency injection and see, I don’t know, three months how it works. If you know if it’s painful then consider some framework for dependency injection. Maybe the framework library for dependency injection. But before it happens, maybe you are overthinking it. Maybe just start.

Robert [01:19:26]: Start simple and remember that a lot of boilerplate doesn’t mean that it’s something wrong with that. I mean, it’s just boilerplate. Yeah, but as long as it’s manageable, it’s fine. So this is pretty similar to what we mentioned a bit earlier about structs. So if you’re passing values between structs and doing stupid mapping it’s great. I mean it’s not a problem when you have dumb code and boilerplate. It’s actually good because you can understand. And you know, the alternative will be reflection and we will have no idea how it works. So having basic, you know, you can even have 1000 lines of code file that is doing injection. It’s fine. Yeah, you can totally manage that. And it’s easier to manage than you think. And for example, if you for example, you start to have dependency, like some things are starting to be dependent on something, you just often need to move big blocks of code from bottom to top. And you actually can see what dependence, what state. And if something doesn’t work, compiler will tell you it’s bad.

Miłosz [01:20:31]: And yeah, exactly. That’s what my experience with, you know, recent my recent projects with main.go Being 2000 lines long with just project setup. It seems really dumb, but it works. Basically if you extend some constructor, you go to main.go and you press the shortcut in your IDE that says next error. You go there, fill the missing pieces and you’re good to go.

Robert [01:21:02]: Yeah. And alternative with this dependency injection frameworks is often some magic that you don’t understand. Highly recommended to try to do it by hand.

Miłosz [01:21:13]: So sometimes boring is nice.

Robert [01:21:19]: Okay, the next question or comment? We don’t know yet. Actually, we’re reading that without checking what’s there. It’s risky. We’ve been pushing my team toward Cleanish architecture. We have modular monolith modules, use domain driven design patterns, but we break the rules where abstractions are too much overhead. That I think makes sense. So. And it’s also totally fine that we also are doing it from time to time that we were thinking that it will be more complex, but it’s not. And it’s important to again be mindful and see that, okay, it’s not serving anything. Let’s maybe remove domain layer and put everything to application, remove encapsulation. And I know it’s hard to do sometimes when you spend time on that. It’s hard like removing unit tests I know, but it’s sometimes something that you’re doing and it’s it’s fine And it’s nothing wrong in being trying to do, trying to or actually to overcomplicate stuff. Everybody does that. We are doing that. You can try to do your best and you will be doing that. It’s fine. No worries.

Miłosz [01:22:31]: Yeah, we kind of mentioned this in the previous episode about breaking rules, so it’s fine to do it, but it’s best if you know why you do it and that you actually are breaking rules, because if you do it by accident, then it’s much harder to decide when to change the approach or when to fix it. But if you do it on purpose, then it probably is fine because you probably know what you are doing. It’s much better than being dogmatic about something and just, you know, announcing that, okay, today we do everything domain-driven design way and any other approach is is wrong. And that’s asking for trouble because there’s no one approach that works for everything. So yeah, that’s sounds like a good, good approach. Wilson. Okay.

Robert [01:23:28]: There is another comment about LLMS. I think there is a difference between deep learning and LLMS or and also LLM-powered applications, the latter opening up to more languages. And since it is just just a bunch of API calls, Go is well suited. Yeah, I totally agree.

Miłosz [01:23:50]: Yeah, we currently work with with this like adding LLM features to our platform and using it in Go is really trivial. That’s pretty cool.

Robert [01:24:01]: Yeah. And again I think I like to compare it to like CPUs or something like that. It’s pretty similar situation that you are using something done in some totally different technology. That’s totally fine. You don’t need to fully understand. It’s useful to understand how it works. But again, it’s something under the API and it’s encapsulated from you. And you can also kind of exchange that. What are your favorite frameworks and why. Oh that’s interesting. So yeah probably as we said earlier. So we don’t have really framework that we like in go but in other languages. So I remember PHP times and Symfony was quite nice for that time. And I like how modular it was. What else?

Miłosz [01:24:58]: Yeah, I’m having a hard time coming up with something because I usually felt overwhelmed.

Robert [01:25:07]: I won’t mention anything front end.

Miłosz [01:25:11]: Yeah. I think I would just say my favorite approach is having those smaller libraries used together. Not in a frameworky way. So I wouldn’t say I have a favorite framework.

Robert [01:25:26]: Yeah. But yeah, it totally was useful to use from frameworks because you have some reference that you can, you know, how it could look like. So it’s helpful. Okay. I’m not sure. For the file structure it seems go has philosophy for some directories like cmd, pkg. So I wonder what is the correct way to structure an API project for the most cases? That’s the hard one. I think we probably should do episode just about structure structure in your projects because it’s pretty okay.

Miłosz [01:26:05]: It’s okay. This is a follow up to the API project that will grow big. So I think the answer here is very similar to before. Don’t overthink it in in the beginning because you don’t know how the project will grow. Right.

Robert [01:26:24]: Yeah, yeah. But probably it’s something that we will cover in the next episode in two weeks. But I think some sort of simple, clean architecture can work. So just having, let’s say three packages even like application interfaces and adapters and you know, it doesn’t differ that much from just having everything in one package, but it’s already some start and it doesn’t cost that much to do that. But again, probably something to go deeper in the next episode.

Miłosz [01:26:56]: Yeah. So for sure some separation makes sense if you use the clean architecture, but you don’t have to worry about stuff like should you use it in cmd or pkg because there is no official Go guidelines for doing it.

Robert [01:27:12]: There is one package that looks like official, but it’s not official. And I think the Go developers already mentioned that that it’s a bit misleading. I don’t remember the name of that, but it’s something like I don’t remember the you know what I mean?

Miłosz [01:27:29]: Yeah, I know this standard Go layout or something like that.

Robert [01:27:34]: Yeah, some standard Go something. So but yeah, it’s not official. It’s worth mentioning.

Miłosz [01:27:38]: But my point is it’s not that important. You know, if you, if you use src and then you want to migrate to pkg, it’s just you know, you do git move and you will do it for you and you will keep the history of the files. So that’s not the important part. It’s you can look up our wild workouts example. We have some similar structure to clean architecture. But, you know, you don’t have to worry that much about it because you will need to adjust it anyway in the future. So I would avoid this idea of, you know, let’s let’s prepare this, let’s prepare this perfect structure now and in the future we will just add features later. That usually will end bad because you don’t know how the project will change over time. So instead, start with something simple. And you know, if I would take it to the extreme, you could start with a single file and see how long you can maintain it, and then just start splitting it into files. Of course it’s extreme, so don’t do it.

Robert [01:28:42]: But yeah, it depends. How big is your team? If your team is one person.

Miłosz [01:28:45]: Yeah, maybe it’s fine. Maybe you can start with a single package and then then grow. I think it’s much easier because you don’t make assumptions about how the code will look like in two months, because you don’t know, and it will be a surprise probably anyway.

Robert [01:28:59]: But yeah, if, for example, your team is ten people, Some lightweight, clean architecture definitely can help. So.

Miłosz [01:29:05]: Yeah. Yeah for sure. So. But as we mentioned last time, clean architecture can be really simple, right? Yeah, but maybe more on this in two weeks. Yep.

Robert [01:29:18]: All right. Okay. So it was comment for this one. So how it’s connected to being idiomatic. So yeah, it’s it’s good to be idiomatic.

Miłosz [01:29:30]: But yeah this is a controversial topic. You know what’s idiomatic. What’s not idiomatic. Need to be careful with that. So you don’t spend too much time worrying about being idiomatic in Go, which can mean different things. I think we have the fifth episode about unpopular opinions about Go. We have. So maybe maybe more later. So yeah, this is what we have already covered before about the mapping and what is our opinion about GoFr? We mentioned that we didn’t use it, so nothing to add here, I think. Yeah. Daniel says we are doing a great job. You’re welcome. Thanks for joining us. Grzegorz asks if we know Substack. It seems to me that your newsletter should be so accessible. Maybe you will gain new observers. Yeah, we read Substack. At least I do sometime. With our newsletter. We just prefer to be independent, I guess. So we don’t plan migrating.

Robert [01:30:55]: We already did three migrations of our newsletter.

Miłosz [01:31:00]: But the nice part is it was invisible to our readers. And with Substack, you never know. It seems like a, you know, cool platform right now. But it’s also a startup. So, you know, I think a similar thing happened with Medium ten years ago or so when they became very popular, and then they started putting paywalls. And basically, you have no control over how the how your newsletter looks like.

Robert [01:31:31]: And I’m pretty sure that it doesn’t have the all flexibility of newsletter platforms like. So we are doing a lot of things with our newsletter, like the podcast notifications and automations. So there is a lot of stuff under the hood, and I’m quite sure that it may not support that.

Miłosz [01:31:51]: So yeah, so we don’t plan on creating but thank you for the suggestion. You can share our newsletter with your colleagues to help us get new observers. And then Wilson says go fx is great. Why would you ever want to migrate away from it? Okay. I guess you misread GoFr. We didn’t use go fx, right?

Robert [01:32:26]: Yeah. Yeah, but from what I see, it’s So this one, as far as I remember, it was reflect based. So this was probably the

Miłosz [01:32:36]: This is a reactive framework or this is a dependency injection?

Robert [01:32:39]: Dependency injection.

Miłosz [01:32:40]: Okay.

Robert [01:32:40]: So I think I think it was reflect based. So it was.

Miłosz [01:32:46]: Okay. Yeah.

Robert [01:32:48]: So it was the thing.

Miłosz [01:32:49]: So yeah I mean it’s a library, not a framework probably. Right. So you can if you wanted to migrate out of it. It’s probably easier than just a framework.

Robert [01:32:59]: Yeah, yeah. But yeah, in terms of dependency injection again, we it’s not that hard to do it by hand. And it’s super explicit and it’s much easier to work with. So yeah we recommend this approach even again don’t be afraid of big files because it’s really simple to handle.

Miłosz [01:33:16]: So yeah. What are your thoughts on urfave CLI or is there an example to the DIY go CLI program? We like it, right? This is the one we use for the tdl cli.

Robert [01:33:34]: I think I’m just checking because I don’t remember.

Miłosz [01:33:38]: Once, but I think we do. And I think it’s quite cool.

Robert [01:33:46]: Yeah, yeah. So I think that this is our favorite one. So we have our CLI for our training platform and we are using exactly this one. So other alternatives I think there are not that idiomatic if you are thinking about that, but they have also some weird quirks. And this one has a very nice API and just works super nicely.

Miłosz [01:34:10]: So we don’t have an example I think of building a CLI tool.

Robert [01:34:16]: But you can check our CLI tool for platform.

Miłosz [01:34:19]: Oh yeah, you can, you can do it and.

Robert [01:34:21]: I can put it probably to this chat into the screen.

Miłosz [01:34:26]: And our going one evening training has some CLI exercises but it’s very basic so I’m not sure if that’s useful. But thank you for the suggestion. We may maybe we should write some article on this, although I think the documentation should be good enough to start this. Really? Not that much.

Robert [01:34:52]: I’ve put the link in the screen if you would like to see some CLI example, so you can check this one. It’s a framework worth mentioning. I mean it’s if you ask the question how easy it will be replaced with something else. Quite easy.

Miłosz [01:35:07]: So I’m just maybe we speak about CLI frameworks. There’s also the bubbletea the one which I used for the pq library in Watermill. Maybe we can also link it later. I mean, it’s a bit different because it’s for building TUIs, but if you consider building a CLI app, that might be useful.

Robert [01:35:36]: All right. So I think there are no more questions on the chat. So last couple of seconds to leave something and and and. Yeah, it’ll be it’ll be time to finish for today. So as we mentioned a bit earlier. So we have our newsletter. So this is the best way to be notified about new episodes. So if you are subscribing to us on YouTube or any podcasting application actually a couple people asked us when will be available on podcasting applications. So we’d like to record one more episode and we’ll upload all three episodes there. And if you’re listening on Apple Podcasts on Spotify, it means that we already did that. But we highly recommend to subscribe to our newsletter so you will not miss any new episode because, you know, YouTube algorithm or Spotify algorithm. We don’t. We cannot impact if it will deliver you the notification newsletter. You will always know that there is a new episode, and the next episode will be about pretty hot topic about clean architecture, and I think we’ll have a lot of nice insights. So highly recommended to be with us. And also, it will be great to hear your comments on that, because it would be great to hear some external feedback, let’s say, for that. And yeah, if you have any comments questions, please leave it on under the YouTube video or on Apple Podcasts or Spotify. We’ll read everything. And if you can include anything to a new episode, please leave it. Yeah. Don’t remember. Don’t forget to give us a thumb up in the YouTube or five star review in Apple Podcasts and Spotify, so it can help us to reach more people and give us some motivation to record new episodes. And yeah, something more?

Miłosz [01:37:38]: That’s it. Thank you, everyone, for joining us.

Robert [01:37:40]: Thank you. And see you in two weeks. Bye!

Miłosz [01:37:42]: Bye bye

When it’s worth to write low-quality code

Wed, 05 Mar 2025 17:00:00 +0000

Quick Takeaways

High-quality code is mainly about keeping good iteration speed over time - can you add features without breaking what works?
Not all code needs to be high quality - focus your efforts on the code that’s most important, changes often, or creates the most value.
The right time to refactor is after you know your product has value, but before technical debt gets too big - make small improvements bit by bit.
Architecture decisions need more thought than other code quality factors since they’re hardest to change later - ask “how hard would this be to remove?”
Balance between MVP and quality depends on your situation - for experiments, cut corners on purpose; for critical systems or enterprise products, focus on quality from the start.

Introduction

In the first episode of No Silver Bullet live podcast, we talk about the balance between writing high-quality code and taking shortcuts.

Based on nearly 20 years of working together on various projects, we discuss when it makes sense to move fast rather than aim for perfect code, and how to avoid technical debt that can kill your project.

We focus on making mindful engineering decisions instead of blindly following rules like “always do X” or “never do Y”. Different situations need different approaches to code quality.

Notes

“No Silver Bullet”: The classic 1986 paper by Fred Brooks that our podcast name references, discussing how there’s no single development that will solve all software engineering challenges.
Our Learning Platform: The project we discussed that started as an MVP and was later refactored
Architecture Patterns:
- Clean / Hexagonal Architecture (we’ll cover in more depth in a future episode)
- Domain-Driven Design (DDD)
Pareto Principle: The 80/20 rule we mentioned - often 20% of the code creates 80% of the value
Future Episodes:
- Why you shouldn’t use frameworks in Go
- Is Clean Architecture Overengineering?
Development Practices:
- Mob Programming - team programming approach for handling complex refactoring
- Time-boxing - setting specific time limits for refactoring efforts

Quotes

Quality is not about some kind of elegance of code because it’s just an artificial thing. It’s not really helpful for your team if it’s pretty.

Miłosz

Often you have in the code places that you don’t touch often or it’s not earning a lot of money… Ask what are the places that we’re changing the most often? What are the places that are creating most of the value?

Robert

I remember one project when we started applying Domain-Driven Design… Everyone loved it in the team… But what I also remember is that it had no paying users and we had to shut it down.

Miłosz

If you did your dirty POC, don’t miss the time when you should clean it up, because it’s easy to go into a spot where it’s no longer possible to do that.

Robert

After working with [legacy systems] for long enough, you might think, ‘I’ve had enough of this, I won’t allow my next project to rot like this one.’ It sounds like a good idea, but it can also be a trap.

Miłosz

Sometimes it can be even the opposite. Having everything super consistent can actually be worse than having inconsistent things, because keeping this consistency requires effort… Sometimes it requires you to use an approach that is not optimal just because ‘we’re doing it consistently.’

Robert

A useful mental model here is to care about the useful product first, not the technical design, which of course is important, but I think it’s easy to overvalue it.

Miłosz

Try to find some places where you can do refactoring in one week… Often you don’t need to rewrite an entire service. You can just do some refactoring in the code and iterate on that.

Robert

Timestamps

Transcript

Miłosz [00:03]: Hello everyone! Today we talk about writing low-quality code. So most often you want to build high quality software, which makes sense, but sometimes you do not. Sometimes good enough is better than perfect. But when does it make sense? And how to avoid technical debt killing your project? I am Miłosz.

Robert [00:27]: And I’m Robert. And this is No Silver Bullet Life podcast where we discuss mindful backend engineering. We spent together with Miłosz almost 20 years working together in multiple teams and multiple projects. And during that time we learned that following advice like “always do X” or “never do Y” is not the best idea. It can limit your growth or lead your projects to failure. In this show, we’ll share multiple perspectives that will help you to make smart choices that we believe may help you to be promoted to senior or principal engineer earlier.

Miłosz [01:04]: We want to make it interactive. So if you have any questions, drop them in the chat. We will answer them at the end. We plan a Q&A session. Or if you have any comments about what we are currently talking about, we might pick them up early during the conversation.

Robert [01:22]: Exactly, but I think it would make sense to start with the question what the code quality is and why it matters. Because this is not something that we are often thinking of. I mean, many people are trying to implement a high quality code, but without really thinking why they are doing that. And what does it mean that it’s high quality and what it gives to that person? So what do you think Miłosz about that?

Miłosz [01:46]: I guess it has many, many layers, probably a long topic in itself. But at a very basic level, I think you can think of the software quality as, you know, the few bugs it has, and this software does its job well. But I think for today that’s not that interesting topic because everyone understands that probably. For me what’s more interesting is the codebase quality, let’s say. And the best indicator for me of it is the iteration speed. So how fast can you add features to your product or just develop your product and while not breaking what it already does well. And what’s probably important is you do it over time. So it’s not for a short project, but over months or years. As the team grows and as the product changes, you are able to keep working with a good pace. I guess in other words, you are not afraid to change code. You can always come back to it and add something new or modify it. So yeah, I would summarize it as iteration speed combined with developer experience. And I guess they often go together well, which doesn’t mean the business is successful because of them yet. But often if you have a team that likes working with a software, it usually indicates that the project is doing well, at least in terms of how fast it is developed. What do you think?

Robert [03:47]: Yes, I think you are right. And well, what can I add more?

Miłosz [03:52]: It’s a good question why it matters at all. But I guess code quality is something, you know, a topic that exists as long as software itself. And, you know, we have many books and papers from decades ago. People were already thinking about this topic. And for me, I come back to it every time I talk with some friends who recently changed jobs. And they usually tell me something like, you know, it’s okay, but I didn’t realize the codebase can be that bad.

Robert [04:31]: But I think it’s quite funny in disguise because you can have signals like that. It’s hard to work with that. But for other side, you can see in the age of AI, let’s say, many people that are you see that they can iterate with code pretty quickly. Just use AI to build stuff. And well, quality is not best in this case, but in some cases they are taking money, making a lot of money. So it can probably indicate that it’s probably not that simple. And there is probably some hidden complexity there.

Miłosz [04:59]: And I wonder how it will change. I guess no one knows at the moment how AI will change this. When I work with, you know, like code generating tools, I mean, AI generating tools right now for code, it’s often, you know, there are like two extremes. I’m either super impressed by how well it does or it produces some garbage. And if I didn’t know what it’s doing, I would be lost probably in the generated code.

Robert [05:33]: I think we can argue if AI can improve and maybe handle this code better. Let’s see. So obviously large language models have many limitations. Probably many of them can be overcome. But yeah, it’s probably let’s see. It’s probably I would not probably say anything, but it will be like that because I know that it’s easy to say that, yeah, it will be solved. AI will be better, but I think it’s not that simple. And all this old knowledge that you mentioned that from old papers, old books, I think it’s quite funny that sometimes those papers, books have 40, 50 years old even. And, you know, they are still pretty relevant. A lot of people forgot about that, about them. But I think they have a very good point. By the way, fun fact, you can probably check from where our live podcast name is coming from.

Miłosz [06:26]: I guess we can summarize it as we still have quality issues as an industry and probably will be even more important topic now. So yeah, I wonder what are some, you know, traits of a healthy codebase. Do you have something that you can, some ways in which you can describe a healthy codebase?

Robert [07:00]: Yes. I think all of that will be connected to what you said earlier, that it should have a goal of supporting faster iteration. So the first thing that is coming to my mind are tests, obviously. So if you have tests, you are able to, you have tests and they are pretty fast, you are able to iterate on things pretty fast, because one thing is that you are not afraid that you will break anything that somebody or you did earlier, and if they are fast, you’re iterating quickly. So this is important thing. The second thing that is also quite important is modularity. And it’s especially important when your team is bigger than just you or just five people, maybe just ten people. Because in this case, when you have much bigger team, having good modularity helps you to change things independently. So I think it’s pretty important to consider this in the high code quality. And the third thing is that how easy it is to understand your code? Because you can have a lot of tests. You can have multiple people working on this codebase, but if it’s taking ages to understand how something works and nobody really knows how something works, it’s also very hard to work, to implement something, because you don’t know what you should touch. And it’s taking really, really long time to changing logic there.

Miłosz [08:27]: Yeah, that makes sense. And probably all this is automated, so you don’t waste time running in circles during development. So it also kind of sounds like developer experience to me.

Robert [08:44]: Yeah, definitely.

Miłosz [08:45]: It’s rather developer experience-related than product-related.

Robert [08:51]: From other side. I have some impression that often developer experience is really about infrastructure. So you know, DevOps and all this stuff. And obviously it’s important. It’s supporting all those things, but I think it’s not the most important thing. I mean, you can have great CI so it will help you to run your tests. But if your tests are bad, the best CI in the world will not help you. If your modularization is bad and you know having the best Kubernetes cluster in the world will not help you because even it can, I think, make things worse. Because changing one thing will require changes in ten microservices for example. So this is opposite of.

Miłosz [09:30]: Yeah, like like we’ve seen in real life.

Robert [09:33]: Yep, yep.

Miłosz [09:34]: The platform can be great but it doesn’t matter. It doesn’t help if you don’t have those more basic pieces in order. For me, it’s also easier to define what isn’t high-quality code. And I come back to times where we started programming a long time ago, and we didn’t know many patterns back then, but somehow I still knew there is a code I like working with and code I don’t like working with. And later I realized it basically boils down to I like working with the code I wrote recently, because I have all this context in my mind, it’s easier to think about it. And I think it’s also a common trap that when you enter a new codebase, you might assume it’s low quality, because maybe it follows some other patterns that you know and you have no mental model of it yet, you don’t know where to find what. And that’s why, you know, it’s like, it often happens that a new joiner in the team recommends overhauling everything because it’s weird or confusing. So that’s and it’s also about our own code for sure. Like, I’m sure you remember, like thinking what an idiot wrote this. And then of course it’s…

Robert [11:12]: Git blame: Miłosz

Miłosz [11:14]: Yeah, of course it was you or me. But this is pretty much the same thing for me. So you forget the codebase over time and you don’t remember it, but it doesn’t matter it’s low quality. It’s just hard to work with for now because you don’t have this context yet. And similarly, quality is not about some kind of elegance of code because it’s just it’s an artificial thing. It’s not really helpful for your team if it’s, let’s say, pretty.

Robert [11:54]: And I think it’s also hard to measure impact of that. I mean, if you think that your code is elegant, so how you see how it affects any of positive stuff that you mentioned earlier. So yeah, definitely, this is a trap. And it also reminds me that some people like some kind of consistency in the code. So it’s actually kind of quite close to that, that people like to have everything consistent in the code. And I would say that it can be even opposite. So having everything super consistent can actually be worse than having inconsistent things, because keeping this consistency, it requires effort to do that. And sometimes it requires you to use some approach that is not optimal because it’s okay, we’re doing it consistently, it doesn’t make sense, but we’ll do it because somebody said that, okay, we’re doing it consistently. At the end, you just have garbage.

Miłosz [12:49]: Since you mentioned that DevOps and platforms, it’s maybe a good example of this because if you have this dream of having a uniform platform that all teams use in the team in the company, then all of them need to follow some similar standards.

Robert [13:07]: Oh, it’s the worst if somebody from outside of your team is coming to you and no, no, you shouldn’t do it like that. You need to do it in that because and it’s even worse if this is the person that really like consistency. So yeah. Yeah, definitely.

Miłosz [13:19]: So consistency is useful sometimes. Sometimes, not really. As usual, there’s always these extremes are bad. Yeah. And I guess we can summarize this as high quality codebase means quicker delivery of code overall like from very high-level perspective. This is the outcome you want. You want to deliver quickly with stable pace. So for me, I think…

Robert [13:57]: Probably it’s a useful question to probably ask yourself like okay, will it help me to. So if you are thinking about code quality, will it help me to deliver stuff faster? Or maybe not. Maybe it is just elegance and it will not give us much.

Miłosz [14:15]: Maybe it’s easier if you think of it as maintainability. If you want to keep working on this project over time and do it with a steady pace, that sounds like something where you want high quality.

Robert [14:31]: The thing that we mentioned already a couple of times is the fact that we are often talking mostly about how to write high-quality code, but I think it would be interesting question to ask if we always should aim to highest quality code, because it might may make sense. May sound like it makes sense. I mean, you would like to deliver fast, so we’re keeping the code quality high, but literally

Miłosz [15:06]: Yeah. Does it make sense to always aim for high quality?

Robert [15:09]: Yeah, yeah.

Miłosz [15:10]: Well, as you know, there is no silver bullet, so probably not. Sometimes low quality can be a strategy or a tool you choose consciously. But also, as always, it depends on the context and some factors probably. Do you know some?

Robert [15:37]: I think that there are a couple of things that could help us with thinking if we should aim for high quality or not. So the first situation that I can imagine is that if we don’t know if this code will be useful for us, so we’re doing some experiment. We’re doing some product that we don’t know if anybody will use. And we know that our long term maintainability may not exist because we may discover that, okay, actually it’s useless. Nobody will use that. So spending hours on making this code maintainable, it will be just waste of time because often maintainability requires some effort at the beginning. Often also we are implementing things that we’re not really touching after implementing that. So if you think about your projects so very often, it’s like that you have some modules or some parts of the applications that you are modifying often and reading often. But there are also some parts that somebody written couple years ago even, nobody have idea how it works, but nobody is really touching that because there is no need to touch that. And it’s totally fine. I mean, it may sound bad, like we should clean up everything or whatever, but it’s the reality that we’ll never have time to do that. And it’s also worth mentioning that we are also trying always to do applications as robust as we can, as stable as we can, but we forget that often those applications are not really mission critical applications. And if it will be down for day, half day or hour, nobody will care or even if somebody will be a bit upset, it’s fine.

Miłosz [17:21]: It’s not a nice thing to think about if you develop the application.

Robert [17:25]: I’m not saying that all applications should be like that, but some applications definitely doesn’t require that big that big stability, et cetera, et cetera. So it’s also worth mentioning that it’s not always that critical to just have stable applications. Again, I know that it may be not always logical, but I think it’s often visible if something will really be broken because, okay, if you ask your stakeholders if okay, it should be stable. Yeah. Yeah, of course you have to make it stable. Later it’s down for a day. Nobody really cares.

Miłosz [18:04]: But as someone developing this product, you probably don’t want to think about it like that. You don’t want to work on something that doesn’t matter. But sometimes it is the case. There are more critical projects. I have a theory of how the extreme take on high quality starts because we often work with some legacy systems. And, you know, after working with one for long enough, you might think, oh, yeah, I have had enough of this, I won’t allow my next project to rot like this one. So from now on, we will start doing everything high quality and we won’t accept any technical debt or some half measures. And it sounds like a good idea, but it can also be a trap if you swing too far to the other extreme.

Robert [19:17]: Do you remember some projects that you worked in and it maybe didn’t need to have that high quality code?

Miłosz [19:27]: I remember the one when we started applying Domain-Driven Design and similar patterns in Go the first time, and I’m often reminded of it when I talk with former colleagues who worked with us on it, and they usually, over a beer, mention something like, do you remember this project? It was so great. It had all the the best patterns and it was so cool to work with. Everyone loved it in the team and I remember, it is true. But what I also remember is that it had no paying users and we had to shut it down.

Robert [20:15]: At least it was beautiful.

Miłosz [20:19]: Yeah. It might be dangerous to daydream like this because most projects don’t look like that. Are not perfect examples of patterns or whatever you want to see in them. But this project was actually a very important moment for my career because I realized how little all those discussions mattered in the end. Like, we try to do the best job we could, but after, you know, after you see the project dead, then you realize, you know, maybe this one hour meeting on naming conventions or something like that wasn’t that crucial in the end, you know, of course, you never know. That’s the hard part. But I think it’s like a good perspective coming from this previous approach of “I hate technical debt, I will do everything to do things right” to more balanced approach of “Yeah, maybe we don’t always need to spend so much time on, you know, the high quality stuff.”

Robert [21:41]: If I’m thinking about that, it’s also one interesting thing. So survivorship bias. So you mentioned that okay this project failed. But I can imagine if this project wouldn’t fail we can think at the end that yeah okay. So it’s all thanks to our elegant code etc. and I think you should all watch out for that. That okay, if people are saying like as we said at the beginning, always do X because it will make your project successful. And it worked in our big projects that everybody knows. But you know, you don’t know if this was the thing that made this project successful. And it’s probably pretty unlikely. I mean, good technology is useful, but I think it’s not the most important thing at the end to reach the product market fit.

Miłosz [22:30]: Maybe it’s just not related that much. You know, there are companies that are very successful, but with probably terrible code quality. And on the other hand, there are failed companies with with great code.

Robert [22:46]: And I think it’s also the opposite. So there are even more successful companies with with great code quality, but I think we will go touch on that a bit later.

Miłosz [22:55]: Maybe a useful mental model here is to care about the useful product first, not the technical design, which of course is important, but I think it’s easy to overvalue it.

Robert [23:16]: And also we don’t want to sound like okay, never write good quality code. We’ll go into that. No worries. Where to, when to it makes sense to go with high quality from the beginning and we’re not too sure or take shortcuts, so wait for a while for that. But I think we may also sound now like, okay, always write bad code. It’s not also like that.

Robert [23:43]: What I think is also pretty important is the fact that it’s more maybe related to what your boss expects from you. So for example, if your boss is saying that okay, it needs to be great code scalability code. So your and you are creating for example, proof of concept. So your job is to ask boss, hey, are you sure that it’s the right time for that? And if he will say that, yeah, yeah, we need to handle thousands of transactions per second and make this code great. And you are not able to convince the boss for going the shortcuts. Well, probably it’s better for you to say what he ordered. I mean, at the end, he’s kind of your customer, and you are building what your boss is ordering from you, so not to be in the situation, like, okay, in one year, for example, your traffic is super big and your boss is asking, well, but you promised us that it will be scalable and it’s not. And you are saying, oh, you know, we needed to do something and it’s also not the best situation. So also watch out. Watch out for that.

Miłosz [24:57]: For sure there is some responsibility on you to explain. Maybe this is not the best idea. Especially if it will take a long time to implement, which usually it will. It’s if you aim for like huge scale or something before you really need it. But at some point, yeah, you’re not the owner.

Robert [25:21]: And remember, if you ask it for the first time, you’ll always hear the question. No, no, it needs to be scalable and high quality. You need to go deeper and say, okay, we can do that. But for example, instead of doing this feature for this time, it will take this time. And, you know, be pretty specific. And you know, your boss is thinking in terms of money and time is money. So, you know, if you use this argument, it can help. But again, it’s not always possible.

Miłosz [25:54]: Maybe we could go the other way now and choose some opposite project where actually high quality would be nice to have.

Robert [26:03]: I remember one project like that and this is a good example that it’s hard to find a good time to do this refactoring or improvement, because sometimes you are starting this project as a proof of concept. You’re doing it in a dirty way, but at some point you are at the point where fixing that will take really, really a lot of time. And I remember one project like that. And yeah, in this case, we should start the, let’s say, refactoring much earlier because it will be easier. And now at the end, again, we were in situation that total rewrite would be the only way to fix that. And unfortunately reality is that it’s often hard to do total rewrite. And even if you are able to do that, it’s often happening that it’s failing, unfortunately.

Miłosz [27:16]: You mentioned about the project that started as an MVP, right?

Robert [27:21]: Yeah, yeah.

Miłosz [27:26]: So it started as some sloppy code, maybe just to validate some idea and then grew over time into something that’s hard to refactor.

Robert [27:37]: Yes, yes, exactly. And again, it validated the MVP in some way and well, but I think the point when we should start to do refactoring was already gone.

Miłosz [27:50]: I think that’s the hardest question. Like when is the moment when doesn’t matter to it doesn’t make sense to keep going the current way. I mean, where is the moment in in the project’s lifetime when you work on the MVP and then you decide, okay, we need to do something about it. We need to change the approach. And when does it make sense to keep going the sloppy way.

Robert [28:28]: It’s an interesting question. I think it’s probably hard to answer. So it’s probably somewhere related to the question. If you start to see that it’s useful and anybody will use that and it will make some money and…

Miłosz [28:45]: Maybe if you ask yourself this question often, then it’s a good sign that probably something could use some refactoring here.

Robert [28:55]: But I think you know what the trap here is that if you are working in a project and it’s bad, it’s easy to get used to that. I mean, so you get used to how bad quality of this project is and you’re starting to think like, yeah, okay, maybe it needs to be like that. What’s useful is probably when some new people are joining your team and every person is saying how bad this codebase is. Obviously, you need to also take in mind that often people are just complaining that code is bad, and when it’s not.

Miłosz [29:28]: Maybe they don’t know it yet.

Robert [29:30]: Yeah. But again…

Miłosz [29:32]: On the other hand, new joiners often have this fresh perspective that’s also useful.

Robert [29:37]: Definitely. But I think I can have. So unfortunately, it’s hard to give one good answer when to do that. But I think I can give you some hints on how to approach that, because I think that this is also important part though. How to approach such refactor. The first thing that I would consider is timeboxing this, because it’s easy to go with never ending refactoring project or total rewrite that very often are just failing because it’s too big, it’s hard to I mean, stakeholders usually doesn’t like projects. Like, you are going to your cave for one year and trying to fix something. After one year, you’re going like, oh, unfortunately, we need to have another two years, so it shouldn’t be like that. So it’s important to be able to cut it to some smaller pieces.

Miłosz [30:37]: And maybe start with some first iteration of changes in a small scope, you know, smaller module or something like that. So you see if it makes sense or not.

Robert [30:50]: And probably it’s good to start with places that are often modified. So you can see the benefit of this. As we said about code quality, it’s all about making your development faster. So if you are refactoring things that nobody touched for five years who cares?

Miłosz [31:07]: I think that’s super controversial. I mean, when you think about refactoring a big project, I think the first intuition is to pick the least used thing, you know, where nothing happens because it’s low risk, so you won’t break anything. We can do whatever. But yeah, first of all, probably it’s simpler because it’s not used much. And second, there is little benefit in refactoring it in the first place. Maybe it doesn’t need to be refactored ever, because who cares if it’s not used that much? So that’s tricky part.

Robert [31:48]: The second thing that’s also pretty important is trying to find place that is not that big and you can change it quickly. So we are not talking about year because it’s super hard to, you know, do one year refactoring of the thing and later deploy that. It’s super risky or you will probably lose a lot of trust. Try to find some places where you can do. I know one day is maybe very optimistic, but let’s say one week. Start with the question what you can refactor in one week, you can deploy after one week and try to do it in small parts. So often you don’t need to rewrite an entire service. You can just do some refactoring in the code and just iterate on that. The good thing is that if you will do it on smaller iterations, you can see if you will break something where it’s broken. Because if you are refactoring something for one year, it’s super hard to later figure out what’s wrong here, because usually you are refactoring some logic that is hard to understand and it’s a mess to try to fix it later.

Miłosz [32:59]: And most probably also key here is to make it working end to end, right? So you don’t work on some unused piece of code for a long time and then plug it in. But do the changes in a kind of vertical slice so it touches on the entire codebase, and you can actually deploy it and see if it works or not.

Robert [33:22]: I know that it may sound like a stretch goal, especially for harder projects that okay, how you can do refactoring and deploy it after one day. Okay. So maybe if not one day, maybe one week, but okay, if you are not able to do it in one week parts, it’s maybe worth asking questions what you can do to deliver it in a week iterations. Maybe we need to add CI for this service / project. Maybe you need to add some tests. No, I think it’s useful to just start asking questions. Okay. Why I cannot cut this refactoring in smaller pieces. And if you spend some time to try to figure it out, you’ll find out that, okay, it’s actually possible. And you will see benefits really, very, very soon on that.

Miłosz [34:08]: There’s one more trap I see here that when you work with code base that you consider legacy, whatever this means in this context, but basically some older project older, which means maybe it’s one year old even, and it’s easy to assume it just has to be this way because everyone keeps complaining about this project. That there is no time to refactor. And just at some point you just give up. And then comes the big rewrite, which is probably not going to help, or it’s going to take massive amount of work and time to finish. So I guess maybe if you see the the project will be there for a while, it’s not going away. It’s probably a good time to start this refactoring earlier than later.

Robert [35:09]: And try to do this refactoring in small iterations. So again, not one year refactoring, aim for one day iterations of changes that you are doing there. If not, start with week. But I wouldn’t go with one month because one month is already very, very long time. So if you are not able to do one week iterations on refactoring, let’s ask questions why and what you can do to be able to do that.

Robert [35:38]: Does it mean that it’s not possible to balance code quality? And we need to always start with ugly POC and later refactor it with time, with pain, sweat, and uncomfortable questions from our stakeholders.

Miłosz [36:03]: It probably boils down to this question of when. When to do it. That’s what I have in mind. So to take for example, the our academy platform, which also started as an MVP, because we just had an idea to create something new and we had no idea if anyone would use it.

Robert [36:28]: As a context. So if somebody doesn’t know. So I think three years ago, we had written down from scratch a platform for learning by building projects. So something that nobody is, I think, doing. And we didn’t know if anybody will like that. So it was kind of risky project, especially that at that time we were working in a daily jobs. So we were doing it over nights, over weekends, and we just didn’t have time to do it better. Now, okay, we know that people like that. Buy at that point we didn’t know. But the good thing is that I think after those almost 20 years working with different projects, we found this good time when we started to clean it up. And it wasn’t that painful.

Miłosz [37:20]: It was exactly this dilemma because it was a rough MVP. And, you know, I remember the first version was basically mostly TODO statements in the code.

Robert [37:35]: There’s still a lot of TODOs there.

Miłosz [37:37]: If we had a proper design session on architecture and design, everything, I mean we had some. But to do it properly, I’m not sure if we would not just give up and just, you know, didn’t have the energy to do it. And I think you just scraped the initial POC in a weekend or something.

Robert [37:56]: I also remember that it’s also quite funny how it’s changed with the time. So at the beginning, I think our idea was that you push the code in GitHub and it will run some kind of CI to validate your exercises. And during that time we totally changed how it works. And if we’ll go with this GitHub part and do it high quality, we’ll lose a lot of time and will not be the correct state because we’ll just spend time on things that didn’t matter.

Miłosz [38:26]: I remember we had completely different idea about how the CLI would work, but then you just spent some evenings trying different UX approaches and we would never know it just before. But anyway, my point is that like last month or two, we were exactly at this spot where we have more time to refactor it. But you know, it’s a tough decision because refactoring is also not not giving you any product improvement. So it doesn’t feel like you improve the project that much at least from the business perspective. But I think it was a the perfect time to actually sit down and fix some of the issues so we don’t continue with adding more hacks to it, because if we did, then one year later it probably would be too late to come out of it. So we fixed the CI improved it a bit. We migrated to Postgres from Firestore and refactor the codebase a bit. And again, it was mostly developer experience, but I think it was a good investment at the time.

Robert [39:53]: And it also didn’t take ages at the end. I mean, you were doing ugly development for three years and yeah, it took some time, of course, but it wasn’t like a one year project to fix that. And we’re able to do rather small iterations.

Miłosz [40:12]: And it was also tricky to pick what to fix, because I think it is also dangerous to take a naive approach of “let’s look at all TODOs in the code and fix them,” because there are still a lot in the code base, but most of them are probably not relevant if we didn’t have to touch them for the last three years. So it’s better to look for something that actually solves some problems, even if it’s developer experience, which is quite important for an MVP.

Robert [40:55]: Especially in bigger teams. Because, you know, if you don’t know us, we are two people team. So not that big team. But so we can still cut some shortcuts that wouldn’t work in a bigger teams. But still there are some things that like CI, that we’re not cutting corners because our time is pretty limited. We are just two and we are not doing only development. We’re also doing a lot of things around. So, you know, having CI that runs ten minutes versus five, it’s a really big difference for us.

Miłosz [41:29]: So if your MVP gets good traction and you know you want to continue developing it, I think it’s better to do it earlier than later. At least some some initial improvements. But this is all about MVP’s and I’m wondering if you think this is the best approach for every project.

Robert [41:57]: You may guess that there is no silver bullet. Ha ha ha ha. Yeah, but so this approach worked pretty nicely for our project. That doesn’t require super big scale. We don’t have any external customers, if it will be down, okay, some people will be upset about it but it will not be that millions of dollars of our customers are lost, but sometimes we were working on such projects. So let’s imagine that you are working in a project that will be later sold to some enterprise customers, and you know that you need to take into consideration bigger scale from the day one. And you know, if you cut some shortcuts and you will be lucky to have some customer with bigger scale, with bigger requirements on the reliability. And, you know, it will be not stable. Your boss may start to ask your questions like, okay, you promised us that it will be scalable. It will be handling 5000 requests per second, and it’s barely handling 100 requests per second. And you know, it will be kind of your fault if you promised something else. And your boss may be a bit upset because it can lead to losing this customer. So it’s not like you can always cut shortcuts. You need to be pretty aware of what kind of project you are working on. And it’s no silver bullet. What can I say?

Miłosz [43:41]: It’s probably a big decision up front. If you want to start with a rough POC or design something more like proper high quality architecture from the ground up.

Robert [43:57]: Do you think that there are some places where you should not cut corners ever? So in other words, if there are any places in your projects that you never, ever should. Okay, never say never, ever. But in most cases, you shouldn’t cut corners. Do you have something like this in mind?

Miłosz [44:19]: I think if you want to break rules, it’s best if know them beforehand. So you know the patterns and the better you know them, the better you know when it makes sense to break them. And that’s how you know when cutting corners works best. And because I think our platform is a good example of where cutting corners on purpose is much different than doing them by accident, which might be sometimes something similar. But at least you you know how to fix them. And you can leave some some context for yourself in the future so you understand why this works in a simpler way right now. Why this is a hack, but it’s not an afterthought, but something you do on purpose.

Robert [45:22]: In other words it would be good to understand if you break some rules why those rules exists.

Miłosz [45:33]: And especially now coming back to the AI discussion, which can spit out huge loads of code for you, which is very useful sometimes, especially if you work on front end and you don’t want to do it. But still, if you don’t understand what’s going on behind the scenes it probably won’t take you far. At least that was my experience so far.

Robert [46:03]: There’s also another example that I think cutting corners doesn’t make sense, and this is architecture. So for architecture there’s one bad thing that is usually hard to change. So we often seen that we’re working in some projects that were already successful and in terms of traffic customers. But there were some decisions that somebody made at the beginning of the project, and it was already impossible to change them. So we were able to refactor some services, what were inside of them, but still they were working environment where some other decision like using some library, using some framework.

Miłosz [46:52]: Frameworks are the worst for it.

Robert [46:54]: Yeah. And you know, if somebody were just be a bit more, more mindful at the beginning of this project and think about this architectural outcomes of this decision, it would be much, much easier to develop something. And sometimes we were participating in some rescue projects that were trying to fix those architecture problems, or maybe removing some frameworks that were used there, but it took like man-months to remove something. So let’s imagine couple teams needed to work for a couple of weeks to remove one library. And it was like just crazy amount of time lost just because somebody used some library at the beginning of the project.

Miłosz [47:37]: And it was tightly coupled to all the microservices in the company.

Robert [47:44]: What helps in this scenario is modularization. So if you are able to modularize things so you can develop them independently. And this is for example a case in our learning platform. So we have some modules that are super ugly. And I have no idea why it works but it works. But we’re not touching that because it works and it works properly. And if it will be the place when we’ll be changing often, obviously we’ll refactor that, but it works. So, you know, it’s maybe it would be good to make it nicer. But let’s be real, we’ll never have time for that.

Miłosz [48:21]: That’s a tricky part, right? Because the architecture should be probably high quality. You want to make it the proper way, but it’s also easy to overengineer it. Right? Especially if you start with an MVP that doesn’t need all this consideration before you can just be stuck with some system that you thought will handle all possible use cases in the future, but turns out it’s not really needed. You will be fine with a simpler approach. So that’s something to be careful about. And on the other hand, no design is usually bad design, so you probably want some initial idea at least about how the system will work.

Robert [49:19]: I think one of the best advices here is to build stuff. I mean, because it’s really hard to give one advice like how to not over design things and how to see the things that will get into fire later. And what I think worked, at least for us best was just building, building, building and building. And after time you will break a couple of things. We’ll see. You will see how in some cases it worked badly after time or aged badly, for example, I think it’s, for example, useful to work in one project for a longer time to see how it will really what will get into fire from your initial assumptions and it’s help. And yeah, this is what we always advise people to build, to build and do it mindfully. Because if you also do it not mindfully and just build stuff without thinking about that much, it will not help you much because you will be just in the same position for longer time and will not develop yourself.

Miłosz [50:26]: Yeah, it makes sense. Sometimes it also depends on the team size, right? And how experienced people are in the team. So, for example, if it’s just the two of us working on the MVP and you know, I see you commit 50 TODOs in one commit, I’m mostly okay with it because I know you will tidy it up later. But in contrast, if you have a team that’s just formed a couple of months ago and people come from different backgrounds, don’t have that much experience with the current architecture and stuff like that, it makes more sense to have more, you know rules in place.

Robert [51:21]: To summarize a bit. So I think there’s one tip that can help you to maybe decide if something is important architecture decision. And this is the question that you can ask yourself. And this question is how? So if you are adding something to the code to think how hard it would be to remove that. So I mentioned the history of removing one library for a long time for the project. And if somebody would ask himself a question, okay, how hard it would be to remove this library. The answer will be “super hard” because it’s just coupled to everything. And you can ask this question about almost everything. So if you are using some framework or you’re, I don’t know, using some external vendor, some database, whatever, you can always ask your question how hard it would be to remove that. And if you don’t have that much experience already with building architecture, building products, it will help you to, you know, start thinking about things. And sometimes, unfortunately, the answer will be it will be hard to remove that. And okay, the follow up question will be do I really need to use that?

Robert [52:38]: Often, okay, if you think for that for a while, maybe not. Maybe some frameworks or external vendors, maybe you don’t need to use that. But sometimes, unfortunately, you need to use things that you need to couple with. Like if you are using Postgres for example, removing this later will be harder. I will not say that it’s impossible because it’s possible, and it’s probably unlikely that we’ll do that, but it’s a decision that you are taking that okay, I’m using Postgres and I will need to stay with that. So if you see anything with you like need to stay for longer, really be sure that this is the thing that will not hurt you. Like Postgres for example, I would not say that using Postgres is let’s say risky, but for example, using some vendor that, you know, just had founding one year ago and they released their proof of concept that and your CTO would like to use that. And you see that it will be just hard to remove from your code. Maybe it’s not the best idea to use it. Maybe, I don’t know, maybe Miłosz you have some good examples of such things.

Miłosz [53:53]: You definitely need to commit. And it’s a good thing to commit to something because it simplifies many things. Just need to be careful when you pick what to commit to. So if it’s a stable technology that’s been around for a long time, like Postgres or basically any SQL database probably, that’s probably very low risk. But as you said, if it’s something experimental sometimes it also makes sense because it’s maybe you know, something, it’s a perfect match to your product. But the issue is if you pick too many of them and then the risk is super high that something will go wrong. And now we have to migrate. And yeah, migrations are fun.

Robert [54:45]: But again, remember some things may be experimental and not stable, but it may be easy to replace. So probably one good example are large language models like Claude or ChatGPT. So the API of it is very small. You are sending a list of strings, and you’re receiving a list of strings in a very simplified way. But more or less it’s that and replacing one with another. It’s not a big deal at the end. So you know.

Miłosz [55:17]: That’s kind of how our migration from Firestore to Postgres went recently, which but our code wasn’t exactly ready for it from, from the start even though we promote it on our blog to to use clean architecture and use interfaces for it. We didn’t care that much initially for the POC, but it was very easy to reintroduce. So that’s the part about conscious corner cutting.

Robert [55:51]: But it’s also probably worth mentioning that in our platform, we are not really storing that much data in Postgres or we didn’t store that much data in Firestore.

Miłosz [55:59]: It was quite simple.

Robert [56:00]: Yeah, but there are applications where you have a lot of data and it will be, i would not say impossible because it’s always kind of possible, but it will take a lot of time. Yeah. So there’s also one thing that I think you generally should not cut corners on. And this is your pipeline. CI/CD pipeline. And we already touched on that a bit earlier. So why CI is important. So again CI when it’s unstable, slow it’s just wasting a ton of your time. Again, it’s just me and Miłosz in our company and we are building our stuff. And you know, we don’t have that much time to do that. And if our CI is unstable or slow, we have even less time. So making it fast and stable just allowed us to go faster. And again, at the beginning it often happens that it’s unstable, but in general, from my experience is that if you try to not keep CI stable soon, it’s really, really hard to improve it later because it’s ongoing work to keep it stable and fast. And doing it later will just require a lot of time.

Miłosz [57:24]: And you keep using it every day, Every person in your team uses it. So all the tiny issues add up and you just keep wasting more and more time over time. It sounds like we talk about developer experience a lot today, but I think it’s also quite important because the morale of your team is a big factor in pushing the project forward. Like if everyone’s frustrated with a slow CI or, you know, flaky tests or a project that’s hard to set up locally, it’s a hard environment to iterate quickly on anything.

Robert [58:15]: And remember, if you need to convince your boss to have time to work on CI, remember your boss is thinking in terms of money. Get the average salary in your team say, okay, our CI pipeline is taking 20 minutes. I can take a tea, eat my breakfast during that time. It costs this amount of money. Our team has this amount of people improving performance of CI by 50, 70, 80% will save us this money.

Miłosz [58:47]: It’s one of the best, the easiest metrics you can measure. It’s very easy to track how many time you spend on waiting on CI, and how much it improves over time.

Robert [58:59]: And if it’s, you know together with flaky tests, it’s even worse. I mean, I’ve seen CI in my life. That pipeline took 20 minutes, but, okay, I know it’s not that bad. I heard stories about pipelines that are taking days, but let’s don’t go there. But, you know, if it’s flaky and you’re waiting 20 minutes three times, and you also there is some delay when you are retrying. It’s just a very big waste of time. Yeah, and it’s also a lot of frustration. That’s it. And to make it worse, it’s also a lot of context switching because okay, you can drink one tea, eat one breakfast, but after the second retry what you will do.

Miłosz [59:44]: That’s also why we spent some time to refactor our CI in the last iteration because we had these issues. It’s much better.

Robert [59:54]: And remember, it’s also sometimes fine if you see that some tests are unstable, disable that. I mean, if they are flaky and you don’t see value in them, disable them. Remove them. It’s also fine.

Miłosz [01:00:11]: Maybe not fine, but better than retrying over and over.

Robert [01:00:16]: Definitely. And I think it’s also important to mention one thing because, okay, I know that it’s already a bit complicated, that there are different kind of projects when you can cut corners, should not cut corners, you shouldn’t cut corners. But unfortunately, I need to add another level of complexity here. And this level of complexity is company culture. So, you know, I know that in some companies you can have time to refactor things after creating POC, but unfortunately it’s not the case in all companies and in some companies out there. If you will develop something, you will never, ever have time to refactor it. So it just makes sense to do things right from the first day, spend more time to do it properly because you know that, okay, if I will not do it right now, I will never have time to do it properly and I will just have mess to work with forever. So it’s also important to keep this dimension in.

Miłosz [01:01:18]: And it also depends on who decides about this choice, because you probably don’t want to ask your manager or product person or whoever, “should this project be high quality software?”, because no one will tell you, “No, it shouldn’t be.” So on one hand, it’s an engineering decision, but on the other it also needs to be a dialogue sometimes with the management. If there’s some tight deadline you need to reach or something like this, but it’s easier to talk about, you know, not the technical quality. Like an example of bad idea is, okay, we will skip tests for this to make it faster. Instead, I would focus on cutting the scope of the feature of the product. And sometimes the Pareto principle applies here. So maybe you can find some, you know, like maybe you can cut 20% of the scope that will take 80% of the time to fix, to work on or to deliver. And if you have this dialogue with the product person or product team, then it’s much easier to cut the scope in a way that won’t affect your codebase quality. So you can keep your developer experience. You can keep having tests and a nice environment to work in, but you can cut some of the scope, at least in the current iteration or the first iteration. And the Pareto principle is maybe a nice idea here to ask about the project in general. Like maybe some parts of it can follow more strict high quality ideas and maybe some parts can be simpler.

Robert [01:03:32]: I’m not sure if you actually explained how Pareto rule works.

Miłosz [01:03:37]: I didn’t. So if you don’t know it, the idea is that often things split in this 20/80 percentages. So let’s say 20% of the code takes 80% of time to write because it’s more complex. And the remaining 80% takes 20% because it’s much easier.

Robert [01:04:03]: Or 20% of the code earns 80% of money. This is also very often the case.

Miłosz [01:04:08]: It turns out this applies to many things in life. So I wonder about the same. Apply that on the project level where we can focus the 80% of focus, let’s say.

Robert [01:04:22]: We already touched on people that like consistency in the code, and we already touched a bit earlier that it’s usually not best idea to do. And it’s also with code quality. So some people would like to approach code like okay, it should be consistent quality and standards over entire code base. But unfortunately in practice it’s a waste of time because like I mentioned multiple times, often you have in the code places that you are not touching often or it’s not earning a lot of money. Or maybe it doesn’t need to be stable because nobody will really care about. So I think some important ideas how to find this 20% that is the most important in our code base is asking a question, “What are the places that we’re changing the most often? What are the places that are creating most of the value of the company? Or what are the places that where the stability of our application matters the most?” And in those places you can use some sophisticated methods like Domain-Driven Design or do some better layering, because this code usually that makes money is often changed. It’s usually the most complicated code in our company, and it’s just worth to keep it high quality because it will pay back pretty nicely because we are touching it often. It’s complex, so we’ll be able to work with that faster. And again, it’s important to mention that not all code in your application needs to be bad. Not all code in your application should be good. It’s well, for some magical reason, usually some sort of part or all. So sometimes it’s enough to have 20% of application that is high quality. Sometimes it’s closer to 80. It doesn’t need to be always 80. I mean, it can be 60, let’s say, or 90. But you know, the idea.

Miłosz [01:06:34]: And maybe the opposite of high quality is not low quality sometimes, but just “good enough” quality, which might be a tricky balance. But basically the idea is that you start with the happy path. And at least for the first iteration of a feature, you don’t need to implement everything. You don’t need to cover all edge cases that can happen, because there’s very often a very long list of them. But you can focus on this 80% of important, often used features and implement them good enough. And of course, there’s always this other factor here that if you don’t cover the edge cases, then you have to support them manually very often. And it often makes sense because I remember we had projects where we skipped some manual support for the most weird edge cases, because even our stakeholders didn’t know how to handle them. It was so complicated. So why spend weeks of developer’s time automating it if it happens very, very rarely?

Robert [01:07:50]: In other words, if you’re automating something, think if, for example, you are not spending a week on automating thing that it’s taking, you know, five hours a year.

Miłosz [01:08:01]: Eexactly. I had a project like this before. And when I calculated it was a dumb process that we had to do every month, but when I calculated it, it took two hours over the year for one person. So it feels dumb that you have to do it manually, but then how can you do it? How can you automate it so it makes it worth to save two hours a year?

Robert [01:08:33]: I think I have a good example even from today. So the downside of developing our training platform after hours is that we didn’t have time for developing very basic features, like we didn’t have feature for changing your email address. I know it sounds silly, but again we did most of this time over weekends and after work and we were just cutting all functionalities that were not possible. And okay, you can think what can be hard with changing email address. And yeah, I had the same idea this morning. So I thought, okay, it’s super simple, let’s even try to use Claude Code for generating this. Four hours later. I found that there is a bit more complexity for that. So we don’t have actually that many requests to change email address over the year. But I already spent those four hours and it’s there with some shortcuts done anyway. But it’s often like that you have some dumb thing that you are doing rarely, but you think that, okay, automating that, it will just take one hour and one week later, you see that, okay, it was a bit harder. So I just saved my minus four days and eight hours.

Miłosz [01:10:01]: But on the other hand sometimes the small issues compound. And you may have, you know, you might spend a bit of time on one issue a year. But if there are 20 issues like that, you may find yourself doing support for full time. So again, balance. You need to pick what needs to be fixed now and postpone what doesn’t need to be fixed right now.

Robert [01:10:38]: I think it depends a lot on the company culture. So the thing that I mentioned a bit earlier. So in startups you can probably live for a while with spreadsheets, CSVs, and, you know, your company is based on spreadsheets and CSVs and it’s fine for the beginning. After you are scaling, it’s probably good to maybe keep track of that, of how much time you are spending on such stupid things and balance that.

Miłosz [01:11:18]: I guess it’s good to set some expectations also on yourself so you don’t feel bad about having some low quality code somewhere in the codebase. And I remember feeling bad, especially when hiring new people. You know, you have some parts of the project where you don’t want them to look, but it’s completely normal, right? Most projects have parts like that, especially if you have some legacy software that has been there for a long time. So I think you also need to accept that some parts of it will be there. And be careful about this mindset, as you know, things need to be nice and tidy everywhere, which I think is common among developers. I also feel it sometimes that, you know, I would like things to be nice and tidy and all the corner cases covered and all the TODOs fixed, but very often it’s not really useful to to do it.

Robert [01:12:35]: And one more thing. If you are joining some new project and you see some old code that you think is bad, try to maybe not blame previous coders because you don’t know actually in what conditions they needed to work.

Miłosz [01:12:50]: Sometimes it’s hard.

Robert [01:12:51]: Yeah, yeah. How long they needed to spend on some code. So for example, I have an example that I worked in one project and later I was talking to one person that was complaining a bit why it’s that bad. And later I added a bit of context that we written this code in probably two days and didn’t touch it mostly later because we didn’t have time, but it worked good enough. So it’s like it was not our bad will to write something badly. It was just we’re doing some hackathon or whatever and we just implemented it quickly. It worked good enough and that’s it. And yeah, I think it’s also making your life a bit easier, if you will just try to assume that your previous colleagues that are no longer with you did your best. Especially that I also seen it multiple times that after some person is leaving, some people are tending to blame this person that oh, okay, this is bad because of this person. And you know, truth is usually a bit more sophisticated. And often those people that are blaming this person that left, they’re also not without any blame. So, you know, watch out for that.

Miłosz [01:14:09]: Now that you mention hackathons, it’s also an interesting topic to think about. Like how do you run a hackathon and then not end up with a pile of garbage, which we did a few times. It’s interesting. But that’s an extreme example. Most of the time you probably don’t create a new product in three days.

Robert [01:14:36]: But again, I think in general the idea is good because sometimes it’s allowing you to really cut corners, because you need to cut corners because you have some deadline to do that. And the outcome I think sometimes can be really interesting because you can go much, much further than you think that normally you wouldn’t be able to do that because sometimes it can even take months because of some stupid conventions. And maybe at the end you will find out that, okay, maybe actually we don’t need to do that because it’s no value in that. So yeah, I can really encourage you to run some hackathons if you are able to do that and, you know, try to not do hackathons that don’t care about functionality, to not use any bad words. And yeah, I think it can be an interesting exercise, but it’s also, again, important to clean it up in places that we mentioned a bit earlier.

Miłosz [01:15:37]: But sometimes, the high quality, low quality difference is not that big. I mean, high quality sometimes doesn’t require so much effort, even for a hackathon, right, for example, like using the repository pattern or even clean architecture maybe.

Robert [01:15:59]: In some lightweight form.

Miłosz [01:16:02]: Yeah. It’s not like a huge investment upfront. And once you do it, it’s easier to manage the project. So even for hackathons, you could use some of the more advanced patterns, let’s say. And it won’t slow you down.

Robert [01:16:25]: And the thing that we mentioned a couple of times. So try to be mindful about code quality and remember that it’s something that you need to learn with building over time. So after you build a lot of projects you see better where you can cut corners, where you shouldn’t cut corners. So during this episode, we did give you some hints. But again, I think practice, practice and practice, it’s very important. And it’s also fine to be wrong here. So even if with Miłosz we have pretty lot of experience, sometimes we’re also wrong in deciding if we should go with higher or lower quality. And it’s fine. I mean, as long as it’s not some decision that is not reversible, it’s fine. So I think there are a couple of questions that it’s worth asking. So if you are thinking about code quality. So the first thing is that something that we mentioned at the beginning. So if this code is more an experiment, or we know that, okay, we already have customers there or it will be potentially sold to some enterprise customers that will need to just make it working properly from the day one. So it’s good to start with that.

Miłosz [01:17:46]: It will also help you decide if you want to use an MVP or not for this project. And a similar question is if this project is mission critical, if something terrible will happen, if it goes down for a couple of hours, let’s say. As I said before, we don’t like to think our code is not important, but maybe at least parts of it are not.

Robert [01:18:18]: It’s also good to think, okay, what will happen with that after we develop that. Maybe this is feature complete part and will not touch it in the next year. I know that, okay, the chance that we will touch any code in five years is probably not that low, but maybe in one year what’s the chance that we will be touching that. If we will not be touching this code in one year, we assume we never know, but we can have some assumption. Maybe we can cut some corners there because if we will not be touching that what it will give us if we make it high quality.

Miłosz [01:18:53]: And if you touch it, then it’s good to have some context of the approach that we used. So for example when refactoring our Academy platform it was useful to have this context in the comments or the todos that scattered it. And they don’t look nice, but it was super helpful for me to know okay, that’s why we took this approach. But it also doesn’t mean that you have to get rid of all the TODOs. Sometimes it just, you know, it just exists there, and it’s fine. We don’t need to to worry about it. Because if you don’t change it over time, what’s the issue?

Robert [01:19:41]: Like prehistorical paintings in caves?

Miłosz [01:19:46]: Yeah, kind of.

Robert [01:19:48]: I love those comments in the code like, “it shouldn’t be like that. We should fix it ASAP.” Git blame: five years, ago and everything is fine.

Miłosz [01:19:59]: It mostly shows that it wasn’t that important maybe. And I remember having some time in the sprint when we are out of work and it’s time to look through the backlog. And I remember looking at the technical debt issues that we added over the years before, months before, and we always complained there was no time to do them. And finally there is time. And I looked at them and I realized I don’t want to work on it because it doesn’t seem important. So then you can just delete them and don’t worry.

Robert [01:20:43]: And if you did your dirty POC, don’t miss the time when you should clean it up, because it’s easy to go into spot when it’s no longer possible to do that. And I think it’s very nicely visible in some products that it’s visible that they were able to develop it quickly at the beginning. But you can see maybe three years later that they were not able to add any single feature to this product because it’s so hard to touch. Everybody is so afraid to touch and change anything there, and it’s just slow death for the product. And it’s already too late because it has too many functionalities that it’s hard to replace and it’s maybe just not justifiable job. You may not justify to rewrite it from scratch.

Miłosz [01:21:42]: It’s probably worth to keep in mind also that code quality is not the end goal. It will help your projects succeed over time, especially in the long run. But be careful not to make it the metric, to not care too much about the quality itself, which is hard to define in the first place.

Robert [01:22:06]: I know that there are many people that are thinking like, I don’t care if this project will be successful. I just care about money. So if you are this person, right, it’s fine. But if you are trying to find a well-paid job, it’s much easier if you have some success stories. So you just didn’t work in the projects that were mediocre or failures because, well, the market is pretty hard now, and I was interviewing many people in my life, and it’s super easy to see if people really worked in their projects that were successful, and they had some positive impact on how those projects were delivered and how they succeeded. And, you know, if you are just working in the projects that “I don’t care if it’s successful”, really, it will be much harder for you. So if it’s you, really, I highly recommend to try to care, even from a selfish perspective.

Miłosz [01:23:06]: Probably also watch out for some extreme tips like “never do X”, “always do Y”, because most of the time it’s not so simple. There are more subtle corner cases and context you need to understand.

Robert [01:23:28]: I think there are multiple dimensions of parameters let’s say that are affecting that. So if you are listening to us from the beginning, so you see that just code quality can have that many dimensions and you see that out there there are a lot of people that are saying that, oh, you don’t need to care about code quality, or code quality is super important for how fast you’re delivering. So you can see that it’s not that easy. And I think it’s just showing that a lot of people that didn’t probably work in many projects or didn’t work mindfully in these projects may not give you the best advice.

Miłosz [01:24:07]: It’s hard to boil it down to some very simple rules. I think it’s quite lazy if someone tells you something like “just just keep it simple”. Yes, of course, I want to keep it simple, but you can’t just take a 12-person team and tell them, just keep it simple guys, it will be fine. So there’s many, many different ways you can approach any project.

Robert [01:24:38]: Check good sources, some good live podcasts, for example.

Miłosz [01:24:46]: Okay, I think we are nearing the end.

Robert [01:24:48]: Yes, I think we should be able to go into our Q&A now. So sorry for not looking for the chat, but our multitasking abilities are limited. So let’s see what questions we have on the chat. So we’re mentioning a bit of our learning platform. So it’s a proof that we have a lot of fans of how it works. And yeah, how the strategy of cutting corners at the beginning and fixing it later, it works. So yeah, we have “Go Event-Driven was really good. Highly recommended for everyone to check out.” That’s true.

Miłosz [01:25:32]: Thank you.

Robert [01:25:35]: All right. So time for the first question. Do you want to take it?

Miłosz [01:25:39]: Okay. So Rohit asks, “do you think that careful design decisions maybe at low level would avoid refactoring later on? Sometimes I have made decisions that to priority addition of changes over code quality. Is that normal?” Maybe that’s the question whether you want to start with a POC and do something dirty and then figure it out. Or do you want to create some proper architecture from the ground up.

Robert [01:26:21]: I think it’s probably important to set some expectations here. So I think it’s good to spend some effort on design because if you always have some design so there is no that you have no design. If you will not make design you will have bad design. But it’s not that you will not have the design, but if you try to do your best, even spend ages on the proper design you probably will also not do it totally right. And it’s fine because you have some assumptions at the beginning how it will work, but it will change over time when you’re implementing stuff, and at some point you will end up with need to refactor something and it’s totally fine. So like we mentioned example of our platform for trainings. So our initial idea was totally different. And at the end we ended up with something different and design wasn’t perfect, but we were able to adjust it with the time.

Miłosz [01:27:27]: I think the part I would be careful about here is the avoid refactoring later on. So this is not something you want to aim for. You don’t want to make something perfect now, so you avoid refactoring later because it will anyway happen. So rather better to adjust as you go with some small improvements rather than try to come up with a perfect solution that will cover all future use cases. That usually ends badly.

Robert [01:28:01]: Remember about the question that I suggested at some point. So I think design is the most important for things that are hard to change and to know if something is hard to change. Ask a question. Is it hard to remove? If it’s hard to remove, probably it’s good to think about the good design or thinking if you need to redo it in this way. I hope we answered your question. If not, please leave a follow up on the chat. Yeah. Second question is a bit simpler. So, “hey guys, will you be releasing on Apple Podcasts later?”. Yes, we’ll release on Apple Podcasts, but we would like to have three episodes done, so it will take more or less one month will be releasing it to another three platforms, so it will stay just on the YouTube for since we will release two more episodes and later it will be on Apple Podcasts, Spotify or whatever application for listening podcasts that you are using. It will be there. If you are in our newsletter. We’ll also send a summary tomorrow. Tomorrow? Yeah, I guess tomorrow. We’ll send summary, transcript, and all this interesting stuff. Another question. “Is hexagonal architecture worth in the real world?” What do you think, Miłosz?

Miłosz [01:29:25]: I want to say yes, but.

Robert [01:29:28]: There is no silver bullet!

Miłosz [01:29:31]: But I can tell you that, we used it a lot in many different projects and teams. Hexagonal, clean architecture or whatever you want to call it. And one thing I noticed is like this. This is the pattern that many former colleagues tell us about that they introduced in their new team. And the new team members also love it. So I think it shows that’s something developers like working with. And we do too. But of course, if it’s a simple POC or some scrappy MVP, you can probably skip it. But then also it is not a big effort upfront. At least, maybe that’s just my perspective because I’m used to it so much.

Robert [01:30:28]: I think it requires more effort if you are doing it for the first time. So I will definitely not do it for the first time. Especially that I think clean architecture is sometimes kind of the trap. So people are trying are overcomplicating that.

Miłosz [01:30:44]: And we will be talking about it in four weeks, so I think we will cover it more in depth, but I think it’s one of the most useful patterns we use overall. And also not that complicated. And it’s not even about the exact approach, but just having some layers, splitting the important code from the less important code.

Robert [01:31:10]: TL;DR yes, it’s worth.

Miłosz [01:31:15]: Definitely in the real world it is useful.

Robert [01:31:19]: It’s just important to not over-complicate that, unfortunately a lot of people are doing and okay, also what other people are doing is applying clean architecture without understanding that. So it’s leading to a complicated code.

Miłosz [01:31:35]: That’s the issue with many patterns. It’s very hard to understand what someone means by clean architecture. do you remember the rant on Reddit about how clean architecture is the worst thing to happen in Go? And you know, I then read the thread. And basically the author said that you shouldn’t use more than three layers.

Robert [01:32:05]: Instant question. So how many layers you are using there? TEN?

Miłosz [01:32:11]: That’s the general issue with naming. I think it’s similar with Domain-Driven Design.

Robert [01:32:16]: Maybe they are combining clean architecture with hexagonal and onion and using all layers from that.

Miłosz [01:32:24]: It’s very hard to understand what someone means by one of the acronyms used in tech.

Robert [01:32:32]: Long story short, we’ll touch a bit more on that in the next episode. Please remember to subscribe on our landing page of the podcast so you can. You’ll not miss that. All right. Next question. “That’s actually how I’m developing first version, I write the POC, then I do the refactoring in cycles and until I’m happy with the story the coding is telling.” Yep, it’s a good approach.

Miłosz [01:33:04]: That’s iterating

Robert [01:33:09]: Yep. Okay, the next one. “Is it better to refactor or code review every certain interval? Because in startup environment almost no one wants to add review as their task.” I think that this is interesting question.

Robert [01:33:29]: So what’s my experience in a similar situation is that if you have problem with people doing your code review, it’s probably something wrong with team responsibilities. If in the team. I mean, if people are kind of working on things individually, not like a team, it’s a problem because it’s not the team’s responsibility to deliver something, but it’s individual responsibility to deliver something. And from my experience, this is the root cause because, well, if you are responsible for your feature, why you should review other people’s work, you kind of don’t care. And if you are doing more team work, so you’re planning the work as a team. You’re implementing entire feature as a team. You know, it’s interest of all of you to deliver it faster. But I’m not sure.

Miłosz [01:34:25]: But, you know, I think the question is more about the interval. So the way I understand this, should you do like two sprints of product work and then one sprint of refactoring or review? And I don’t have good experiences with this approach of this, you know, like dedicating entire sprint to code quality. I don’t remember it ending well, at least in the projects I was in. So I don’t know. What do you think, Robert? I think it’s better to iterate. So just do small improvements as you go.

Robert [01:35:14]: Yes. There’s also one thing in this question that I think it’s also pretty relevant.

Miłosz [01:35:18]: Oh, unless you do what we did. So we actually had to dedicate this one iteration pretty much to changing, to refactoring the code base so you can start working quicker. So sometimes this “go faster” initiative also can work. I would just be careful with stuff like okay, maybe it’s about if you know what you want to do beforehand. So if you know we have this issue with the CI, let’s focus one week on improving the CI. Then it sounds pretty good to me. But if you want to do something like okay, we worked six weeks on the product. Now let’s take two weeks to fix the TODOs. That sounds like more tricky part because you don’t know what you what you want to solve. So I would try to, you know, first understand what you want to fix.

Robert [01:36:21]: And there is one more thing that, so please let us know if we’ve answered this question properly. We’re after ten hours work day and it’s almost 8 p.m. here in Poland. There’s also one more thing that may be worth mentioning here. So for some hardcore refactorings. So what we also recommend for that is mob programming. So you are gathering with the team. One person is coding and you’re kind of helping this person. I will not cover how mob programming works, but I think that not a lot of people know about that. Worth checking, especially for harder refactorings because it’s easier to kind of synchronize with the work that you are doing with the entire team. I know that for some people it’s like, oh, we’re wasting entire team’s time for this. And it would be easier if one person will work on that. But for other side, somebody needs to review that. And the good thing about mob programming is that you have real time review. So everybody is looking on that and you are kind of doing a review together with them.

Miłosz [01:37:27]: The bad part is that it promotes looking for consistency. So you have to be very careful about cutting offtopic discussions or nit picking of how something works.

Robert [01:37:43]: But I think it’s good for things that let’s say you will be working on for a week and it wouldn’t be possible to do code review of that, because sometimes it happens that you have that big refactoring that there’s no way to do code review of that. If you are doing mob programming, there is kind of no need of doing code review. All right. So I hope we answered. If not, please let us know. All right. So the next one Miłosz maybe you would like to pick up this question?

Miłosz [01:38:14]: “It is always good practice to keep your code at a minimal level without looking for perfection, but at least enough clean to be comfortable to give it to someone else without being nervous about it.” Yep, sounds like a good baseline level. Like, don’t don’t stress too much about perfection. But you know, take effort. Yeah I think that’s actually a good question to ask yourself. Am I comfortable giving it to someone else and having them work on it? Would I be ashamed of myself. I used to, you know, half-joke about this, about our platform, that if we hired someone to work on it, our reputation would be over because they would see all the TODOs in the code. Yeah. So that’s pretty good. Good idea.

Robert [01:39:15]: All right. And so time for so far, the last question. If anybody has more questions, you have still some time. So “how do you measure that you’re overthinking and overengineering.” So I’m not sure what you think about that, but I would say that what I personally like is I think that most of us doesn’t like, but it’s sometimes also useful. So having some timebox or deadline. So if you’re timeboxing your work for some time, or giving yourself some deadline, or maybe some days doing it externally. So it’s kind of forcing you to not overthink things because, okay, you have some time to do that and you need to focus on thinking on things that really matter. And I think I noticed it for the first time when I had some external deadlines that I noticed that, okay, I needed to refocus on things that did matter and other things I didn’t have time to run it. Again, I know that we don’t like deadlines, but this is the small, useful part of them. Definitely. I’m not sure what’s your take on this?

Miłosz [01:40:29]: I think it’s hard to measure yourself because we don’t like to think that we are wrong about something. But what I like is cutting scope. And I think it’s something we did this week, actually. And when we we have a new project going on and we had an initial idea how to do it and with like a brainstorming session and then we thought, okay, let’s now remove everything that is not absolutely required to make it work. And let’s just finish this v0.1, deploy it and see if someone uses it. And very often you later realize, oh, actually, this is good enough. Maybe we didn’t need all this complexity. We initially planned for. Or as Robert mentioned before, with our platform, we had much more complex setup in mind, first with CI connected to it and all of it, and then we stripped it from all this and focused on the critical path. And that usually helps because you can see that maybe you don’t need all this.

Robert [01:41:55]: Also having some kind of MVP, maybe connected with some deadline that you’re putting yourself on that doesn’t need to be that strict. But again it helps. And I think what also helps is to have some end-to-end part working. So for example now we are working to adding some AI on our platform. But also I think we also have some nice idea how to have it working end to end with some limited functionality. And we’re not overthinking that that much because okay, we’re focusing on the simplest part and we can start to overthink it later after we’ll see that it was working. Because the problem with overthinking is that you can overthink things, that you later find out that it’s not useful, and you need to overthink another thing.

Miłosz [01:42:45]: Sometimes it also helps to have someone else who can tell you, you know, maybe you just want to use this AI API or this database. Maybe you should do it in your side project, not in this one. You know, like someone who can tell you if you if you go too far, it should be easier for someone else. But then you need high levels of trust to accept this.

Robert [01:43:20]: Okay. So I think it would be it for today so we can finish up. So if somebody doesn’t know us yet. So I highly recommend to check our blog. I don’t know if you can see QR code. It’s not that big. Let me try. We’re using this platform for the first time, so let’s do it like this. Okay. It’s not so big.

Miłosz [01:43:54]: True MVP.

Robert [01:43:55]: This is QR code with our blog, so highly recommend to check this out so there is more content there that is mostly about Go. So if you know Go it may be interesting for you. If you don’t know Go, it may be also interesting for you. If you liked this live podcast episode, fun fact is that this is the first episode that we recorded ever, so I hope that you liked this. So if you liked this, I highly recommend you to subscribe to us on YouTube so it can reach more people. And, you know, you can be also sure that you are not missing any new episode because we are already planning a couple more. Oh, I see that actually a lot of people just joined us because it was 70 people and now it’s 94. Hello for everybody who just joined, we’re finishing, but you can scroll to the beginning to check that. So yeah, if you would like to also to support us, please let all your friends know about that so it will help us to release new episodes. What else? So we already have a couple more episodes planned. So the next one will be why you shouldn’t use frameworks in Go. So if you are writing Go, it may be very relevant for you. If you’re not writing in Go, it may be also relevant because I think it may be pretty universal advice. The next episode will be Is Clean Architecture Overengineering. So we covered a bit about that earlier, but we’ll go a bit deeper in this episode.

Miłosz [01:45:50]: That’s a broad topic.

Robert [01:45:52]: Yeah. After that, we’ll cover how we believe you can learn faster. So we also covered a bit of that today, but we’ll cover definitely much more. Also if you’d like to not miss an episode we totally recommend to subscribe to our newsletter on our blog, because YouTube may not send you notification or Twitter may not send you a notification. In our newsletter, you will be sure that you are receiving the notifications so you will not miss the episode. And yeah, if you have any comments questions and you are listening it a bit later, leave us a comment on the YouTube. We’ll read that. And if you have anything that we should include in the future, please write down and we’ll try to include it in the next episode. Please also give us a thumbs up if this is already uploaded to Spotify, Apple Podcasts, or whatever, please also leave us a five star review so it can reach more people. Something to add Miłosz?

Miłosz [01:46:57]: Thank you, Robert, for meeting me today.

Robert [01:46:59]: Yeah. Thank you.

Miłosz [01:47:00]: See you in two weeks.

Robert [01:47:01]: Yeah. See you in two weeks. Thank you.

Miłosz [01:47:03]: Thank you everyone. Bye bye.

Robert [01:47:05]: Bye bye.

Iteration #1: "Go Faster"

Wed, 26 Feb 2025 00:00:00 +0100

We registered Three Dots Labs as a company in 2014, hoping to quit our jobs soon to work on our products. The “soon” turned into ten years of launching side projects and working evenings and weekends. We finally decided to take the leap last year, and we plan to share how it’s going. If you’re in a similar spot, you may find it helpful.

By the end of 2024, we spent some time planning the next year and our approach to work. We reviewed our backlog and discussed what we wanted to do next. We’ll experiment with 11-week iterations, with a week off in between. It’s not like the usual sprint — it’s more a way to stay focused during this time.

We’ve just wrapped up the first iteration, and you’ll find the summary of it below. Our main goal was to get ready for the next iterations. There were many things we wanted to wrap up before starting new projects. We also gave ourselves time to chill a bit after working two full-time jobs for a long time.

The main theme was “go faster”, so it’s mostly housekeeping stuff. We decided it’s important to do it now because there are just two of us, and we can’t afford to waste time during development. Our ultimate goal is to create a place where we love to work, and frustrating things like slow CI or a messy codebase are real motivation killers.

What we do now falls roughly into three categories:

Academy (our learning platform).
Watermill (our open-source library).
Content (blog posts, podcast, etc.).

Academy

We had an idea for hands-on programming training a few years ago, but we weren’t sure if people would enjoy learning like this. We created an MVP to see if it’s technically possible. Go in One Evening was the first training we created.

The project started as a scraped-together prototype that we iterated on to improve the UX. When it gained traction, we built more features and created another, much bigger training. We did all this while still working full-time, so we struggled to find time and never bothered to refactor the project. We ended up with quite a low-quality codebase that made the most money out of all our products. (I used to half-joke that if the codebase ever leaked, our reputation would be over.)

If you ever worked on a professional project, you know it’s an extremely common scenario. There are projects with great technical quality that no one uses, and they die. And there are terrible big balls of mud that make millions. But having technical debt is fine if you take time to clean up some of the mess.

For us, it’s an interesting place to be in. When you work as a developer, it’s easy to complain about not having enough time to do things right, while the business needs to move fast and make money. Now, we feel both pressures at the same time. We need to move the product forward but don’t want to get stuck working with a messy codebase. The frustration quickly adds up.

We finally have time for things like this, so we focused on them during the last few months.

Moving to a monorepo and updating the CI

Our CI got slow and fragile over time, and as you may know, it’s one of the most frustrating things to have around when working on a project. It’s also your main feedback loop when committing code. You waste precious time if you have to wait or fight to make the CI pass.

While cleaning it up, we also decided to move to a monorepo, which makes sense when only the two of us are working on it. We ended up with one repository that keeps all of our active projects (including this blog) and a set of GitHub Actions that run only when the relevant project changes.

It went smoothly, apart from the regular experience of testing the CI (commit, push, wait, read the error, repeat). It also improved the developer experience right away.

Various CI workflows in one repository.

The pipeline for Academy.

As you can see, the test times could still be improved. That’s mainly because of the cache and dependencies.

The best part is that we can now deploy just what has changed. For example, changing something on the website won’t deploy the backend, and vice versa.

Migrating out of Firestore

We started Academy with Firestore as the database. It’s easy to start a project with it, as there’s no need to set up any infrastructure. It was also about 10x cheaper for our use case than hosted SQL.

But working with the document model is far from ideal for typical web applications. You can’t just run a quick SQL query to get the data you need among all collections. Migrations are painful, as you must write code to transform the data. And the emulator used in the local environment often has its own issues.

So, we decided to migrate to Postgres.

Do you know how useful it is to hide your storage code behind an interface in case you want to change the database? We promote it a lot, but we didn’t care that much for the MVP. Some people claim you never change the database, so it’s not an argument for using interfaces. But we had to deal with this exact use case.

First, I changed the code to use the clean architecture pattern. Thanks to this, I could test both implementations with the same tests, and changing between databases was a single config change. It’s super helpful to be able to quickly roll back a risky change like this. It also helped to clean up the code from a few hacky patterns we used in the MVP.

Still, the change wasn’t as easy as just replacing the implementations. Migrations are always more complex than they seem. I spent about two weeks finishing everything. The refactoring alone took a few days. Then, I had to set up the infrastructure, write the migration script, run tests, and debug issues. We also keep two regions of the platform (EU and US), so it was another factor to consider.

It’s a good example of a tough refactoring decision since this change did not improve the product in any way. In this case, we agreed it’s worth it for the long-term benefits. Small issues add up over time, especially if it’s something as critical for the project as the database.

Jumping around the exercises in the CLI

We added a minor but often requested feature to our tdl CLI tool. You can now jump to another exercise you completed before.

Watermill

Watermill is our open-source Go library for building event-driven applications. We released Watermill 1.4 last autumn but didn’t wrap up everything we wanted to.

A few PRs needed our attention, like #25 in watermill-kafka. Often, a PR looks like a trivial change, but then it turns out to be a rabbit hole. In this case, we had to adjust the tests to make the changes work, which was a solid day of Robert’s work.

Another example of tough dilemmas is the issue #402. We often don’t foresee all use cases when adding a new feature. Then, we must decide whether to keep the current behavior, even if it’s incorrect, or change it and potentially break someone’s code. Together with the fix, Robert updated the CQRS examples to show the best way to use the component.

We deprecated the gogo/proto marshaler and introduced a new one, cqrs.ProtoMarshaler. It was long overdue, as gogo/proto has been deprecated for a long time.

Finally, watermill-sql v4 is still a release candidate. New issues are coming in, and I’ve been struggling to stabilize the tests with the recent changes.

Maintaining an open-source project often feels like a full-time job, and we can’t spend all our time like this. At the same time, we want Watermill to stay independent of external sponsors so no one tries to monetize it, which would probably push it towards a vendor lock-in approach.

On the other hand, it has a nice synergy with our Go Event-Driven training, so we have a good reason and motivation to keep improving it. We’ll try to find a reasonable balance here. As always, big shoutout to all contributors who help us push the project forward.

Content

On our blog, it was a relatively quiet end of the year. I released The Over-Engineering Pendulum, a less technical post about my experience with over-engineering in startups. We want to share more on similar topics in the future.

Last year, we ran a couple of Watermill release parties and live Q&A sessions for Go Event-Driven trainees. We liked the format, and we wanted to experiment with it more. During this iteration, we started preparing to launch our live podcast, No Silver Bullet. You can read more on it in Robert’s post.

We have a long list of topics for the blog and the podcast. After long sessions of sorting them, we ended up with close to 150 items.

Our newsletter is the primary way to communicate with our readers and also how we sell our trainings. We’ve been using ConvertKit to send emails, and it worked quite well, but the automation workflows were a huge pain to work with. We had to do some nasty workarounds to make sure the ebooks were delivered reliably to newsletter subscribers. The product wasn’t improving, so we moved out to avoid spending more time on it.

This wasn’t the first migration we made. A few years ago, we moved off MailChimp for similar reasons. This time, we moved to MailerLite, which seems to be a better fit for us. It took about two weeks on Robert’s end, but hopefully, it will save us headaches in the future.

Plans for the next iteration

Looking forward, I hope we’re done with migrations for now. 😅

We have a few goals for 2025, but we try to not plan too far ahead. Often, we get a new random idea and want to pursue it right away (that’s how the Academy started).

We want to experiment with the live podcast episodes and see where they lead. Another experiment is adding new features to the Academy backed by LLMs. We have some ideas that should turn out more interesting than the usual “we now use AI chatbot as customer support” stories. Our main goal is to help people move through the training more smoothly.

We definitely want to release another training in the coming months. We don’t fully know the scope yet — we’ll ask our newsletter subscribers for feedback soon.

Another goal is finishing Go with the Domain, and releasing a revised edition. It’s probably what we get asked for the most, but we’re not sure about any dates yet.

Thanks for reading! I hope to see you live during one of the No Silver Bullet episodes and on the Iteration #2 summary.

We quit our jobs to help people write software more mindfully

Sun, 19 Jan 2025 00:00:00 +0100

Leaving a principal software engineer role while having a newborn kid, a mortgage to pay, and a house being built may not sound like the best idea. Still, I took a significant pay cut so I could make a living by educating people about software. Some people literally told me that I’m crazy.

Well, Miłosz (the co-founder of this blog) and I decided to build a company we love to work at without VC funding.

It wasn’t an easy decision. Finances, of course, are always a concern. Also, we may not have as much time to work after hours as we used to. Let’s also be brutally honest; we won’t be alive forever. Still, we have families we need to provide for, and we want to build a company we can be proud of.

Let’s start with how it all began.

Our Journey

I met Miłosz in 2008, when we were in high school. At that time, we started writing software together — after school and after work (sometimes, we even worked at the same company). For a decade, we have been working two full-time jobs.

Our after-hours projects were more or less successful. For example, in high school, one of our online game servers became the biggest one in Poland. We earned enough money so that Miłosz could buy a car, and we even threw high school parties. We were making good money for the time!

During that time, we saw how different software development strategies work in different scenarios. We learned how to cut corners and write code in the simplest way possible. For two-team projects, following “the simple way” was usually fine, but for bigger projects, we needed to learn more advanced techniques.

Starting this Blog

A couple years later, when we were leading and mentoring teams, we saw that some techniques were hard to grasp for our less experienced teammates. Sharing our knowledge to a wider audience sounded like a good idea. So, 7 years ago, we started this blog. Since then, we have created 39 detailed articles. A good example is our article about Live website updates with Go, SSE, and htmx. There are a couple other articles describing SSE in Go, but most of them are quite surface-level. We like to go extra deep to show real-life scenarios.

It’s great to see that there is a demand for in-depth content about complex techniques. In fact, every year, our blog is visited by around 270k people!

Watermill - Monetizing Open Source While Staying Open Source

Around the time we started our blog, we also worked on a project that explored how event-driven architecture would make the application faster and easier to develop. Unfortunately, most of our team didn’t have any experience with this kind of architecture.

At that time, we asked ourselves a question: “How can we make building event-driven Go services as simple as HTTP API?” This is how Watermill was born. It is now the most popular Go library to build event-driven applications (at least according to GitHub stars, since we don’t have a better measure).

You may be asking yourself: “Don’t you want to monetise it? Is it time to change the license and start charging people?”. That’s not our plan. We treat Watermill like a good marketing tool, and that’s it. We have also heard that Watermill is useful for people, and that makes us happy!

We are self-funded, so we can afford such extravagance. Truly, we don’t believe that potential income would be worth losing our community’s trust.

About 4 years ago (3 years after starting our blog), we released our book Go With The Domain. We already knew the content we were creating was useful. This was the first in-depth book about Domain-Driven Design in Go. We later heard from people that their teams used it as a guide on how to develop their applications. But it was still missing one critical piece.

You can read tons of books and watch hundreds of videos and take all sorts of courses, but if you don’t put into practice what you have learned, you won’t accomplish anything.* There is no shortcut here; you need to mindfully use things you want to learn and evaluate the tradeoffs and hidden complexity. The best way to learn is by working on real-life projects. Not TODO apps or calculators; such applications are too simple. You need to learn how to overcome problems you’ll likely face while building real-life applications.

Learning in the Age of AI

These days, everyone (including me and Miłosz) uses AI during the development. Learning practical skills is even more important now than ever before. AI is widening the gap between junior and principal engineers. Junior engineers now struggle to find a good job. Ironically, it’s hard to find skilled principal engineers who can build applications that AI can’t handle (and likely won’t be able to handle anytime soon).

Principal engineers can do their job faster, while junior engineers may spend hours mindlessly copy-pasting AI-generated code without actually learning anything. The demand for engineers who can do simple applications is decreasing, as AI can handle these jobs more efficiently.

If you are afraid of AI taking your job, you should find a way to make yourself indispensable.

Building a Custom Learning Platform

Learning by practice requires a lot of time and patience. You need to find a good project theme and proper materials. It may also be frustrating when you are stuck and don’t have anybody to ask for guidance.

To speed up the learning process, we built a learning platform that supports users through building complex, real-life projects. It’s not yet another platform where you watch videos and code in the browser. It’s a way to put what you’ve learned to practical use.

Building a learning platform from scratch requires more time than running an open-source project or a blog. We didn’t have a choice; we had to work on it during the weekends and on holidays. We have been very productive! People were surprised when we told them we were going to work full-time at Three Dots Labs. They asked, “Aren’t you already working at Three Dots Labs full-time?”

Friday night. 9:37 p.m. Time to party? Nope! Coding!

We created a CLI tool, which clones and executes code. Our CLI will clone exercises for you, then execute them. It will save you a lot of time. This is why the feedback loop is so fast:

A Platform is Nothing Without Content

Our first training on the platform is called “Go In One Evening.” Many people said, “No way! It’s not possible to learn Go that fast!” We love to build things that people don’t believe can be real.

We were leading teams of people who had never written a single line of Go in their lives. We saw for ourselves how quickly they became productive - so we know it’s possible. Data we have gathered confirms this.

Statistics on how long it took to finish Go In One Evening:
50% of trainees finished training in less than 7h, and 75% finished training in less than 12h

The agenda includes creating a simple social media platform clone and building the ls CLI tool. It’s designed for software engineers who know how to code already but want to learn Go. We don’t waste your time with how variables and functions work; you already know that stuff. Instead, we focus on things that will be essential in your work.

“Go In One Evening” is not our only training. We also have our flagship training, “Go Event-Driven,” where we teach how to build production-grade, event-driven systems. The core idea is the same: we teach by building real-life projects.

What people say about our trainings

I won’t spend much time describing how it works; I would encourage you to try it!

Financials

Last year, 746 people joined our trainings, translating to $129,741 in revenue.

While that may sound like very good money, it’s less than we earned at our jobs. Our income is not stable yet, so we don’t take it for granted. If you have a steady office job, you may not appreciate that you will always be paid for your work. It doesn’t matter if you have a bad month. In this business, however, you can work on a project for a year, earn $0, and waste your time and money. Hackernews is full of such stories:

Hacker News is full of “failure stories”

What Are Our Next Plans?

This is where we are right now. What are our next steps?

We want to continue doing all initiatives, like our blog and open-source projects. But we have a couple new things on our plate, including:

starting our own podcast,
doing more online trainings, and
sharing what we are working on (building in public).

That may not sound very original; hundreds of people are doing the same things. But there are a couple of things we want to do that are totally different!

Let me explain.

Why Do We Need Another Live Podcast?

If you go to the gym, you may hear the term bro science. Some guys will tell you things like, “Take this pill, and you can exercise less and still get bigger. Don’t be a moron who wastes time.” For less experienced people, that may sound appealing, especially when you hear it from some big guys. The same thing happens in the software world.

There are a lot of bros who give software advice that is supposed to be a silver bullet and make your life easier. I don’t accuse anyone of lying; they are probably sharing what worked for them. However, people should stop advising as if it’s the silver bullet for all problems and acting as though their solution is the only way to do things. Something that works for a solo developer will not necessarily work on a global financial platform. Social media algorithms make this problem worse because they favor polarization and extreme opinions.

You may ask, “Why should I care?”. Companies are good at rejecting people who don’t know how to solve company problems. If you don’t want to find yourself out of a job market, you need to learn how to adapt. I’ve been interviewing people for years, and I could easily tell whether someone is truly interested in solving real-world problems or just follows CV-driven development.

If you are ambitious, you probably prefer to work with passionate people as well. Joining such teams is not easy – they usually have a high bar to join them.

That’s why we’re creating a space where you can learn battle-tested programming practices that actually scale - from solo side hustles to complex enterprise systems. No bro science, just real-world experience.

Polarization in Software

Polarization is quite prevalent in the software community now. There are two very distinct camps out there:

Engineers: For them, technical excellence is important; they care about maintainability
Shippers: For them, technology is secondary; they care mostly about shipping as many features that customers can use as quickly as possible; they are known to take shortcuts

There is an ongoing war between those two camps. You often hear very different arguments from both camps, like: “It won’t scale,”, “Just ship it,” “That’s not how it should be done,” “You don’t need unit tests,”
“It won’t be maintainable,”_ and “Quality doesn’t matter; we can always rewrite it.”

Both camps have valid points. The problem starts when people are unable to see the other side. Often, Engineers focus too much on technical excellence, and they may not be delivering anything usable for far too long. On the other hand, Shippers, after releasing a successful MVP, may wade into technical solutions that won’t work for bigger teams or products.

Can’t We Do . . . Both?

We should be open to diverse opinions. Wouldn’t it make more sense to merge these two approaches?

It’s not easy to learn how to balance the pros and cons of shipping and engineering. This requires time. We need to learn how to build software, and we also need to understand how this software affects development. Miłosz covered this topic nicely in the The Over-Engineering Pendulum post. Even though Miłosz and I have been building software together for almost two decades, we still struggle with this balance.

NEW! 🎉 No Silver Bullet Live Podcast: A Software Deep Dive from Multiple Perspectives

No Silver Bullet ideally illustrates the message and unique selling point of our live podcast: showing multiple perspectives on which tools and techniques work in various scenarios.

Over the last 17 years, we have written software for various projects. From legacy companies to startups. From simple applications to global settlement platforms. Most of the time, we acted as hands-on engineers, but we used to be managers and founders as well. We want to share our unique, diverse experience with you. We want to teach you how to apply patterns and use appropriate tools for different projects.

Like always, we aim to push the bar even higher. We want you to be part of our show so you can comment or ask questions about the topics we cover. It’s a win-win; we can go deeper into topics you care about and improve our content.

In No Silver Bullet, we’ll cover more general software engineering topics. But Go-related topics are also in the queue!

Our setup is ready; we will go live for the first time at the beginning of March! We have a list of the first 5 episodes on our live podcast page. We want to go deep for every topic, so we will stream an episode every two weeks.

Don’t miss our live podcast episodes!
Join our newsletter with over 18k subscribers!

🔒 We do not send spam. You can unsubscribe at any time!

NEW! 🎉 Next Trainings and Training Mentoring

Trainings are the best way for Three Dots Labs to make money. We also know that these trainings give people the most value for their money. Our trainings are a great way to leverage your skills and, hopefully, find a better-paying job.

You may be wondering: “When will the next training be available?”

“When will training be available?” This is a question we hear on a daily basis!

Before we launch the next training, we want to finish our mentoring functionality, which will greatly help you learn these skills even faster. When you are learning something new, it’s important to be able to ask for help. This can also help you save a lot of time with problems that may be solved by someone more experienced quickly. With the current state of LLMs and our knowledge of your most common questions and obstacles, we can give you hints like we would sit next to you! I’m already sure that it will be one of the coolest features that we have.

We aim to announce the next training sometime in March.

Why We Are Focused on Go for Now

Go is one of the best-paid languages, according to the JetBrains State of Developer Ecosystem Report 2024. You don’t need a PhD to learn it. Compared to other newer languages (like Rust), finding a job as a Go engineer is much easier.

This year, we will remain focused on Go trainings. If you don’t know Go yet, how about learning a new programming language with the “Go In One Evening” training?

NEW! 🎉 Building in Public

We want to share our journey as we create a sustainable business that we love — and that you can benefit from. If you are interested in following us, we will share what worked and what didn’t work every 12 weeks. We’ll share information about our finances, and we’ll discuss what mistakes we made along the way. Miłosz is already working on a summary of the first iteration, which will be out soon!

Aim for Sustainable Growth

We aim to build a dream company that we genuinely love to work at, a company we don’t want to sell. We are realistic; we always keep a list of things we don’t like to do but need to get done. It’s all a matter of finding the right balance.

Often, the “fun things” are not justifiable from a business perspective. Since we don’t have any external deadlines, though, we can work on things that matter and make sure the products we are putting out are the best they can possibly be. We can spend this one extra mile to change good things into outstanding.

Like with our trainings platform — we spent a lot of time optimising code execution and deploying it in multiple locations to make it fast. In effect, you may feel like it’s executed locally. Would it change a lot if it were slower? Probably not, but it’s cool!

Will We Succeed?

Our unusual approach may seem naive to many people. But based on our experience, we believe we will succeed. We always love to question the status quo and blaze new trails. After all, we created Watermill and wrote Go With The Domain. Now, we are working on online trainings.

You can expect to see new articles on the blog, and we will continue working on Watermill.

Will this strategy work? What if our assumptions are wrong? Will we have to return to full-time jobs to pay our mortgages?

The best way to follow our journey is by joining our newsletter. We prefer this over social media, as we aim for a long-term relationship with you. Social media posts show up, then disappear. An email stays in your inbox forever. We are also independent of its algorithms and owners.

If you are already one of our 17k newsletter subscribers, please tell your colleagues about our live podcast!

Will you cheer us on? Do you think we will succeed? Let us know!

The Over-Engineering Pendulum

Tue, 17 Dec 2024 00:00:00 +0100

I used to picture my dream job as this: I work for an early-stage startup that recently raised a round. The business idea is promising, there’s much to build, everything seems possible, and we have time to prove the product-market fit. I join as the first engineer, lay the foundations, and pick the tech stack.

Can you imagine a better gig? Finally, you get to do things your way. No more arguing and no old code dragging you down. This time, the project will have a high-quality codebase and no technical debt. Starting from scratch feels great — I’m excited just thinking about it.

In the end, I joined early-stage startups after someone else had laid the groundwork. The upside was that I could see the mistakes from a safe distance instead of making them myself. I ended up rethinking and refactoring existing codebases more than bootstrapping them.

After a few rounds of this and hearing startup stories from friends, I was shocked at how similar startups are. (I love the irony — they try to become unicorns, after all.) One common issue stood out: Product teams often struggle due to the mess caused by over-engineering in the early days.

Once the team grows, it suddenly takes ages to deliver what was trivial before. Eventually, someone asks: “Why does it take so long?” The engineers are surprised and frustrated. It can’t be the codebase — it follows the best industry standards.

It’s extremely difficult for founders to notice this before it’s too late. Often, it’s a surprise even to the technical leaders.

How Over-Engineering Starts

You’ve probably seen over-engineering before. It’s a two-person team using ten microservices and obsessing over tiny performance tweaks where they don’t matter. Or adopting the popular but unstable technology and going big on infrastructure when a single VM would be good enough. Or reinventing solved problems and implementing them from scratch.

It’s fun to build complex systems and imagine that technology gives you the edge. But it’s rarely true — most software is boring. Only small parts of what you create are unique.

Still, there are reasons to think big. The founders have a grand vision for the technology, and they make it clear: We need to build for scale. We’re creating an operating system to disrupt our industry. We have to write the core software now to simply add features later.

Perhaps this leads to overthinking what has already been solved for years. If you’re building the next unicorn, why use one database running on one server? It doesn’t make you unique among other wannabes. If you aim for the stars, better design something that supports this imaginary future.

But you also need to deliver and validate quickly at this stage. You must keep the momentum as you build the product over time, so you fear getting lost in a messy codebase. You feel pressure to keep the standards high.

It’s tempting to over-engineer in the hope of avoiding this mess, but it only makes it worse. It takes the time that you could spend on actually helpful essentials. Who cares if you implemented a service mesh if your CI pipeline isn’t green? Do you need that custom framework if the local environment is awful to work with? How will horizontal scaling save you from unmaintainable code?

High Quality

Your challenge is finding the balance between over-engineering and building a messy prototype. It’s somewhere between moving fast and doing things right.

We tend to see speed and high quality as opposites. You need to be fast to stay alive. But what does quality even mean?

Engineers have strong opinions and vague ideas about quality. Some understand it as “I don’t like this code. It should use another design pattern.” They are religious about formatting. They moan, “Give us time to refactor.” It’s a naive take, far from product engineering. It uses quality as an excuse to boost the ego. Code shouldn’t be the end goal.

Another take is thinking of quality as flexibility. Building systems that are easy to extend and configure feels rewarding. It’s cool if you want to solve code puzzles for fun. But if you want to deliver working software, you need some constraints. Focus on what’s in front of you. Use a proven, boring tech stack. Stay away from creative or dynamic solutions. If you’re getting excited about building something universal to solve issues in the future, check if you could simplify it into something you can use now.

You likely won’t need to scale anytime soon. But if you invest in the basics, they will serve you well once you need to do it.

The Essentials

In the early days, I like to focus on the developer experience. Instead of creating a framework or platform that supports all possible use cases, I prefer to prepare a great environment for engineers to work in.

There’s much to build when starting from scratch, and new work quickly piles up. Good foundations limit time wasted on non-product things. They help you iterate quickly, which has real value compared to refactoring for code prettiness.

Above all else, frustration is the worst productivity killer. It can radically change an engineer’s mindset from “it doesn’t feel like work” to “it’s the last thing I want to do.”

My first essential pick is an automated build system (a CI/CD pipeline) that runs tests and static analysis, builds the project and deploys it to production — ideally, a modern one that is easy to configure. The same steps run on feature branches, so the main branch is never broken.

It doesn’t have to be complex. Make it run git pull if that’s what you need. It’s easy to set up, and the benefits will be there every day for all engineers. You can’t afford to have deploys blocked by broken changes or spend time discussing trivial issues a linter could catch.

Next, prepare a great local development environment. It should be easy to set up, work everywhere, and run fast. You should be able to run the CI tools locally with the same versions and results. Use live code reloading so it’s pleasant to work with.

Finally, create guidelines on how to do common things. Think of the building blocks from which you build the application. Assembling elements is easier than creating a universal framework. It lets you keep some standards without trying to predict all future use cases.

It’s more about healthy constraints than doing things right. You don’t want to discuss every week how to write an HTTP handler, call an external API, use database transactions, or run a function periodically. Decide on something and cut the discussions early so you can focus on writing the code that matters.

In any case, watch out for decisions that are hard to reverse. Opinionated frameworks make starting easy, but getting out of them can take long months.

Aim for the Middle

I like the pendulum metaphor: once we go to one extreme and get bitten by the results, we tend to avoid it at all costs and swing the other way. It applies to many things in life, including technology hype cycles.

For example, consider delivering software. A service is a way of deploying code, not an isolation unit. So, use modules for isolation. But be ready to spin up a separate service if needed. Sticking with a massive monolith forever is as extreme as starting with a distributed system. But somehow, we tend to swing from one extreme to the other.

Similarly, once we work with a messy codebase, we start paying extra attention to quality. Then, it’s easy to go too far in the other direction and over-engineer your next project.

I still find it challenging to find this balance. It’s a topic that leads to never-ending discussions with no clear answers. There’s one thing I’m sure helps: If you have seen the extremes, at least you know to aim somewhere in the middle.

Before It’s Too Late

When I hear of teams struggling with messy systems, it strikes me how a simple decision in the past could have prevented it. Once you’re there, migrating out is incredibly difficult. In contrast, building from scratch feels much easier.

I wish we had metrics to track this, but I’m skeptical if that’s possible. For example, achieving high test coverage is easy, but it doesn’t mean your tests do anything meaningful.

One thing I’ve seen work well is regularly shipping small features. The key skill is splitting the work into chunks that make sense by themselves. The naive approach is splitting by artificial boundaries, such as one feature per microservice or separate backend and frontend parts. Instead, aim for changes that are end-to-end complete but small enough to ship quickly.

Watch out for making it a target metric, though. It’ll encourage shortcuts in the code, pulling you towards the messy codebase.

Watermill 1.4 Released (Event-Driven Go Library)

Tue, 29 Oct 2024 00:00:00 +0100

It’s Autumn over here, and it usually means another release of Watermill! 🍂 It’s hard to believe it’s already been five years since the v1.0 release.

In case you’re new to Watermill, here’s TL;DR.

Watermill is a Go library for building message-driven or event-driven applications the easy way. Think of it like an HTTP router but for messages. It’s a library, not a framework, so you don’t need to change your architecture to use it.

Over the last year, Watermill grew to 7.5k GitHub stars and 55 contributors, just on the main repository. It supports 12 Pub/Subs including Kafka, Redis Streams, NATS, Google Cloud Pub/Sub, Amazon SQS/SNS, SQL, and RabbitMQ.

To learn more, see the documentation and examples.

What’s new?

Let’s see what happened in Watermill over the last year.

For the complete list of changes, see the GitHub releases page.

Hall of Fame 🏆

Big thank you to everyone who contributed to all Watermill repositories since the last release! 💪

Note

Side Note on Design

Watermill 1.0 was released in 2019, and the public API has been stable. I especially like the modular design, which allows you to combine low-level components like building blocks to create more complex features. You can see this in rather simple logic of components like FanIn, FanOut, or the Forwarder.

You can also notice this approach in the new features described below. We added a few simple components that work well separately, but you can combine them for more interesting effects.

Delayed PostgreSQL Requeuer

One common issue developers new to messaging have is dealing with errors. People often share an output like the one below, asking what’s happening.

Infinite loop of errors.

What you see above is a single message failing to process and being retried indefinitely. It blocks other messages, so it seems like the application is stuck in an infinite loop.

This is how most Pub/Subs work, and it’s crucial if you care about message ordering. But sometimes, you don’t want a broken message blocking the entire topic.

We used to work with Google Cloud Pub/Sub a lot, and one helpful feature it has is retrying messages. Once a message fails, you can make it reappear in the queue after a delay. Other messages are not blocked, and the failed message can be retried later.

It’s a helpful feature I missed in other Pub/Subs. There are ways to recreate this, but it’s not as straightforward. For example, you can use delayed messages with RabbitMQ, but you need a plugin, so it’s not compatible with all providers.

I thought it would be cool to have this feature in Watermill working out of the box, and here we are!

Watermill now supports a universal requeuer component. It works with all Pub/Subs, but for now, it also requires PostgreSQL to be around (this can be extended in the future with more implementations). The PostgreSQL is used as an intermediate poison queue where the failed messages go until they are retried. It also supports delaying the messages, so you can avoid this infinite loop of errors.

There are a few layers of how this works, and I’ll dive deeper into them below. Let’s first see the high-level API you need to get started. You can see the full example in the Watermill repository.

First, create the SQL requeuer. You need to pass a publisher to the regular Pub/Sub so it knows where to route messages.

delayedRequeuer, err := sql.NewPostgreSQLDelayedRequeuer(sql.DelayedRequeuerConfig{
 DB: db,
 Publisher: redisPublisher,
 Logger: logger,
})
if err != nil {
 panic(err)
}

Full source

In your router, add middleware provided by the requeuer.

router := message.NewDefaultRouter(logger)
router.AddMiddleware(delayedRequeuer.Middleware()...)

Full source

Finally, run the requeuer:

go func() {
 err = delayedRequeuer.Run(context.Background())
 if err != nil {
 panic(err)
 }
}()

Full source

That’s it if you’re happy with the default settings. The messages will be moved to the SQL poison queue if an error happens. After a delay, they will be moved back to the original topic.

Note

The PostgreSQLDelayedRequeuer is available in watermill-sql v1.4.0-rc.1: it’s still a release candidate. Read more below in the section on watermill-sql.

Poison Queue CLI

When working with poison queues, the main challenge is visibility into what’s happening. You need to know what messages are failing and why.

Watermill 1.4 ships with a pq CLI tool to help you check this.

go install github.com/ThreeDotsLabs/watermill/tools/pq@latest
export DATABASE_URL=postgres://user:password@localhost:5432/db
pq -backend postgres -topic requeue

You can preview the poison queue, move messages back to the original topic, or delete them.

Delayed Messages

Delayed messages can be useful for more than requeuing. Many scenarios involve delays: “Ask the customer for feedback in two days”, “Send an invoice reminder in a week”, etc. It’s not a complex logic to implement, but if you already use events, you can leverage it and have delays working out of the box.

Watermill’s new delay package allows you to add delay metadata to messages. The delay metadata does nothing by itself. You need to use a Pub/Sub implementation that supports it to make it work. For now, it’s supported by the PostgreSQL Pub/Sub with NewDelayedPostgreSQLPublisher and NewDelayedPostgreSQLSubscriber.

There are two APIs you can use. If you work with raw messages, use delay.Message:

msg := message.NewMessage(watermill.NewUUID(), []byte("hello"))
delay.Message(msg, delay.For(time.Second * 10))

If you use the CQRS component, use delay.WithContext instead (since you can’t access the message directly). You can also use delay.Until instead of delay.For to specify time.Time instead of time.Duration.

ctx = delay.WithContext(ctx, delay.Until(invoiceDueDate))

err := commandBus.Send(ctx, cmd)
if err != nil {
 return err
}

See the full example for more details.

PostgreSQL Queue

The Delayed PostgreSQL Pub/Sub is based on the new queue schema adapter.

It’s a simpler schema that doesn’t support consumer groups. In return, it allows you to specify a custom WHERE clause to filter messages by a custom condition, like metadata or payload fields. You can also enable delete on ack, so the table doesn’t grow with time: the messages are deleted after they are acknowledged.

sub, err := sql.NewSubscriber(
 db,
 sql.SubscriberConfig{
 SchemaAdapter: sql.PostgreSQLQueueSchema{
 GenerateWhereClause: func(params sql.GenerateWhereClauseParams) (string, []any) {
 return "metadata->>'urgent' = true", []any{}
 },
 },
 OffsetsAdapter: sql.PostgreSQLQueueOffsetsAdapter{
 DeleteOnAck: true,
 },
 },
 logger,
)

Requeuer

The Requeuer component is a wrapper on the Router, similar to the Forwarder, but simpler. It listens to messages on one topic and routes them to another, dynamically generated one.

You can combine it with the Poison middleware that moves failed messages to a separate topic. You can use the Requeuer to move them back to the original topic based on the Poison metadata. This is exactly what the PostgreSQLDelayedRequeuer does. It combines the Requeuer, Poison middleware, DelayOnError middleware, and PostgreSQL queue adapter and hides all this behind a simple API.

AWS SNS/SQS Pub/Sub v1.0

Using SNS or SQS makes a lot of sense when your services are hosted on AWS. Adding support for Pub/Subs managed by AWS was one of the most requested message brokers.

In the current release, we have added support for both SNS and SQS. Thanks to that, you can choose the tool that fits your needs better.

While adding support for SQS is not challenging, implementing a proper Pub/Sub on top of SNS is not trivial. SNS differs from other Pub/Subs: it’s not just a matter of subscribing to a topic and receiving messages. You need to create a queue, subscribe to the topic, create a valid access policy, and then receive messages from the queue. It also doesn’t help that AWS error messages are sometimes a bit cryptic.

The good news is that we already made this work for you, and you can use it out of the box.

newSubscriber := func(name subName) (message.Subscriber, error) {
 subscriberConfig := sns.SubscriberConfig{
 AWSConfig: aws.Config{
 Credentials: aws.AnonymousCredentials{},
 },
 OptFns: snsOpts,
 TopicResolver: topicResolver,
 GenerateSqsQueueName: func(ctx context.Context, snsTopic sns.TopicArn) (string, error) {
 topic, err := sns.ExtractTopicNameFromTopicArn(snsTopic)
 if err != nil {
 return "", err
 }

 return fmt.Sprintf("%v-%v", topic, subName), nil
 },
 }

 sqsSubscriberConfig := sqs.SubscriberConfig{
 AWSConfig: aws.Config{
 Credentials: aws.AnonymousCredentials{},
 },
 OptFns: sqsOpts,
 }

 return sns.NewSubscriber(subscriberConfig, sqsSubscriberConfig, logger)
}

As always, you can find a fully working example (including a local emulator so that you can run it in Docker locally):

Big thanks to @Mariscal6 for creating a big part of the implementation!

watermill-amqp: v3 released

We released a major version of the AMQP Pub/Sub. The breaking change was introduced in the PR #27 originally contributed by @tobiasjaster .

The BuildTopology method of TopologyBuilder was missing the routing key. We decided to introduce a params struct to make it more flexible in the future.

Upgrading

You don’t need to do anything if you use the default TopologyBuilder.

If you implement a custom TopologyBuilder, update the BuildTopology method. It now takes params amqp.BuildTopologyParams instead of queueName string, exchangeName string.

- BuildTopology(channel *amqp.Channel, queueName string, exchangeName string, config Config, logger watermill.LoggerAdapter) error
+ BuildTopology(channel *amqp.Channel, params BuildTopologyParams, config Config, logger watermill.LoggerAdapter) error

Inside the method, replace queueName with params.QueueName and exchangeName with params.ExchangeName.

watermill-sql v3 and v4

We introduced the v3 major version of watermill-sql a while ago. This made the PostgreSQL implementation more bulletproof.

Now, the v4 is around the corner. Similar to watermill-amqp, we decided to make SchemaAdapter and OffsetsAdapter interfaces more flexible. Instead of raw arguments, they now take a params struct and always return an error.

It’s a release candidate because we want to include one more breaking change so we don’t bump the major version again soon. There’s an ongoing development in PR #29: streamline the database interfaces and add pgx adapter contributed by @julesjcraske . The aim is to allow watermill-sql to be used with pgx and ORMs not compatible with database/sql, a common request from the community. Stay tuned!

Mapping Watermill’s log levels to slog

Sometimes, the default INFO logs of Watermill may be too verbose for your application. To help you with that, we added a new feature to map Watermill’s log levels to slog levels.

You can now map Watermill’s log levels to slog levels with NewSlogLoggerWithLevelMapping:

logger := NewSlogLoggerWithLevelMapping(
 nil,
 map[slog.Level]slog.Level{
 slog.LevelInfo: slog.LevelDebug,
 },
)

New docs website

We refreshed Watermill’s documentation. It should be now easier to find your way around it.

See the new version at watermill.io.

Refreshed logo

Watermill’s Gopher got a new look! 🎨

New Examples

In case you missed it, there’s a new example featuring Server-Sent Events and htmx.

You may also like the examples on distributed transactions where the outbox pattern is used with Watermill.

There are also the mentioned above examples on delayed messages and delayed requeue

Releases Dashboard

You can now track the releases of all Watermill repositories in one place: releases.threedots.tech

Terraform

We set up a Terraform repository to manage all Pub/Sub repositories. If you want to take a look, it’s here: watermill-terraform.

Celebrating the Launch

Just like the last two times, we’re hosting a live event to celebrate the release of Watermill v1.4!

We will share more details about the changes, show live demos, and answer your questions during Q&A.

We start on October 30 at:

🇪🇺 6 PM CEST
🇺🇸 9 AM PT / 12 PM EST
🇮🇳 9:30 PM IST

Join us on YouTube (Add the event to Google Calendar)

We will also have news about our Go Event-Driven training.

We have a tradition of running a quiz during the live events. The prize is a unique crotchet Gopher. 😎

Meet the Gophers at our live event!

See you there!

Optimising and Visualising Go Tests Parallelism: Why more cores don't speed up your Go tests

Thu, 17 Oct 2024 00:00:00 +0200

Recently, I struggled for a couple of hours to understand why the API tests of one project were slow. In theory, we designed tests to run in a fully parallel way – the duration of tests should be close to the longest-running test. Unfortunately, the reality was different. Tests took 7x longer than the slowest test without using 100% available resources.

In this article, I will show you a few techniques to help you understand and optimize your tests execution. Optimizing tests that use CPU efficiently is simple (in most cases just add more resources). We’ll focus on a scenario of optimizing single-threaded, CPU-heavy integration, component, API, and E2E tests.

It’s hard to fix a problem that you can’t see

It’s difficult to understand how tests run from the output generated by go test. You can see how long each test took, but you don’t know how long a test waited to run. You also can’t see how many tests ran in parallel. It becomes even harder when your project has thousands of tests.

Surprisingly, I didn’t find any tool that helps visualize Go test execution. As a hobbyist frontend engineer, I decided to build my own over a weekend.

`vgt` the missing tool for Visualizing Go Tests

vgt can parse JSON Go test output to create a visualization. The quickest way to use it is by calling:

go test -json ./... | go run github.com/roblaszczak/vgt@latest

or by installing and running

go install -u github.com/roblaszczak/vgt
go test -json ./... | vgt

In a perfect world, this is how ideal test execution of tests that are not CPU-bound should look:

All non-CPU bound tests are executed in parallel. Output from vgt.

Each bar represents the test execution time of a single test or subtest. Unfortunately, the tests I was recently debugging looked more like this:

These tests can be optimized. Output from vgt.

While the CPU was not fully used, tests ran one by one. It is also a good sign: we have a big room for improvement.

Parallelizing Go tests

By default, in Go all tests within a single package are run sequentially. This is not a problem when tests use all CPU cores efficiently and tests are split into multiple packages. But our CPU wastes cycles when a database query, API call, or sleep blocks tests – especially if we have a lot of tests in a single package.

Tests without t.Parallel() flag. Output from vgt.

To fix this problem, *testing.T provides the t.Parallel() method, which allows tests and sub-tests to run in parallel.

Warning

t.Parallel() is not a silver bullet and should be used with caution.

Use t.Parallel() only for tests that have blocking operations like database queries, API calls, sleeps. It can also make sense for CPU-heavy tests using only a single-core.

For fast unit tests the overhead of using t.Parallel() will be higher than running them sequentially. In other words, using t.Parallel() for lightweight unit tests will likely make them slower.

Parallelism limit

Even if you use t.Parallel(), it doesn’t mean that all tests will run in parallel. To simulate this scenario, I wrote an example test that will simulate 100 tests doing API calls.

func TestApi_parallel_subtests(t *testing.T) {
 t.Parallel()

 for i := 0; i < 100; i++ {
 t.Run(fmt.Sprintf("subtest_%d", i), func(t *testing.T) {
 t.Parallel()
 simulateSlowCall(1 * time.Second)
 })
 }
}

func simulateSlowCall(sleepTime time.Duration) {
 time.Sleep(sleepTime + (time.Duration(rand.Intn(1000)) * time.Millisecond))
}

As long as the target server is not overloaded and tests are appropriately designed, we should be able to run all tests in parallel. In this case, running all tests should take at most 2 seconds. But it took over 16 seconds instead.

Despite using t.Parallel(), the execution graph shows many gray bars representing pauses. Tests marked as PAUSED are limited due to the parallelism limit.

These tests can be optimized. Output from vgt.

First, let’s understand how tests with t.Parallel() run. If you’re curious, you can check the source code of the testing package. What’s important, test parallelism is set to runtime.GOMAXPROCS(0) by default, which returns the number of cores reported by the OS.

parallel = flag.Int("test.parallel", runtime.GOMAXPROCS(0), "run at most `n` tests in parallel")

Full source

On my Macbook, runtime.GOMAXPROCS(0) returns 10 (as I have a 10-core CPU). In other words, it limits tests run in parallel to 10.

Limiting tests to the number of our cores makes sense when they are CPU-bound. More parallelism will force our OS to do more expensive context switching. But when we are calling a database, API, or any blocking I/O, it makes tests longer without fully using our resources.

The situation can be even worse when API tests run in CI against an environment deployed in a separate VM or cloud. Often, CI runners for API tests may have 1-2 CPUs. With 1 vCPU, API tests will run one by one. We can simulate this by setting the env GOMAXPROCS=1.

Tests with 1 vCPU – this is how it may look in CI. Output from vgt.

Parallelism is effectively set to 1, so we see a lot of gray bars representing waiting time. To fix this problem, we can use the -parallel (or -test.parallel – they have the same effect) flag. Go documentation says:

 -parallel n
 Allow parallel execution of test functions that call t.Parallel, and
 fuzz targets that call t.Parallel when running the seed corpus.
 The value of this flag is the maximum number of tests to run
 simultaneously.
 While fuzzing, the value of this flag is the maximum number of
 subprocesses that may call the fuzz function simultaneously, regardless of
 whether T.Parallel is called.
 By default, -parallel is set to the value of GOMAXPROCS.
 Setting -parallel to values higher than GOMAXPROCS may cause degraded
 performance due to CPU contention, especially when fuzzing.
 Note that -parallel only applies within a single test binary.
 The 'go test' command may run tests for different packages
 in parallel as well, according to the setting of the -p flag
 (see 'go help build').

Tip

Don’t change GOMACPROCS to a value higher than the available cores to force more parallelism.

It will have more effects than just on tests — it will spawn more Go threads than cores. It will lead to more expensive context switching and may slow down CPU-bound tests.

Let’s see how the same test will behave with the extra -parallel 100 flag:

go test ./… -json -parallel=100 | vgt

We achieved our goal - all tests run in parallel. Our tests were not CPU-bound, so the overall execution time can be as long as the slowest executed test.

Tip

If you are not changing the test code and want to test their performance, Go may cache them.

To avoid caching, run them with the -count=1 flag, for example go test ./... -json -count=1 -parallel=100 | vgt.

Tests in multiple packages

Using the -parallel flag is not the only thing we can do to speed up tests. It’s not uncommon to store tests in multiple packages. By default, Go limits how many packages can run simultaneously to the number of cores. Let’s look at the example project structure:

$ ls ./tests/*

./tests/package_1:
api_test.go

./tests/package_2:
api_test.go

./tests/package_3:
api_test.go

Usually, we may have many more packages with tests for more complex projects. For readability, let’s simulate how tests will run on one CPU (for example, in a CI runner with 1 core):

GOMAXPROCS=1 go test ./tests/... -parallel 100 -json | vgt

Running tests on a runner with 1 core with 3 packages. . Output from vgt.

You can see that every package runs separately. We can fix this with the -p flag. It may not be a problem if your tests run on machines with multiple cores and you don’t have many packages with long-running tests. But in our scenario, with CI and one core, we need to specify the -p flag. It will allow up to 16 packages to run in parallel.

GOMAXPROCS=1 go test ./tests/... -parallel 128 -p 16 -json | vgt

Running tests on a runner with 1 core with 3 packages. Output from vgt.

Now, the entire execution time is very close to the longest test duration.

Tip

It’s hard to give -parallel and -p values that will work for all projects. It depends a lot on your types of tests and how they are structured. The default value will work fine for many lightweight unit tests or CPU-bound tests that efficiently use multiple cores.

The best way to find the correct -parallel flag is to experiment with different values. vgt may be helpful in understanding how different values affect test execution.

Parallelism with sub-tests and test tables

Using test tables for tests in Go is very useful when you need to test many input parameters for a function. On the other hand, they have a couple of dangers: creating test tables may sometimes be more complex than just copying the test body multiple times.

Using test tables can also affect the performance of our tests a lot if we forget to add t.Parallel(). This is especially visible for test tables with a lot of slow test cases. Even one use of test table can make our tests considerably slower.

func TestApi_with_test_table(t *testing.T) {
 t.Parallel()

 testCases := []struct {
 Name string
 API string
 }{
 {Name: "1", API: "/api/1"},
 {Name: "2", API: "/api/2"},
 {Name: "3", API: "/api/3"},
 {Name: "4", API: "/api/4"},
 {Name: "5", API: "/api/5"},
 {Name: "6", API: "/api/6"},
 {Name: "7", API: "/api/7"},
 {Name: "8", API: "/api/8"},
 {Name: "9", API: "/api/9"},
 {Name: "10", API: "/api/9"},
 {Name: "11", API: "/api/1"},
 {Name: "12", API: "/api/2"},
 {Name: "13", API: "/api/3"},
 {Name: "14", API: "/api/4"},
 {Name: "15", API: "/api/5"},
 {Name: "16", API: "/api/6"},
 {Name: "17", API: "/api/7"},
 {Name: "18", API: "/api/8"},
 {Name: "19", API: "/api/9"},
 }
 for i := range testCases {
 t.Run(tc.Name, func(t *testing.T) {
 t.Parallel()
 simulateSlowCall(1 * time.Second)
 })
 }
}

Using a test table without t.Parallel(). Output from vgt.

The solution is simple: add t.Parallel() to the test table. But it’s easy to forget about it. We can be careful when using test tables. But being careful doesn’t always work in the real world when you’re in a hurry. We need an automated way to ensure that t.Parallel() is not missed.

Linting if `t.Parallel()` is used

In most projects, we use golangci-lint. It allows you to configure multiple linters and set them up for the entire project.

We can configure which linter should be enabled based on the file name. An example configuration will ensure all tests in files ending with _api_test.go or _integ_test.go are using t.Parallel(). Unfortunately, as a downside, it requires keeping a convention in naming test files.

Tip

Alternatively, you can group your tests by type in multiple packages. So, one package can contain API tests, and another can contain integration tests.

Note

As mentioned earlier, it’s not only pointless, but even slower to use t.Parallel() for all kinds of tests. Avoid requiring t.Parallel() for all types of tests.

This is an example .golangci.yml configuration:

run:
 timeout: 5m

linters:
 enable:
 # ...
 - paralleltest

issues:
 exclude-rules:
 # ...
 - path-except: _api_test\.go|_integ_test\.go
 linters:
 - paralleltest

With this config, we can run the linter:

$ golangci-lint run

package_1/some_api_test.go:9:1: Function TestApi_with_test_table missing the call to method parallel (paralleltest)
func TestApi_with_test_table(t *testing.T) {
^

You can find a reference for configuration in golangci-lint docs.

Tip

Not all tests can always run in parallel. In this case, you can disable the linter for this specific test with //nolint:paralleltest.

//nolint:paralleltest
func SomeTest(t *testing.T) {

 //nolint:paralleltest
 t.Run("some_sub_test", func(t *testing.T) {

 })
}

Parallelism quirks: does grouping tests with `t.Run()` affect performance?

I’ve often seen in many projects a convention of grouping tests with t.Run(). Have you ever wondered if it affects performance in any way?

To check this hypothesis, I wrote 50 tests like this:

func TestApi1(t *testing.T) {
 t.Parallel()

 t.Run("1", func(t *testing.T) {
 t.Run("1", func(t *testing.T) {
 t.Run("1", func(t *testing.T) {
 simulateSlowCall(1 * time.Second)
 })
 })
 })

 t.Run("2", func(t *testing.T) {
 t.Run("2", func(t *testing.T) {
 t.Run("2", func(t *testing.T) {
 simulateSlowCall(1 * time.Second)
 })
 })
 })
}

func simulateSlowCall(sleepTime time.Duration) {
 // for more reliable results I'm using constant time here
 time.Sleep(sleepTime)
}

To compare, I also wrote 50 tests without using t.Run() for subtests:

func TestApi1(t *testing.T) {
t.Parallel()

simulateSlowCall(1 * time.Second)
simulateSlowCall(1 * time.Second)
}

func simulateSlowCall(sleepTime time.Duration) {
// for more reliable results I'm using constant time here
time.Sleep(sleepTime)
}

Does it affect test performance by affecting parallelism in any way? Let’s see what vgt will show us.

Tests without using r.Run() for subtests. Output from vgt.

Tests using r.Run() for subtests. Output from vgt.

Despite the chart looking a bit uglier, execution times are the same for grouped and non-grouped tests. In other words, grouping tests with t.Run() does not affect performance.

Summary

Having fast and reliable tests for efficient development is crucial. But over the years, I’ve learned that reality is not always so simple. There is always more work to do, and tests are not something that our product’s users or boss directly see. On the other hand, a small investment in tests can save a lot of time in the future. Reliable and fast tests are one of the best investments in your project’s return on investment (ROI).

Knowing some tactics to convince your team leader or manager to find time to improve tests is useful. It’s helpful to think in terms that your manager or boss uses. You should consider the benefits they care about. Reducing delivery time - you can even calculate how many minutes per month the entire team wastes by waiting for tests or retrying them. It may be useful to multiply this by the average developer’s salary. It’s also helpful to track bugs shipped to production because tests were so flaky that nobody noticed them.

There are more ways to make your tests more useful. I’ve also observed that people often struggle with naming different types of tests – we will give you some hints about that, too. If you are interested in more articles about testing, check out:

Is vgt useful for you? Don’t forget to give it a star on GitHub and share it with your friends or social media!

Distributed Transactions in Go: Read Before You Try

Wed, 02 Oct 2024 00:00:00 +0200

In the previous post, I looked into running transactions in a layered architecture. Now, let’s consider transactions that need to span more than one service.

If you work with microservices, a time may come when you need a transaction running across them. Especially if the way they are split was an afterthought (the unfortunate but likely scenario). Service A calls service B, which calls service C, and if something goes wrong at the end, the system becomes inconsistent. It would be helpful to have a way to roll back the changes in all the services.

Now, you’re looking at distributed transactions or the saga pattern. It’s a solved problem. Sometimes, there are good reasons to use them. But more often, it’s overkill. If you go down this path, your architecture quickly becomes much more complex than you’d like. (I learned this the hard way.)

If you need things to be consistent, why split them in the first place? The whole idea of using microservices is to keep independent concepts separate.

Let’s face it: your microservice boundaries are likely wrong if you consider using a distributed transaction. This is super common and not easy to solve. But it can get much worse if you apply the wrong pattern to the mess you already have.

Before you try it, I want to show you an alternative that might be simpler. It’s not a silver bullet, but consider it before committing to distributed transactions.

❌ Anti-pattern: Distributed Transactions

Stay away from transactions that span more than one service, except when there's no other way.

They are hard to test, debug, and maintain. Your system becomes tightly coupled and more challenging to extend.

The Example

The snippets below continue the example from the previous post. Here’s a quick refresher: we work with an e-commerce web app where users get virtual points. They can use these points as a discount for their next order. The challenge is keeping the user’s points and the applied discount consistent.

Let’s assume we work with two services: users for authentication and orders for placing orders and tracking users’ discounts. Initially, the split seemed like a good idea because they are separate areas. Later, we realized we couldn’t update users and their discounts within one transaction.

The orders service exposes an HTTP endpoint for adding a discount. In the users service, the handler for using points as a discount looks like this:

func (h UsePointsAsDiscountHandler) Handle(ctx context.Context, cmd UsePointsAsDiscount) error {
 err := h.userRepository.UpdateByID(ctx, cmd.UserID, func(user *User) (bool, error) {
 err := user.UsePoints(cmd.Points)
 if err != nil {
 return false, err
 }

 return true, nil
 })
 if err != nil {
 return fmt.Errorf("could not update user: %w", err)
 }

 err = h.ordersService.AddDiscount(ctx, cmd.UserID, cmd.Points)
 if err != nil {
 return fmt.Errorf("could not add discount: %w", err)
 }

 return nil
}

Full source

We first take the user’s points. After committing the transaction, we call the orders service to add a discount. It works well in a happy path scenario, but the HTTP call works outside the transaction. If anything goes wrong, the points are gone, but we add no discount. The best we can do is log the error and fire an alert so someone on-call can manually add the discount for the annoyed customer.

Welcome to the distributed monolith.

❌ Anti-pattern: The Distributed Monolith

Be extra careful when designing the boundaries of microservices. What seems like a minor decision in the beginning can have terrible consequences in the future. If in doubt, stick to a single service and decouple your code using modules — the modular monolith approach.

One way out of this mess is to accept past mistakes and merge the microservices into one service. Depending on the approach used and the size of the services, it may not be a quick task. And if your product backlog is full and people who designed the microservices are still around, it’s unlikely to happen.

If you see the problem as “we miss consistency between services”, running a distributed transaction seems like a good solution. But the real problem is “the boundaries are wrong”. It’s easy to make the project much more complicated because of this misinterpretation.

One alternative is to embrace eventual consistency.

Eventual Consistency

The idea is that the data (the user’s points and applied discount in our example) stays consistent but no longer within a single transaction. In practice, there’s a slight delay — usually milliseconds — during which the system is inconsistent but then catches up.

The system may become out of sync for longer if something goes wrong. But the important part is that it eventually becomes consistent again. We don’t accept inconsistency — we’re just okay with waiting for it to happen. Think about sending a bank transfer. The money disappears from your account immediately but isn’t visible on the other account right away. You know it will eventually happen within a few hours or days, so it’s not an issue.

Whether it’s acceptable in a given scenario is often a business decision. Since business stakeholders generally don’t consider this, engineers must explain it and suggest solutions.

Having a transaction boundary sounds reasonable in the “using points as a discount” scenario. But since we already have to work with a distributed monolith, choosing eventual consistency might be the next best thing instead of implementing a distributed transaction.

Note

I could have picked another example that would be more fit for eventual consistency. In some cases, data doesn't need to be consistent at all.

But this scenario is closer to what you can see in your daily work. Using a distributed transaction here may be tempting because it's not obvious if the data can be eventually consistent.

Events

Eventual consistency usually means working with events.

Going this path, we no longer call the orders service directly. We take the user’s points and publish an event, like PointsUsedForDiscount, to record that it happened.

The event is stored on a message queue (AKA a Pub/Sub), a reliable, highly-available infrastructure for messages. The orders service receives this event and reacts by applying a discount for the next order. We can then update the discount value on the website (for example, with Server-Sent Events). Most of the time, it takes milliseconds, and the customers can’t tell a difference.

It’s sometimes better to retry the operation internally instead of showing a scary error message to the user. After all, the issue might be temporary, and retrying is often a good enough fix. However, you need to consider it separately for each scenario. It’s often crucial that the user knows an error has occurred.

If the orders service goes down, it won’t interrupt the process. The event will be delivered again and again until it’s processed successfully. Once the service comes up, the discount will be applied as expected.

It doesn’t fix all the issues. For example, the customer won’t see the discount if the service is down for a few hours. But it gives us a solid way of retrying the operation if it’s a short issue. In many scenarios, retrying for a few hours is also acceptable (generating reports, issuing invoices, etc.).

In most cases, we don’t need to manually investigate what data needs to be backfilled, and no one needs to wake up in the middle of the night. The system auto-heals with time.

✅ Tactic: Embrace Eventual Consistency

Not all operations need to be strongly consistent, even if it initially seems like it.

Keep eventual consistency in mind when designing distributed systems. It's often simpler than going the distributed transactions way.

Note

Events or messages?

While sometimes used as synonyms, there's a difference between events and messages.

A message is a transport unit you publish to the Pub/Sub or receive from it. It's similar to an HTTP request with a method, path, headers, and body.

An event is a specific message. It represents a fact about something that has already happened in the system. The event is represented by the payload held by the message (encoded as JSON or another format). It's what you would include in the HTTP request's body.

Don't miss new posts.
Join over 18k subscribers of our newsletter and get a free e-book!

Go With The Domain Three Dots Labs

🔒 We do not send spam. You can unsubscribe at any time!

Implementation

To start working with messages, you must pick a broker (the infrastructure part), just like when choosing a database. You publish messages to the broker, which asynchronously pushes them to all subscribers. In this post, I’m going to use Redis.

Most popular brokers (Pub/Subs) have a Go library (SDK) that lets you interact with it. But they usually offer a low-level API. As much as I like Go’s net/http, I still prefer using Echo or Chi for their high-level API. Similarly, I will use Watermill to work with messages.

Watermill is a Go library we maintain. I won’t explain how it works here in detail. See the Getting Started page for an overview. It supports many Pub/Sub backends, like Kafka, AMQP, NATS, or Redis.

Note

All the ideas below are universal and can be implemented with any other messaging library.

Watermill is not a framework, so it's easy to add or remove from a project. It's one of the core design ideas.

You can see that I won't include Watermill-specific code in the logic layer. Feel free to use anything else if it fits your needs better.

We’ll use the pair of EventBus and EventProcessor components from Watermill’s cqrs package.

First, let’s replace the HTTP call with publishing an event. The event bus lets us publish messages with a JSON payload. The setup should be easy to grasp even if you don’t know Watermill yet.

client := redis.NewClient(&redis.Options{
 Addr: redisAddr,
})

logger := watermill.NewStdLogger(false, false)

publisher, err := redisstream.NewPublisher(
 redisstream.PublisherConfig{
 Client: client,
 },
 logger,
)
if err != nil {
 return nil, err
}

eventBus, err := cqrs.NewEventBusWithConfig(publisher, cqrs.EventBusConfig{
 GeneratePublishTopic: func(params cqrs.GenerateEventPublishTopicParams) (string, error) {
 return params.EventName, nil
 },
 Marshaler: cqrs.JSONMarshaler{},
 Logger: logger,
})

Full source

We create a Redis publisher and an EventBus. It’s going to use the event’s name as a topic. (A topic is what you subscribe to to receive messages.) It’s going to marshal events into JSON.

In the command handler, we replace the synchronous call with publishing an event.

type EventPublisher interface {
 Publish(ctx context.Context, event any) error
}

type PointsUsedForDiscount struct {
 UserID int `json:"user_id"`
 Points int `json:"points"`
}

func (h UsePointsAsDiscountHandler) Handle(ctx context.Context, cmd UsePointsAsDiscount) error {
 err := h.userRepository.UpdateByID(ctx, cmd.UserID, func(user *User) (bool, error) {
 err := user.UsePoints(cmd.Points)
 if err != nil {
 return false, err
 }

 return true, nil
 })
 if err != nil {
 return fmt.Errorf("could not update user: %w", err)
 }

 event := PointsUsedForDiscount{
 UserID: cmd.UserID,
 Points: cmd.Points,
 }

 err = h.eventPublisher.Publish(ctx, event)
 if err != nil {
 return fmt.Errorf("could not publish event: %w", err)
 }

 return nil
}

Full source

In the orders service, we replace the HTTP handler with an event handler.

Let’s set up an EventProcessor for subscribing, just like we set up the event bus for publishing.

client := redis.NewClient(&redis.Options{
 Addr: redisAddr,
})

logger := watermill.NewStdLogger(false, false)

router := message.NewDefaultRouter(logger)

eventProcessor, err := cqrs.NewEventProcessorWithConfig(router, cqrs.EventProcessorConfig{
 GenerateSubscribeTopic: func(params cqrs.EventProcessorGenerateSubscribeTopicParams) (string, error) {
 return params.EventName, nil
 },
 SubscriberConstructor: func(params cqrs.EventProcessorSubscriberConstructorParams) (message.Subscriber, error) {
 return redisstream.NewSubscriber(
 redisstream.SubscriberConfig{
 Client: client,
 ConsumerGroup: "orders-svc",
 },
 logger,
 )
 },
 Marshaler: cqrs.JSONMarshaler{},
 Logger: logger,
})

Full source

The setup is almost identical to the publishing side. The main difference is that we provide a constructor instead of a single subscriber. I won’t dive into the reasons, as they are irrelevant to what we want to do. To receive messages correctly, we must use the same marshaler and topic configuration as in the event bus config.

Next, we need an event handler. It simply maps the event payload to the command and executes it, just like the HTTP handler did.

type OnPointsUsedForDiscountHandler struct {
 addDiscountHandler AddDiscountHandler
}

func (h OnPointsUsedForDiscountHandler) Handle(ctx context.Context, event *PointsUsedForDiscount) error {
 cmd := AddDiscount{
 UserID: event.UserID,
 Discount: event.Points,
 }

 return h.addDiscountHandler.Handle(ctx, cmd)
}

Full source

We add it to the event processor.

err = eventProcessor.AddHandlers(
 cqrs.NewEventHandler(
 "OnPointsUsedForDiscountHandler",
 onPointsUsedForDiscountHandler.Handle,
 ),
)

Full source

The behavior is exactly the same as that of the HTTP endpoint. The message is just another transport, and the handler is another entry point to the application. The main difference is that it’s asynchronous, so no client waits for it to complete.

That’s it! With a few changes, we replaced the synchronous HTTP call with a message handler.

Of course, there’s the hard part of running a production-grade Pub/Sub. I used Redis, but you can pick what you feel comfortable with or what your cloud provider offers. You can even start with an SQL database if you don’t want to set up new infrastructure.

Are you experienced engineer who wants to learn Go basics?
You don't become an engineer by watching videos.
Learn Go hands-on by building real projects.

The Outbox Pattern

There’s still one scenario when things can go wrong. Since publishing the message works over the network, it can fail, just like the HTTP call to another service. Pub/Subs are usually highly available, but issues still happen (never assume the network is reliable!). We could lose the messages after the transaction is committed.

It’s one of the most common mistakes we’ve seen: ignoring this scenario with the hope it will never happen. It’s unlikely, sure, but you don’t want to take this risk. Imagine figuring out what events you lost and manually correcting the system’s state.

We need to store changes in the database and publish the event within the same transaction. Is this possible?

Yes, we can use the outbox pattern. The idea is to save the event in the same database as the regular data within the same transaction. Then, we asynchronously publish it to the Pub/Sub. This way, the event and the stored data will always be consistent, and we will not rely on the Pub/Sub to be around. The events can be simply stored in a dedicated SQL table.

The main challenge is waiting for new events and moving them to the Pub/Sub. Depending on the database, you can use some custom software, although configuring it is often quite complex. You can implement it from scratch, but it’s not trivial.

In this example, I’ll use Watermill’s Forwarder component to achieve this. Its behavior is straightforward, thanks to the heavy lifting done by the Pub/Sub implementation. It listens to messages from the given Pub/Sub and publishes them to another Pub/Sub.

Watermill supports Postgres as one of the Pub/Sub implementations, so adding it to the existing setup is easy. (Yes, you can simply use an SQL database as your messaging infrastructure.)

We face another design dilemma. How can the publisher and repository work together without mixing the logic and the database code?

We’ll extend the UpdateFn function so it decides what events should be published. (For more on how this pattern works, see the post on database transactions.)

func (h UsePointsAsDiscountHandler) Handle(ctx context.Context, cmd UsePointsAsDiscount) error {
 return h.userRepository.UpdateByID(ctx, cmd.UserID, func(user *User) (bool, []any, error) {
 err := user.UsePoints(cmd.Points)
 if err != nil {
 return false, nil, err
 }

 event := PointsUsedForDiscount{
 UserID: cmd.UserID,
 Points: cmd.Points,
 }

 return true, []any{event}, nil
 })
}

Full source

The repository publishes the returned events.

updated, events, err := updateFn(user)
if err != nil {
 return err
}

if !updated {
 return nil
}

_, err = tx.ExecContext(ctx, "UPDATE users SET email = $1, points = $2 WHERE id = $3", user.Email(), user.Points(), user.ID())
if err != nil {
 return err
}

eventBus, err := NewWatermillEventBus(tx)
if err != nil {
 return err
}

for _, event := range events {
 err = eventBus.Publish(ctx, event)
 if err != nil {
 return err
 }
}

Full source

Note how a new event bus is created using the tx variable. This is how we ensure publishing happens within the same transaction.

We must amend how the event bus is created in the users service. We will no longer publish directly to Redis. We need to change the original publisher from Redis to Postgres.

publisher, err = watermillSQL.NewPublisher(
 db,
 watermillSQL.PublisherConfig{
 SchemaAdapter: watermillSQL.DefaultPostgreSQLSchema{},
 },
 logger,
)

Full source

Next, we wrap this publisher with the forwarder’s Publisher. It introduces an envelope around the message so the forwarder knows how to handle it. Its config is minimal.

publisher = forwarder.NewPublisher(
 publisher,
 forwarder.PublisherConfig{
 ForwarderTopic: forwarderTopic,
 },
)

Full source

The last part is running the forwarder. While it could be a standalone service, keeping it running as another goroutine within the users service is also fine.

We need an SQL subscriber to receive the stored events from the database, and a Redis publisher to publish the messages where the orders service expects them.

sqlSubscriber, err := watermillSQL.NewSubscriber(
 db,
 watermillSQL.SubscriberConfig{
 SchemaAdapter: watermillSQL.DefaultPostgreSQLSchema{},
 OffsetsAdapter: watermillSQL.DefaultPostgreSQLOffsetsAdapter{},
 InitializeSchema: true,
 },
 logger,
)
if err != nil {
 return nil, err
}

client := redis.NewClient(&redis.Options{
 Addr: redisAddr,
})

redisPublisher, err := redisstream.NewPublisher(
 redisstream.PublisherConfig{
 Client: client,
 },
 logger,
)
if err != nil {
 return nil, err
}

fwd, err := forwarder.NewForwarder(
 sqlSubscriber,
 redisPublisher,
 logger,
 forwarder.Config{
 ForwarderTopic: forwarderTopic,
 },
)

Full source

Note that the forwarder topic needs to match between the forwarder publisher and the forwarder itself. In this case, it’s the same forwarderTopic constant (can be any string).

Finally, we run the forwarder.

go func() {
 err := forwarder.Run(context.Background())
 if err != nil {
 panic(err)
 }
}()

No changes are necessary on the orders service side. As far as it’s concerned, it receives events from Redis, as it used to.

✅ Tactic: The Outbox Pattern

Use the Outbox pattern to publish events within a database transaction. Don’t leave it to chance whether all events will be successfully published.

Simplifying or Complicating?

There are many moving parts here. Event-driven patterns are not trivial, and you need to understand what you’re doing to some degree. If you never used these concepts before, it can seem complex.

I used to have these concerns as well. The system seems simpler with just the synchronous APIs and a database. Why complicate it with adding all this background processing?

As counterintuitive as it is, you can simplify your distributed system using these ideas. Because you work with facts that happened (events), you don’t need to consider rollbacks, which terribly complicate your flow.

A rollback can interfere with other flows, so you need distributed locks to protect the state. Rollbacks can also fail. What do you do when this happens?

As the snippets show, little code is needed to start working with events, and the handlers are also straightforward. This is where Watermill helps by removing the low-level boilerplate. You can focus on setting up the handlers and publishing messages with an API that’s easier to use than HTTP endpoints.

Once you grasp the basics, your mental model will start changing, and eventually, it will click. You’ll get the intuition on how to design event-driven systems.

Events & Coupling

Make no mistake, events are still a form of a contract. The fact you use them doesn’t magically make your services decoupled. Incorrectly designed events are another trap you can fall into.

What I like about the pattern is that it limits what the services know about each other. But it’s easy to fail at this.

The example above shows that the event is designed quite poorly. The UsePointsAsDiscount name makes the users service aware of what happens in the orders service after it receives it. It’s too late to escape this now, though. The service boundaries are just wrong from the beginning.

Well-designed events state a fact that happened in this service’s domain. The service publishing the event should not know or care what happens when another service consumes it.

Testing

Testing event-driven services is not much different, but here are some tips.

Run your Pub/Sub locally in a Docker container and test against it. Cloud-based Pub/Subs usually offer an emulator (but often with no feature parity, so watch out).

Testing the event handler code makes little sense. Event handlers should just execute your application logic with the data from the event. Instead, use component tests to test the behavior of the service using the public API.

A test scenario can be something like:

Once this event is published, an entity read over HTTP should change.
Once this HTTP endpoint is called, an event should be published.
Once this event is published, another event should be published.

If you use testify, the Eventually and EventuallyWithT functions are helpful. (The second one allows using asserts inside). They run the assert code periodically until it’s successful or until it times out. Use these instead of plain sleeps, as they slow down your tests.

assert.EventuallyWithT(t, func(t *assert.CollectT) {
 row := discountDB.QueryRowContext(context.Background(), "SELECT next_order_discount FROM user_discounts WHERE user_id = $1", userID)

 var discount int
 err := row.Scan(&discount)
 require.NoError(t, err)

 assert.Equal(t, expectedDiscount, discount)
}, 2*time.Second, 100*time.Millisecond)

Full source

When reading events, watch out for others that may interfere. With many tests running in parallel, it’s common to have many events of the same type published. You need to filter them based on the ID or some metadata. Using UUIDs for this is best so your tests are completely independent.

Finally, run your tests in parallel to speed them up. With lots of time spent on I/O and waiting, this quickly adds up. There are some quirks to running Go tests in parallel, though. Robert is currently working on an article on how to do it right — join our newsletter, and we’ll let you know once it’s ready.

Monitoring

Observability is a long topic, and there’s much to be said about metrics and tracing for event-driven systems. This is beyond the scope of this post, but I want to mention monitoring the queue of waiting messages.

If a message fails to be processed (i.e., the handler returns an error for whatever reason), it will be delivered again. Depending on the Pub/Sub you picked and its configuration, it can block other messages from being delivered.

This is your key metric to monitor. Having unprocessed messages for a long time usually means something went wrong, and it’s not a temporary issue. You need manual intervention, or your system will be out of sync.

More Examples

This scratches the surface of switching to eventual consistency, but it should be enough to get you started. To learn about different use cases, check Watermill examples. There are some basic ones, but also more advanced and interesting examples.

As with most programming topics, reading about something is rarely enough to fully get it. For me, building something on my own is what makes me fully understand it. But it’s often hard to come up with an idea for a project to build. We wanted to recreate this way of learning, so we designed a hands-on event-driven training. Consider joining us if you like learning like this.

Database Transactions in Go with Layered Architecture

Thu, 05 Sep 2024 00:00:00 +0200

As I join a new company, I often feel like an impostor. After all the interviews, they really seem to know what they’re doing. I’m humbled and ready to learn from the best.

On one such occasion, a few days in, I dealt with a production outage and asked the most senior engineer for help. They came to the rescue and casually flipped a value in the database with a manual update. 🤯 The root cause was that a set of SQL updates were not done within a transaction. Suddenly, I regretted not asking for higher compensation right away.

Onboarding is fun. What I learned this way is that even if something seems like a fundamental concept (e.g., SQL transactions), it may often be overlooked.

SQL seems like something we all know well, and there are few surprises. (It’s 50 years old!) Perhaps it’s a good time to reconsider, as we’re past the “NoSQL is cool” hype phase and back to “just use Postgres”, or even “SQLite is good enough”.

I want to focus here on how to keep transactions in the code rather than on their technical complexity. Once you need to organize a bigger project, perhaps by using layers, splitting the logic from the database code is not always that obvious. It’s easy to end up with a mess leading to obscure bugs.

The key idea of layers is to keep the critical parts of code (the logic) out of the implementation details (like the SQL queries). One way to achieve this separation is the Repository pattern. The hard part is handling transactions in this setup. When using transactions, you usually fetch data from the database, run some logic, update the data, and commit the transaction.

How do we model this in the code so we don’t end up with a mess across the layers? I often see people asking about this, and I know the pain from my experience. We had to figure this out, and the ideas evolved over time. Here’s a list of various approaches you can consider and everything I’ve learned along the way.

SQL Transactions 101

Let’s quickly reiterate why we need transactions at all.

Consider an e-commerce web app where users receive virtual points for purchasing products. The points can later be spent on discounts.

The user decides how many points they want to spend on a discount. We have to check if the user has enough points, then decrease them and apply a proper discount for the next order.

Let’s say a user wants to use 100 points. It could look roughly like this:

SELECT points FROM users where id = 19;
-- in the code: check if points >= 100
UPDATE users SET points = points - 100 WHERE id = 19;
UPDATE user_discounts SET next_order_discount = next_order_discount + 100 WHERE user_id = 19;

We should consider what can go wrong here, even if it’s unlikely.

Our application can suddenly exit after the first UPDATE statement. Or, a network outage could prevent the second UPDATE from being executed. In this case, we would decrease the user’s points without giving them a discount.

Or a different scenario: If the user sends two (or more) requests very quickly, they could be processed concurrently. If the requests execute the SELECT query before any updates, the check for the points balance won’t work correctly, and the user could end up with -100 points and a 200 discount value.

❌ Anti-pattern: Skipping transactions

Use transactions for sets of queries that rely on each other.

Even if unlikely, things can go wrong mid-request. You’ll be left with inconsistencies in the system that need manual effort to be understood and fixed.

If something can theoretically happen, it will likely happen in production at some point. Don’t accept solutions that almost always work. The 1% chance can turn into a nightmare investigation scenario in which you can’t understand what happened in your application and how to fix it.

We can easily improve this by introducing a transaction.

BEGIN;
SELECT points FROM users where id = 19 FOR UPDATE;
-- in the code: check if points >= 100
UPDATE users SET points = points - 100 WHERE id = 19;
UPDATE user_discounts SET next_order_discount = next_order_discount + 100 WHERE user_id = 19;
COMMIT;

Now, either all of the updates get saved or none.

Besides the BEGIN and COMMIT statements that manage the transaction, note the FOR UPDATE clause added to the SELECT. It locks the row so that further SELECTS have to wait until this transaction is done. This lets us correctly process parallel requests.

Warning

FOR UPDATE works well for simple uses cases, but it may cause performance issues. It locks the entire row, so if many concurrent updates are competing for it, they will all have to wait. On a bigger scale, this can lead to a bottleneck.

An alternative is to use another isolation level, like REPEATABLE READ. This is out of the scope of this post but you can check out this article for details.

Whatever you choose, consider running stress tests to see how your code behaves under load. It will help you avoid surprises in production when a traffic spike hits.

Layers

Splitting code into layers is pretty common, no matter how many you use or what you call the approach (clean, hexagonal, ports & adapters, onion, etc.). It makes sense for many reasons, like separation of concerns, enabling parallel work, and easier testing. We wrote about these concepts in our other articles.

For the rest of this post, I assume you keep your SQL code in a place (struct, file, package, whatever) separate from your application logic and inject one into the other. In the example snippets, I keep the “layers” in different files but in the same package for simplicity.

http.go — the HTTP handler (we don’t care about this one much).
app.go — the application logic.
repository.go — the repository (the database storage).

In the application logic, I define the UsePointsAsDiscount command and a handler for it (see Robert’s CQRS article for more details on this). You may prefer using something slightly different, like a service struct with methods or use cases. That’s all fine. The key is that this part of the code knows nothing about the database used. The repository is injected inside the command handler using an interface defined close to the handler’s definition.

Here’s how the code can look like with no transactions.

type UsePointsAsDiscount struct {
 UserID int
 Points int
}

type UsePointsAsDiscountHandler struct {
 userRepository UserRepository
 discountRepository DiscountRepository
}

type UserRepository interface {
 GetPoints(ctx context.Context, userID int) (int, error)
 TakePoints(ctx context.Context, userID int, points int) error
}

type DiscountRepository interface {
 AddDiscount(ctx context.Context, userID int, discount int) error
}

func NewUsePointsAsDiscountHandler(
 userRepository UserRepository,
 discountRepository DiscountRepository,
) UsePointsAsDiscountHandler {
 return UsePointsAsDiscountHandler{
 userRepository: userRepository,
 discountRepository: discountRepository,
 }
}

func (h UsePointsAsDiscountHandler) Handle(ctx context.Context, cmd UsePointsAsDiscount) error {
 if cmd.Points <= 0 {
 return errors.New("points must be greater than 0")
 }

 currentPoints, err := h.userRepository.GetPoints(ctx, cmd.UserID)
 if err != nil {
 return fmt.Errorf("could not get points: %w", err)
 }

 if currentPoints < cmd.Points {
 return errors.New("not enough points")
 }

 err = h.userRepository.TakePoints(ctx, cmd.UserID, cmd.Points)
 if err != nil {
 return fmt.Errorf("could not take points: %w", err)
 }

 err = h.discountRepository.AddDiscount(ctx, cmd.UserID, cmd.Points)
 if err != nil {
 return fmt.Errorf("could not add discount: %w", err)
 }

 return nil
}

Full source

There are two repositories: one for the user and one for the discounts. The handler’s logic is rather easy to grasp: we validate the command (points must be positive), check if the user has enough points, and subtract them from their account while adding a discount.

The repositories are trivial:

type PostgresDiscountRepository struct {
 db *sql.DB
}

func NewPostgresDiscountRepository(db *sql.DB) *PostgresDiscountRepository {
 return &PostgresDiscountRepository{
 db: db,
 }
}

func (r *PostgresDiscountRepository) AddDiscount(ctx context.Context, userID int, discount int) error {
 _, err := r.db.ExecContext(ctx, "UPDATE user_discounts SET next_order_discount = next_order_discount + $1 WHERE user_id = $2", discount, userID)
 return err
}

Full source

Note

In your project, you may call the repository a storage or something else. It’s a similar concept, although the Repository pattern is not only about database access. See Robert’s post for more details.

We’d like all of this to run within a single transaction, but we have a bunch of logic mixed with two repositories. Let’s see how to approach this.

Note

All examples are available in the go-web-app-antipatterns repository. There’s a Docker Compose definition that lets you run all of them locally. They use Postgres as the database.

Transactions in the logic layer (avoid if you can)

The first idea for dealing with transactions across repositories is passing a transaction object around.

For running code in a transaction, we can use a helper function like the runInTx below.

func runInTx(db *sql.DB, fn func(tx *sql.Tx) error) error {
 tx, err := db.Begin()
 if err != nil {
 return err
 }

 err = fn(tx)
 if err == nil {
 return tx.Commit()
 }

 rollbackErr := tx.Rollback()
 if rollbackErr != nil {
 return errors.Join(err, rollbackErr)
 }

 return err
}

Full source

It takes a function (you’ll see many anonymous functions in this post) that receives the transaction object. Whatever happens within the fn function doesn’t know or care how the transaction is started, committed, or rolled back.

Here’s how it works when used inside the command handler.

func (h UsePointsAsDiscountHandler) Handle(ctx context.Context, cmd UsePointsAsDiscount) error {
 return runInTx(h.db, func(tx *sql.Tx) error {
 if cmd.Points <= 0 {
 return errors.New("points must be greater than 0")
 }

 currentPoints, err := h.userRepository.GetPoints(ctx, tx, cmd.UserID)
 if err != nil {
 return fmt.Errorf("could not get points: %w", err)
 }

 if currentPoints < cmd.Points {
 return errors.New("not enough points")
 }

 err = h.userRepository.TakePoints(ctx, tx, cmd.UserID, cmd.Points)
 if err != nil {
 return fmt.Errorf("could not take points: %w", err)
 }

 err = h.discountRepository.AddDiscount(ctx, tx, cmd.UserID, cmd.Points)
 if err != nil {
 return fmt.Errorf("could not add discount: %w", err)
 }

 return nil
 })
}

Full source

The good part is that this approach works. But we can’t ignore the fact that it mixes the application logic with implementation details (the SQL transaction). It may not seem like a big deal: we will likely not change the database, so why be concerned about it?

For one, handling the transaction with the commit/rollback sequence complicates the flow. Instead of working with plain application logic, you now have to consider this additional behavior, which has nothing to do with what your application does. As the logic grows, you must carefully consider whether something should work within the transaction or outside of it. A rollback will affect everything you put inside the function.

Not to mention the weird tx argument you must pass to the repository methods. Testing becomes awkward because you need to pass the transaction even if the method doesn’t need it at the time (the GetPoints method could also work without transaction in another context). You can’t mock the SQL connection to test the command handler, so it becomes an integration test instead of a simple unit test checking the logic.

You could try to work around this with an abstract Transaction object or perhaps pass it through the Context, but it only masks the root issue of mixing the implementation details with the logic.

❌ Anti-pattern: Transactions mixed with logic

Avoid mixing transactions with your application logic. It’s challenging to grasp how it works, test the logic, and debug issues.

Transactions inside the repository (better, but far from perfect)

If the transaction belongs to the database layer, why not keep it there? The hard part is that we deal with two repositories, so they must somehow share the transaction object. Sometimes, it’s really the case (and I’ll show you how to do it in a bit). But first, we should consider if we really need two repositories.

It’s common to think of the database tables as your entities. This way, you end up with one repository per table and the application logic orchestrating them. Often, this split of repositories has no practical value.

❌ Anti-pattern: One repository per database table

Don’t create a repository for each database table. Instead, think of the data that needs to be transactionally stored together.

Data that should be transactionally consistent should also be coherent and kept as one. Domain-Driven Design proposes the idea of an aggregate — a set of data that must always be consistent. If you follow this idea, you don’t keep a repository per SQL table, but a repository per aggregate. (It’s the short version of the whole idea. We have a proper article coming up on aggregates soon.)

Don't miss new posts.
Join over 18k subscribers of our newsletter and get a free e-book!

Go With The Domain Three Dots Labs

🔒 We do not send spam. You can unsubscribe at any time!

We think of the user and the discounts as separate concepts (entities, structs, etc.). And it makes sense, as conceptually, they are different things. The user is used for identity and authentication. Placing orders with discounts is just one of many things a user can do on the website. But we also want to keep the user’s points and discounts consistent. We can consider them part of the same aggregate — a set of objects stored together transactionally.

In practice, we can consider discounts part of the User aggregate, even if we store them in another SQL table. In that case, we need just one repository. And this lets us move the transaction handling there.

The application logic becomes trivial now.

func (h UsePointsAsDiscountHandler) Handle(ctx context.Context, cmd UsePointsAsDiscount) error {
 if cmd.Points <= 0 {
 return errors.New("points must be greater than 0")
 }

 err := h.userRepository.UsePointsForDiscount(ctx, cmd.UserID, cmd.Points)
 if err != nil {
 return fmt.Errorf("could not use points as discount: %w", err)
 }

 return nil
}

Full source

And the repository exposes a method specific to this one operation.

func (r *PostgresUserRepository) UsePointsForDiscount(ctx context.Context, userID int, points int) error {
 return runInTx(r.db, func(tx *sql.Tx) error {
 row := tx.QueryRowContext(ctx, "SELECT points FROM users WHERE id = $1 FOR UPDATE", userID)

 var currentPoints int
 err := row.Scan(&currentPoints)
 if err != nil {
 return err
 }

 if currentPoints < points {
 return errors.New("not enough points")
 }

 _, err = tx.ExecContext(ctx, "UPDATE users SET points = points - $1 WHERE id = $2", points, userID)
 if err != nil {
 return err
 }

 _, err = tx.ExecContext(ctx, "UPDATE user_discounts SET next_order_discount = next_order_discount + $1 WHERE user_id = $2", points, userID)
 if err != nil {
 return err
 }

 return nil
 })
}

Full source

✅ Tactic: Aggregates

Keep the data that needs to be strongly consistent within the same aggregate. Keep a repository for the aggregate.

This approach still has some downsides. One is that we had to move the “has enough points” logic to the repository. The method name (UsePointsForDiscount) makes it obvious the repository knows something about the logic, and occasionally it may be fine. But ideally, we’d have the logic in the command handler.

The other issue is that with many methods like this, the repository’s interface grows and gets difficult to maintain and test. This is against the layers split we wanted to achieve.

While useful, this approach doesn’t scale well. It seems we could use a more generic Update method.

The UpdateFn Pattern (our go-to solution)

A universal Update method should load the user from the database, allow us to make any changes, and store the result. The challenge, again, is how to make it work with transactions.

First, we need a model that combines the user with discounts (our aggregate). I use encapsulation to hide the fields, as we want the model to be always valid in memory. The only way to change the state is by calling exported methods.

type User struct {
 id int
 email string
 points int
 discounts *Discounts
}

func (u *User) ID() int {
 return u.id
}

func (u *User) Email() string {
 return u.email
}

func (u *User) Points() int {
 return u.points
}

func (u *User) Discounts() *Discounts {
 return u.discounts
}

type Discounts struct {
 nextOrderDiscount int
}

func (c *Discounts) NextOrderDiscount() int {
 return c.nextOrderDiscount
}

Full source

Adding a discount works by calling UsePointsAsDiscount.

func (u *User) UsePointsAsDiscount(points int) error {
 if points <= 0 {
 return errors.New("points must be greater than 0")
 }

 if u.points < points {
 return errors.New("not enough points")
 }

 u.points -= points
 u.discounts.nextOrderDiscount += points

 return nil
}

Full source

Easy to grasp, trivial to test. No need to think about database transactions when reading it. In fact, no need to think about database tables, too. That’s the kind of code I want to work with.

Now, let’s introduce the update method interface.

type UserRepository interface {
 UpdateByID(ctx context.Context, userID int, updateFn func(user *User) (bool, error)) error
}

Full source

The updateFn argument is how we keep the logic separate from the database details. The function receives the User model that can be modified. Any changes made to it are saved to the storage if no error is returned.

The updateFn returns an updated boolean value indicating whether the user has been updated. If true is returned, the repository saves the user. We don’t have a use case for returning false now, but it’s often helpful when updating multiple values (like in a generic PATCH update request). The update may be unnecessary and can be skipped then.

The command handler looks like this now.

func (h UsePointsAsDiscountHandler) Handle(ctx context.Context, cmd UsePointsAsDiscount) error {
 return h.userRepository.UpdateByID(ctx, cmd.UserID, func(user *User) (bool, error) {
 err := user.UsePointsAsDiscount(cmd.Points)
 if err != nil {
 return false, err
 }

 return true, nil
 })
}

Full source

Let’s take a look at the repository implementation.

func (r *PostgresUserRepository) UpdateByID(ctx context.Context, userID int, updateFn func(user *User) (bool, error)) error {
 return runInTx(r.db, func(tx *sql.Tx) error {
 row := tx.QueryRowContext(ctx, "SELECT email, points FROM users WHERE id = $1 FOR UPDATE", userID)

 var email string
 var currentPoints int
 err := row.Scan(&email, &currentPoints)
 if err != nil {
 return err
 }

 row = tx.QueryRowContext(ctx, "SELECT next_order_discount FROM user_discounts WHERE user_id = $1 FOR UPDATE", userID)

 var discount int
 err = row.Scan(&discount)
 if err != nil {
 return err
 }

 discounts := UnmarshalDiscounts(discount)
 user := UnmarshalUser(userID, email, currentPoints, discounts)

 updated, err := updateFn(user)
 if err != nil {
 return err
 }

 if !updated {
 return nil
 }

 _, err = tx.ExecContext(ctx, "UPDATE users SET email = $1, points = $2 WHERE id = $3", user.Email(), user.Points(), user.ID())
 if err != nil {
 return err
 }

 _, err = tx.ExecContext(ctx, "UPDATE user_discounts SET next_order_discount = $1 WHERE user_id = $2", user.Discounts().NextOrderDiscount(), user.ID())
 if err != nil {
 return err
 }

 return nil
 })
}

Full source

The repository has no logic at all. It fetches the complete data from the user and discounts tables, translates them to application models (with the Unmarshal functions), and calls the updateFn function. Then, it stores everything back in the database. All of it happens within a transaction.

Tip

The repository is quite verbose now. Using an ORM can help you get rid of the boilerplate. But be careful what library you choose, as some ORMs can do more harm than good. Check the libraries that never failed us.

With this, we achieved the goal: the logic stays in the command handler, and the transaction remains in the repository.

✅ Tactic: The UpdateFn pattern

Use an Update method that loads and stores the aggregate. Keep the logic in the updateFn closure.

Note

Isn’t this inefficient?

A common concern about the Update method is that it needlessly updates all the fields even if they didn’t change (for example, the email in the example above).

Unless you’re saving huge datasets or dealing with massive scale, it shouldn’t really matter. You could track which fields changed and update only these, but it will likely only complicate your architecture with no actual performance gains. Be pragmatic and avoid premature optimization. If in doubt, run stress tests.

The Transaction Provider (for the edge cases)

I mentioned that sometimes, keeping a transaction across two repositories makes sense. While your first reaction should be, “Why isn’t this a single repository?” there may be a valid use case for it.

It usually works well for things that are not strictly related to your application’s domain but are more technical or platform-related. For example, consider keeping an audit log of the users’ actions. Saving it within the same transaction makes sense, but keeping it inside the User aggregate seems like overkill.

We can work around having an explicit transaction object mixed with logic using the Transaction Provider pattern. It works similarly to the first approach, but the application code is easier to read with the transaction mechanism working in the background.

type UsePointsAsDiscountHandler struct {
 txProvider txProvider
}

type txProvider interface {
 Transact(txFunc func(adapters Adapters) error) error
}

type Adapters struct {
 UserRepository UserRepository
 AuditLogRepository AuditLogRepository
}

Full source

The transaction provider has a Transact method that takes a function as an argument. Within it, we can access adapters — a struct of all the dependencies we need, working within the same transaction.

Here’s how it looks in the command handler.

func (h UsePointsAsDiscountHandler) Handle(ctx context.Context, cmd UsePointsAsDiscount) error {
 return h.txProvider.Transact(func(adapters Adapters) error {
 err := adapters.UserRepository.UpdateByID(ctx, cmd.UserID, func(user *User) (bool, error) {
 err := user.UsePointsAsDiscount(cmd.Points)
 if err != nil {
 return false, err
 }

 return true, nil
 })
 if err != nil {
 return fmt.Errorf("could not use points as discount: %w", err)
 }

 log := fmt.Sprintf("used %d points as discount for user %d", cmd.Points, cmd.UserID)
 err = adapters.AuditLogRepository.StoreAuditLog(ctx, log)
 if err != nil {
 return fmt.Errorf("could not store audit log: %w", err)
 }

 return nil
 })
}

Full source

Note that both UserRepository and AuditLogRepository come from adapters, not the handler.

Also, we pass no transaction object to the repository methods. The repositories now take the tx in the constructor. It’s handy not to depend on *sql.Tx directly but to use an interface. It also lets you pass an *sql.DB (for example, in tests).

type db interface {
 QueryRowContext(ctx context.Context, query string, args ...interface{}) *sql.Row
 ExecContext(ctx context.Context, query string, args ...interface{}) (sql.Result, error)
}

type PostgresUserRepository struct {
 db db
}

func NewPostgresUserRepository(db db) *PostgresUserRepository {
 return &PostgresUserRepository{
 db: db,
 }
}

Full source

The TransactionProvider implementation looks like this:

type TransactionProvider struct {
 db *sql.DB
}

func NewTransactionProvider(db *sql.DB) *TransactionProvider {
 return &TransactionProvider{
 db: db,
 }
}

func (p *TransactionProvider) Transact(txFunc func(adapters Adapters) error) error {
 return runInTx(p.db, func(tx *sql.Tx) error {
 adapters := Adapters{
 UserRepository: NewPostgresUserRepository(tx),
 AuditLogRepository: NewPostgresAuditLogRepository(tx),
 }

 return txFunc(adapters)
 })
}

Full source

This approach seems clean enough, although you have to be careful. With a set of transaction adapters, it’s easy to go too far and start calling methods from different repositories in a single handler. It shouldn’t become your go-to tool. Stick to the UpdateFn pattern instead.

✅ Tactic: The Transaction Provider

Sometimes, it makes sense to share a transaction across repositories. In such a scenario, use the transaction provider pattern. Be careful not to over-use it.

Warning

The transaction provider is a risky pattern. If you use it together with FOR UPDATE, you need to be extra careful, as all the SELECTs need to include the FOR UPDATE clause to work correctly.

This can quickly get out of control when working within a team with rapid changes.

Once again, consider using a different isolation level, like REPEATABLE READ.

Distributed Transactions

This wraps up our way of working with SQL transactions within a single service. But there’s a related topic that often comes up and that we should consider: transactions across services. I cover this in Distributed Transactions in Go: Read Before You Try.

Check the Source Code

You can see the complete source code on GitHub. There are basic tests that verify that all examples work in the same way. You can run it all locally with Docker Compose.

Live website updates with Go, SSE, and htmx

Thu, 13 Jun 2024 00:00:00 +0200

In case you missed the memo, the Single Page Application hype period is over, and we’re now back to ~~PHP and jQuery,~~ I mean rendering HTML on the server. I’m excited! It brings me back to the early 2000s when we were all web developers, not frontend or backend engineers.

But there’s one thing I would miss from the SPA era: live updates. The classic websites often relied on the “refresh” button, which wasn’t that great. While polling for updates periodically is a solution, it’s inefficient. It’s much better to push updates to the client once they happen.

This post shows how to push live updates to your website using Go, Server-Sent Events (SSE), and htmx. As the example project, I use a tiny microblogging website where you can react to posts.

Below, you can see the embedded example in two “windows”. You can click on reactions and see them update in real-time in the other window. You will also see the reactions and views counters update as other readers interact with the posts. (You can open the example in a new tab here).

Server-Sent Events

WebSockets seem like the most popular option for pushing live updates from the HTTP server to the browser. Meanwhile, Server-Sent Events (SSE) is a great alternative. It is simple to set up and good enough for many use cases. It uses standard HTTP connections, so you don’t need a custom protocol. All modern browsers support SSE.

SSE endpoints work just like standard HTTP endpoints with a slight twist. You set the Content-Type header to text/event-stream and keep writing data in a text format like this:

event: notifications
data: {"unread_messages": 14}

event: message
data: <h1>Hello,
data: world!</h1>

What follows the event: line is an optional event type, which can be any string you want. The multiline data field is the payload sent to the browser. Every “event” is separated by an extra new line. The specification mentions a few more things, but that’s all you need to start.

In Go, SSE endpoints are slightly different from net/http endpoints because you don’t just write data to the ResponseWriter and return from the function. Instead, you reply with the status code (200 OK) and keep writing the “events” in the format above. The connection stays alive until the client (the browser) closes it.

Here’s a complete example of sending back a “ping” event every 10 seconds.

func main() {
 http.HandleFunc("/ping", func(w http.ResponseWriter, r *http.Request) {
 w.Header().Set("Content-Type", "text/event-stream")
 w.WriteHeader(http.StatusOK)

 for {
 select {
 case <-r.Context().Done():
 return
 default:
 }

 fmt.Fprintf(w, "data: ping\n\n")

 if f, ok := w.(http.Flusher); ok {
 f.Flush()
 }

 time.Sleep(10 * time.Second)
 }
 })

 http.ListenAndServe(":8080", nil)
}

Two things are worth mentioning here.

To avoid buffering, we use the http.Flusher to send the response immediately.
To exit the infinite loop, we check if the request’s context is done.

On the client side, you can handle it like this:

<script>
 const es = new EventSource("http://localhost:8080/ping");
 es.onmessage = (event) => {
 console.log(`Received: ${event.data}`);
 };
</script>

Most online tutorials end here and wish you happy coding. Perhaps it’s good enough if you want to create a demo project in one afternoon. But the example is far from something you would use in production. I don’t want you to read this post, then go back to your project and think, “Uhh, so what do I do now?”

If you follow our blog, you know we like to focus on real-world examples. Let’s see how to use SSE in a more complex scenario — the microblog example you saw at the top of this page. It’s not quick and easy, and there are many things to consider, but it should make you comfortable enough to use a similar mechanism in your projects.

Don't miss new posts.
Join over 18k subscribers of our newsletter and get a free e-book!

Go With The Domain Three Dots Labs

🔒 We do not send spam. You can unsubscribe at any time!

The Microblog Example

The core component of the example is a “post”. The key feature is that the reactions and views are updated in real-time as other users interact with the post.

In this example, we’ll render HTML on the server side, using no JavaScript code for interactivity except the htmx library.

The complete example is on GitHub. You can run it locally using docker-compose.

Note

JSON API Example

For another example featuring a Twitter-like web app, see the server-sent-events example on the Watermill repository. It shows a similar approach using a Single Page Application with Vue.js and a JSON API.

Tools used

Here’s the stack I use in the example. I won’t dive deep into all the components, but I’ll mention them briefly here.

Echo — a lightweight HTTP router I like for error handling that is simpler than in net/http.
templ — an HTML templates library based on code generation. It can get weird at times, but overall, I’m happy with it and like it more than html/template. It’s best used together with an IDE plugin.
htmx — a library for using AJAX and SSE with no need to write JavaScript.
Watermill — an event-driven library we maintain for working with messages.
PostgreSQL and Google Cloud Pub/Sub for storage and messaging infrastructure. (You can choose a different Pub/Sub for messaging, even Postgres.)

Deciding what and when to push

When designing an SSE endpoint, you must decide what the payload should be, when to send an update, and to whom to send it.

What

The payload is just text, and it’s up to you how to encode it. It can be a regular JSON API response or an HTML you would embed directly on the website. Remember that each line should have the data: prefix, and the payload needs to end with two new lines (\n\n).

When

You need a way to know when something changes in your application so you can push the updates. For example, if the user receives a message, you show a red bubble in the UI.

The SSE endpoints are long-running, so you may have hundreds or thousands of goroutines running in the background that you must notify of the change. In reaction, each should send an event to the client. Since you’re likely running more than one instance of your service, this can’t work in memory.

To Whom

You often only want to notify some users of something that happened. If I send you a message, I expect a notification to appear on your screen, but not for anyone else. So, you need a way to filter what happens and choose who should get the update (and which SSE endpoints to trigger).

In this example, it goes as follows:

What: the post “stats” model, including the numbers of views and reactions as HTML.
When: when someone sees the post or reacts to it.
To Whom: everyone who sees the updated post. Other posts are not updated.

Implementing SSE Endpoints

While you absolutely can create SSE endpoints from scratch (or with a library), and it’s not that complex, the hard part is triggering the updates in reaction to something that happened. (And doing this over the network since running a single service instance rarely happens in production.) As with running an HTTP server, you don’t want to reinvent the wheel here.

We usually approach anything events-related with Watermill. It’s a Go library we maintain that abstracts away the low-level details of Pub/Subs. (Getting close to 7k GitHub stars ⭐️). You can use it with any existing codebase as it’s not a framework but a lightweight library (just like htmx). It supports many Pub/Subs, so it’s easy to start with the infrastructure you already have (even an SQL database).

Watermill Primer

(Feel free to skip this part if you’re already familiar with Watermill.)

The documentation goes in-depth on how Watermill works. Below is a TL;DR version.

First, you need a Pub/Sub — a system that lets you work with messages across the network (also known as a “message broker” or a “queue”). Common picks are Kafka or RabbitMQ, but it could just as well be an SQL database.

Watermill abstracts away all Pub/Subs into two interfaces:

type Publisher interface {
 Publish(topic string, messages ...*Message) error
 Close() error
}

type Subscriber interface {
 Subscribe(ctx context.Context, topic string) (<-chan *Message, error)
 Close() error
}

You can Publish messages and Subscribe to them. There’s always a topic involved — a string that decides who gets the message.

Here’s basic Watermill architecture in one picture:

The core part of Watermill is the Message. It is what the Request is for the net/http package. The simplest message has just an optional ID and a payload. The payload is a slice of bytes, so you can use any marshaling you want (JSON, Protocol Buffers, plain strings, etc.).

msg := message.NewMessage(watermill.NewUUID(), []byte("Hello, world!"))

While all Watermill’s components are based on the Publisher and Subscriber interfaces, using them directly is a relatively low-level API. In this example, we’ll use the CQRS component of Watermill, which is a higher-level API. It’s based on the same ideas but removes some boilerplate, like serialization and deserialization. We’ll use the EventBus to publish events and EventProcessor to subscribe to them.

High-level architecture overview

We want to publish two events:

PostViewed is published when someone sees the post.
PostReactionAdded is published when someone reacts to a post.

Each will have an event handler that updates the post’s stats in the database. (A similar concept as an HTTP handler.) The handlers should also publish the PostStatsUpdated event. We will use it to trigger the SSE updates.

Publishing events

First, let’s create a publisher. I use Google Cloud Pub/Sub Publisher, but it can be swapped with any other publisher supported by Watermill. All the configuration needs is a project ID.

(Note that Google Cloud Pub/Sub is just one the Pub/Subs Watermill supports. You could easily change this to another supported Pub/Sub. Kind of like an ORM would work with MySQL, PostgreSQL, and SQLite.)

logger := watermill.NewStdLogger(false, false)

publisher, err := googlecloud.NewPublisher(
 googlecloud.PublisherConfig{
 ProjectID: cfg.PubSubProjectID,
 },
 logger,
)

Full source

Publisher works with messages, meaning you must marshal events (structs) into bytes and choose what topics to publish them to. It’s very common to use the same marshaling for all events and topics that follow some convention, like the event name being part of them.

We will use the EventBus component to simplify the publishing API. You can think of it as a high-level wrapper on the publisher (as you can see, it’s the first argument). You pass the configuration options once and then can publish events with a single method call.

eventBus, err := cqrs.NewEventBusWithConfig(
 publisher,
 cqrs.EventBusConfig{
 GeneratePublishTopic: func(params cqrs.GenerateEventPublishTopicParams) (string, error) {
 return params.EventName, nil
 },
 Marshaler: cqrs.JSONMarshaler{},
 Logger: logger,
 },
)

Full source

The configuration takes a Marshaler, so we use the cqrs.JSONMarshaler{} (all messages will be marshaled to JSON).

The GeneratePublishTopic function returns the topic’s name based on the available parameters. Instead of passing a topic directly to Publish, we define this function to determine the topic based on the message. The EventBus calls this function every time a message is published. In this case, we’ll use the params.EventName. So, if you consider a struct like this:

type PostViewed struct {
 PostID int `json:"post_id"`
}

Full source

It will be published on the PostViewed topic. (The chosen marshaler provides a way to extract the event name).

Publishing events using the event bus is trivial. Thanks to the setup of marshaler and GeneratePublishTopic, we pass the event struct to Publish, and the rest happens behind the scenes. In the HTTP handler, we can use something like this:

event := PostViewed{
 PostID: post.ID,
}

err = h.eventBus.Publish(ctx, event)

Full source

Subscribing to events

I decided to make the HTTP endpoints just publish the events. The event handlers update the post’s stats in the database asynchronously. This way, the client doesn’t need to wait for the changes to be applied, and the view will be eventually updated via SSE.

We need two event handlers to update the stats in the database. The first updates the views count, and the second updates the reactions count.

The CQRS component used for subscribing to events is the EventProcessor. As with the EventBus, some setup needs to be done initially. But thanks to this, writing the handlers later will be very pleasant. The idea behind it is similar to the EventBus, but it’s the other way around.

First, create a Router. It’s a similar concept to the HTTP routers you’re familiar with. The component runs in the background and routes messages to handlers.

router, err := message.NewRouter(message.RouterConfig{}, logger)

Full source

Similarly to HTTP routers, Watermill’s router supports middlewares. For example, you can add the Recoverer middleware so panics in handlers don’t blow up your server.

router.AddMiddleware(middleware.Recoverer)

Now we can create the EventProcessor.

eventProcessor, err := cqrs.NewEventProcessorWithConfig(
 router,
 cqrs.EventProcessorConfig{
 GenerateSubscribeTopic: func(params cqrs.EventProcessorGenerateSubscribeTopicParams) (string, error) {
 return params.EventName, nil
 },
 SubscriberConstructor: func(params cqrs.EventProcessorSubscriberConstructorParams) (message.Subscriber, error) {
 return googlecloud.NewSubscriber(
 googlecloud.SubscriberConfig{
 ProjectID: cfg.PubSubProjectID,
 GenerateSubscriptionName: func(topic string) string {
 return fmt.Sprintf("%v_%v", topic, params.HandlerName)
 },
 },
 logger,
 )
 },
 Marshaler: cqrs.JSONMarshaler{},
 Logger: logger,
 },
)

Full source

The first argument is the router. It’s similar to how the EventBus “wrapped” the publisher.

Then comes the config. The Marshaler and GenerateSubscribeTopic are the same concepts as in the EventBus. The only difference is they come at the other end of the Pub/Sub. In the EventBus, the marshaler marshals the message, and the function decides to which topic to publish it. Here, the Marshaler unmarshals the message back on the struct, and GenerateSubscribeTopic decides to which topic to subscribe to.

SubscriberConstructor is what the name says: it returns a new Subscriber. You may wonder, why not use a single subscriber, as we did with the publisher in the EventBus?

Publishing messages is straightforward: you marshal a struct, send the bytes to a topic, and you’re done. Subscribing is where things get more interesting. For example, you run two replicas of the same service. How do you ensure that only one replica receives a message from the Pub/Sub?

The strategy depends on the Pub/Sub. In Google Cloud Pub/Sub, you use a single “subscription” bound to a topic and share it among the replicas. That’s why having a subscriber constructor is helpful in this context. It allows us to specify what subscription to use for each event type. In this example, the subscription joins the topic name with the handler name. For example, PostViewed_UpdateViews.

As promised, with the setup done, adding message handlers is quite simple. Note that the functions are generic (with inferred types), so you work with strongly typed events! The handler name is used to generate the subscription name, so it needs to be unique within handlers.

err = eventProcessor.AddHandlers(
 cqrs.NewEventHandler(
 "UpdateViews",
 func(ctx context.Context, event *PostViewed) error {
 return repo.UpdatePost(ctx, event.PostID, func(post *Post) {
 post.Views++
 })
 },
 ),
 cqrs.NewEventHandler(
 "UpdateReactions",
 func(ctx context.Context, event *PostReactionAdded) error {
 return repo.UpdatePost(ctx, event.PostID, func(post *Post) {
 post.Reactions[event.ReactionID]++
 })
 },
 ),
)

Full source

Note

If you’re curious about the UpdatePost method, check Robert’s article on the repository pattern.

Tip

Be careful when refactoring

The names used for creating the topics and subscriptions are essential. If they were accidentally changed, you could lose messages.

You should not change the event or handler names once they’re running in production. If you need to, consider creating a new event or handler. For example, PostViewedV2.

The last part is running the router, just like you would run an HTTP server.

go func() {
 err := router.Run(context.Background())
 if err != nil {
 panic(err)
 }
}()

Full source

Note

To learn more about message routing, see the Consumer Groups Example.

Publishing PostStatsUpdated

We’ll use one more event to trigger the SSE updates: PostStatsUpdated. It includes the post’s ID and a record of what has been updated (views or the reaction ID).

type PostStatsUpdated struct {
 PostID int `json:"post_id"`
 ViewsUpdated bool `json:"views_updated"`
 ReactionUpdated *string `json:"reaction_updated"`
}

Full source

Note

Since the release of this post, I realized the naive approach of getting the post from the database on each update doesn’t scale well (hundreds of SELECT queries on each update, depending on how many visitors we have). I’ve updated the example so the PostStatsUpdated event includes all stats. This way, the SSE endpoint doesn’t need to query the database at all, except for the initial call. You can see how big was the impact on the database load.

I left the original version in the code snippets for simplicity. You can see the changes on GitHub.

After updating the post, both handlers should publish the PostStatsUpdated event.

err = eventProcessor.AddHandlers(
 cqrs.NewEventHandler(
 "UpdateViews",
 func(ctx context.Context, event *PostViewed) error {
 err = repo.UpdatePost(ctx, event.PostID, func(post *Post) {
 post.Views++
 })
 if err != nil {
 return err
 }

 statsUpdated := PostStatsUpdated{
 PostID: event.PostID,
 ViewsUpdated: true,
 }

 return eventBus.Publish(ctx, statsUpdated)
 },
 ),
 cqrs.NewEventHandler(
 "UpdateReactions",
 func(ctx context.Context, event *PostReactionAdded) error {
 err := repo.UpdatePost(ctx, event.PostID, func(post *Post) {
 post.Reactions[event.ReactionID]++
 })
 if err != nil {
 return err
 }

 statsUpdated := PostStatsUpdated{
 PostID: event.PostID,
 ReactionUpdated: &event.ReactionID,
 }

 return eventBus.Publish(ctx, statsUpdated)
 },
 ),
)

Full source

SSE Router

It’s time to implement the SSE endpoints. Watermill also provides an SSE component that works well with other internals.

The main component is called SSE Router, and the idea behind it is pretty simple. When you call its AddHandler method, it subscribes to the given topic in the configured subscriber. The method returns a regular HTTP handler you can use with any HTTP router you want. Whenever a message appears in the chosen topic, it will be propagated in a fan-out fashion to all running SSE endpoints within.

First, let’s create the SSERouter (it comes from the watermill-http package — use the v2 version!).

The config requires an UpstreamSubscriber: you need to subscribe to a Pub/Sub that contains the events. We use Google Cloud Pub/Sub again. I’ll explain the details of the configuration a bit later.

We use a string marshaler as we’re going to return plain HTML.

subscriber, err := googlecloud.NewSubscriber(googlecloud.SubscriberConfig{
 // ...
}, logger)
if err != nil {
 panic(err)
}

sseRouter, err := http.NewSSERouter(http.SSERouterConfig{
 UpstreamSubscriber: subscriber,
 Marshaler: http.StringSSEMarshaler{},
}, logger)
if err != nil {
 panic(err)
}

Full source

Then, you need to implement the StreamAdapter interface:

type StreamAdapter interface {
 InitialStreamResponse(w http.ResponseWriter, r *http.Request) (response interface{}, ok bool)
 NextStreamResponse(r *http.Request, msg *message.Message) (response interface{}, ok bool)
}

Full source

These two methods are very similar. The first one is how you respond to the initial HTTP request. If needed, it lets you return an error and write it to the ResponseWriter. This is important because as soon as you write any data, it’s too late to change the response code or the headers. So, InitialStreamResponse is where you handle things like validation or authentication. If any errors happen, return ok equal false to stop the handler. Otherwise, what you return becomes the first event sent to the client.

NextStreamResponse is called for each incoming Message. You can return a response to be sent to the SSE clients that use this endpoint. Or you can skip the message (again, return ok equal false).

By default, whatever you return as the response is marshaled to JSON with event type data. You can override this with a custom marshaler, as we did here. You can also return the ServerSentEvent struct, which lets you explicitly specify the Event and Data fields.

In our case, InitialStreamResponse simply returns the post’s response.

func (s *statsStream) InitialStreamResponse(w http.ResponseWriter, r *http.Request) (response interface{}, ok bool) {
 postIDStr := r.PathValue("id")
 postID, err := strconv.Atoi(postIDStr)
 if err != nil {
 w.WriteHeader(http.StatusBadRequest)
 w.Write([]byte("invalid post ID"))
 return nil, false
 }

 resp, err := s.getResponse(r.Context(), postID, nil)
 if err != nil {
 w.WriteHeader(http.StatusInternalServerError)
 w.Write([]byte(err.Error()))
 return nil, false
 }

 return resp, true
}

Full source

NextStreamResponse is similar, but it also checks if the post’s ID in the event matches the one in the URL. If not, it skips the message. It means a post has been updated, but not the one this endpoint returns.

func (s *statsStream) NextStreamResponse(r *http.Request, msg *message.Message) (response interface{}, ok bool) {
 postIDStr := r.PathValue("id")
 postID, err := strconv.Atoi(postIDStr)
 if err != nil {
 fmt.Println("invalid post ID")
 return nil, false
 }

 var event PostStatsUpdated
 err = json.Unmarshal(msg.Payload, &event)
 if err != nil {
 fmt.Println("cannot unmarshal: " + err.Error())
 return "", false
 }

 if event.PostID != postID {
 return "", false
 }

 resp, err := s.getResponse(r.Context(), postID, &event)
 if err != nil {
 fmt.Println("could not get response: " + err.Error())
 return nil, false
 }

 return resp, true
}

Full source

As you can see, there’s no usual error handling here. The best we can do is log the error and return false to skip the message. The handler already replied with 200 OK, so it’s too late to change the status code. Alternatively, we could return a custom response with the error message to be displayed to the user.

With the stream adapter ready, we can create a handler on the SSE Router:

statsHandler := sseRouter.AddHandler("PostStatsUpdated", &statsStream{repo: repo})

Full source

The first argument here is the topic to listen to. The returned value is a ready-to-use http.HandlerFunc.

Most HTTP libraries and frameworks are compatible with net/http, so you can use it with whatever HTTP router you like. I use echo, so there’s a small conversion of the path value needed:

e.GET("/posts/:id/stats", func(c echo.Context) error {
 postID := c.Param("id")
 c.Request().SetPathValue("id", postID)

 statsHandler(c.Response(), c.Request())
 return nil
})

Full source

Finally, you need to run the SSE router in a separate goroutine:

go func() {
 err := sseRouter.Run(context.Background())
 if err != nil {
 panic(err)
 }
}()

Full source

And that’s it! Whenever a message is published on the PostStatsUpdated topic, the SSE Router propagates it to all clients listening to the GET /posts/:id/stats endpoint.

The handlers created by the SSE Router already handle all implementation details, so you don’t need to worry about setting the headers.

When called without an Accept header or with a value other than text/event-stream, the handlers act as regular GET handlers, returning the response from InitialStreamResponse. Creating an EventStream in JavaScript automatically passes the header for you, but keep this in mind when debugging your endpoints with a tool like curl!

# Regular HTTP response
curl localhost:8080/posts/1/stats

# SSE response
curl -H "Accept: text/event-stream" localhost:8080/posts/1/stats

Configuring the Subscriber

Remember the part that we want each event to be processed only by one service replica at a time? In the case of events used for SSE, you need a counterintuitive approach: all subscribers need to process each event, as the SSE endpoints will be running across all of your service instances.

In other words, in this case, each replica should have its own subscription that’s not shared with anyone. For Google Cloud Pub/Sub, an easy way to do it is to generate a unique subscription name on the service’s startup. For example, using a “short UUID” would generate subscription names like PostStatsUpdated_lkcNowPZ99M123xPwqcxp1.

Keep in mind this can have some impact on your Pub/Sub. For example, when using Google Cloud Pub/Sub, it’s best to set the expiration policy for such subscriptions for one day, so they’re deleted when no longer used. There’s a hard limit of 10,000 subscriptions; you could quickly hit it this way.

subscriber, err := googlecloud.NewSubscriber(
 googlecloud.SubscriberConfig{
 ProjectID: cfg.PubSubProjectID,
 GenerateSubscriptionName: func(topic string) string {
 return fmt.Sprintf("%v_%v", topic, watermill.NewShortUUID())
 },
 SubscriptionConfig: pubsub.SubscriptionConfig{
 ExpirationPolicy: time.Hour * 24,
 },
 },
 logger,
)

Full source

If that sounds like a lot to consider, it’s because there is! Running production-grade Event-Driven systems comes with many advantages, but it’s not trivial. That’s why I go into detail here, so you know what to expect in production, not just in toy examples. (If that sounds like something you’d like to learn in-depth, see our Go Event-Driven training.)

htmx

The last piece of the puzzle is the client-side code.

In the example, we use htmx, a library that lets you make AJAX requests with HTML attributes. It also supports SSE with an extension. The templating in the snippets below comes from templ.

<div hx-ext="sse" sse-connect={ "/posts/" + post.ID + "/stats" } sse-swap="data"></div>

Full source

The sse-swap attribute is the event type to look for from the stream. By default, Watermill’s SSE endpoints use data, so that’s what we use. Every time an event is received from the /posts/:id/stats endpoint, its payload will be injected inside the div. (Remember, our events are HTML.)

We also use htmx to send the reaction form asynchronously (a classic POST AJAX request). In this case, we use hx-swap="outerHTML", which replaces the entire form with the response from the server. It’s a button with a ✅ “check” suggesting that the reaction has been added. The SSE will eventually update the stats. (Although there might be a slight delay. If you care about UX, returning a “fake” number increase could make sense here.)

<form hx-post={ "/posts/" + postID + "/reactions"} hx-swap="outerHTML">
 <input type="hidden" name="reaction_id" value={ reaction.ID } />
 <button type="submit" class={"btn", "btn-outline-secondary", "m-1", templ.KV("animated", reaction.JustChanged)}>
 <span class="emoji">{ reaction.Label }</span>
 <span class="counter">{ reaction.Count }</span>
 </button>
</form>

Full source

Animations

If you’re used to working with Single Page Applications, using htmx might initially feel weird. For example, consider animating an element that has just been updated. In a classic SPA, you would get the JSON response from the SSE endpoint, compare the values with what’s in the “model”, and decide whether to animate the element.

While htmx allows for “hooks” after the request is done, it’s probably not what you want. Instead, you need to adjust your mental model a bit. The server code is the source of truth in this setup. There is no “client” that decides how to display things.

In this example, I use a CSS class to mark the updated element. The class includes an animation that pops up the element for a moment. The server code decides whether to add the class or not (based on the data in the incoming event). (templ.KV is how you add a class conditionally using templ. It will be present if stats.Views.JustChanged is true.)

<div class={ "d-flex", "align-items-center", templ.KV("animated", stats.Views.JustChanged)}>
 <span class="me-1">👁️</span>
 <small class="text-muted">{ stats.Views.Count + " views" }</small>
</div>

Full source

Other things to consider

Two kinds of SSE endpoints

How you return events from SSE endpoints is totally up to you. Here are two ways that make sense in different scenarios.

An endpoint that returns the same data model initially and on every update. With each triggered update, you kind of “refresh” the model, perhaps embedded on the website. This is what we use in the example above.
An endpoint that returns nothing initially and then keeps sending unique updates as they happen. You can append each new event to some list, for example. It’s how you would implement notifications or a web chat.

At-least-once delivery

When working with virtually any Pub/Sub, you must be aware of the “at-least-once” delivery guarantee. You may receive the same message twice because of network issues or your server going down at the wrong moment.

Don’t try to work around this. Instead, embrace that this can happen and design your handlers to be idempotent. It means that processing the message twice (or more) has the same effect as processing it once.

In the example above, we don’t guard against it. If the same message was processed twice, it would add an extra view or reaction in the database. It’s not a big deal in this case, and we can live with it. One way to prevent it would be to store the processed message IDs in the database and check it on each update.

Watch out for HTTP/1.1

In modern browsers, there’s a limit of 6 open connections per server over HTTP/1.1, which can be a big issue when using SSE. Your website won’t work well if someone opens it in several tabs.

For best results, use SSE with HTTP/2 where this limit doesn’t apply. Most modern web servers support HTTP/2, so make sure you enable it.

Local environment tricks

Here are two tips unrelated to SSE that might be useful for running your app locally.

Mounting /go/pkg and go cache

In the docker-compose definition, you can mount the /go/pkg and /go-cache directories to speed up the build process. This way, you don’t have to download the dependencies whenever you rebuild the container.

services:
 server:
 # ...
 volumes:
 - go_pkg:/go/pkg
 - go_cache:/go-cache

volumes:
 go_pkg:
 go_cache:

Full source

Reflex for regenerating templ and rebuilding the server

Reflex is my go-to tool for live code reloading. (See my post on the dev environment setup.) When working with templ, you can use a configuration like the one below to regenerate the templates and rebuild the server after every change.

It’s a great way to speed up your development process, so you don’t need to keep switching between code, the terminal, and the browser.

-r '(\.go$|go\.mod$)' -s go run .
-r '\.templ$' templ generate

Full source

Go build something!

That should be all the theory you need to build something cool. If you have any questions, let me know in the comments.

Once again, the complete source code is on GitHub You can run it locally with docker-compose up.

Give this stack a try; I had lots of fun working with it. Good luck!

Making Games in Go for Absolute Beginners

Fri, 24 Nov 2023 00:00:00 +0100

Here’s a rant I often see in developer communities:

I used to love programming because I like building stuff. But my full-time job killed my passion. I spend more time in meetings, fighting over deadlines, and arguing in reviews than working with code. Am I burned out? Is there hope, or do I need a new hobby?

Sounds familiar? No wonder we keep looking forward to using a new framework or database — we’re bored. So here’s my tip to cure some of your burnout: Start coding for fun again and reclaim your hobby. And what is more fun than creating your own world with code?

Wanting to make my own video game pushed me into programming 20 years ago. I didn’t make a career out of it, even though I created some small games using different tech stacks. I keep returning to it, and for me, it’s the most fun activity related to coding. Especially if done with friends over a weekend for a game jam.

This post is my attempt at getting you to write games from scratch. (And if you want to pick up a new language too, check Go in One Evening!)

Why Go? I like it, and Ebitengine, the library I will show here, is great to work with. Like many Go libraries, it helps you do what you need and stays out of the way — it’s not a framework.

Of course, a bigger game engine simplifies many things but also comes with bloat and a specific approach. A big part of why making games is fun is you get to do things your way and don’t need to care if it’s the right way. And it’s a great exercise to learn how games work.

First, a warning. Once you understand how games work, some of the magic is gone when playing them. You will start wondering, “How does it work?” instead of enjoying the game. Thankfully, making games compensates for this.

Note

Disclaimer: I’m a hobbyist game developer. I do it as it’s tons of fun, but I never worked on a professional game. Expect personal experience here. I might show some code that would be dumb when used in a “real” game (performance-wise, for example). Let me know in the comments if you find any errors!

The basics

I remember having a hard time understanding how exactly games work. How do you make the objects in the game do what they do? It’s more obvious if you use a game engine, but what if you start from scratch?

You probably know that a movie is a very long collection of images, displayed one after the other so fast that our brains can’t tell the difference. Usually, 24 or 60 images (frames) per second (FPS).

A video game is the same idea, but the images (frames) don’t exist beforehand, so you must generate (draw) them on the fly. And you can press some buttons to change how the next frame looks.

This may seem too simplistic, right? “What? But how do you make a character jump?” You move the image up the screen over time and then back down. And you stop it from moving when it “touches” other images. 2D games are basically made of images drawn on top of each other.

I was surprised to learn that games are basically infinite loops. You keep drawing images on the screen.

for {
 DrawFrame()
}

Besides drawing, you need to update the logic of the game. It includes things like checking buttons pressed, updating the player’s health, and checking collisions. The logic decides what images to draw and where, and the Draw function does it. Each time the logic is updated is known as a tick.

for {
 // One tick
 UpdateLogic()
 DrawFrame()
}

I know this looks hardly exciting, but let this idea sink in: your game does these two things over and over, 60 times per second. Just a single thread with this infinite loop (for now, forget goroutines exist).

Even though the idea is simple, keeping the loop running at the same speed on all systems is not trivial. Ebitengine takes care of this for you, running the game at 60 ticks per second.

Logic is the part you should be familiar with: it’s regular Go code. Drawing images is where things get more complex, and that’s why you use Ebitengine. It helps with drawing images, playing sounds, and checking inputs. How the logic works is mainly on you.

Ebitengine provides a Game interface you need to implement.

type Game interface {
 Update() error
 Draw(screen *ebiten.Image)
 Layout(outsideWidth, outsideHeight int) (screenWidth, screenHeight int)
}

The Update and Draw functions work as in the loop above. Layout is used for scaling the game screen — you don’t need to worry about it now.

Update is called every tick and updates the game logic. If an error is returned, the game quits.

Draw is called every frame. The screen argument is an image you draw on, like a canvas.

Setting up the project

First, let’s kick off the boring stuff: create a module and add Ebitengine (formerly called Ebiten, hence the package name).

go mod init game
go get github.com/hajimehoshi/ebiten/v2

Here’s the minimal code you need to start a game. It’s just an empty implementation of the Game interface. Running it should show a blank black window.

package main

import "github.com/hajimehoshi/ebiten/v2"

type Game struct{}

func (g *Game) Update() error {
 return nil
}

func (g *Game) Draw(screen *ebiten.Image) {

}

func (g *Game) Layout(outsideWidth, outsideHeight int) (screenWidth, screenHeight int) {
 return outsideWidth, outsideHeight
}

func main() {
 g := &Game{}

 err := ebiten.RunGame(g)
 if err != nil {
 panic(err)
 }
}

Note

I’m pretty liberal with error handling in these examples. We’re just prototyping and having fun, so I’m okay with raising a panic. With most errors, you can either crash the game or log it and continue.

Loading assets

If your graphic skills are similar to mine (the term “programmer art” exists for a reason 🙃), you may prefer using ready assets when prototyping.

Unless you want to create them yourself, I encourage you to play with free-to-use assets. Personally, I love the collection from Kenney. Every time I look for inspiration, I browse the 2D assets there and find something that looks like a cool game to make. I’ll use the space shooter pack below to make a trivial Asteroids clone.

To load the assets in code, use embed — it’s like it was meant for making games. You know the paths are right on compile time. You also don’t need to care about the distribution, as the binary contains the assets. It works even for mobile apps!

You will likely have many assets, so consider embedding the entire directory:

import "embed"

//go:embed assets/*
var assets embed.FS

You can now load the images from the assets collection. I’m using the must pattern here and keep sprites as global variables for simplicity. (A sprite is just another name for a 2D graphic.)

Note the empty "image/png" import used for decoding.

import (
 "image"
 _ "image/png"

 "github.com/hajimehoshi/ebiten/v2"
)

var PlayerSprite = mustLoadImage("assets/player.png")

func mustLoadImage(name string) *ebiten.Image {
 f, err := assets.Open(name)
 if err != nil {
 panic(err)
 }
 defer f.Close()

 img, _, err := image.Decode(f)
 if err != nil {
 panic(err)
 }

 return ebiten.NewImageFromImage(img)
}

Drawing Images

All the drawing happens in the Draw method.

func (g *Game) Draw(screen *ebiten.Image) {

}

The screen argument is the image displayed every frame (after the method returns). Your job is to draw other images (or text) on it.

The key method of ebiten.Image is DrawImage, which draws another image onto it.

screen.DrawImage(PlayerSprite, nil)

The first argument is the image you want to draw. The second is the ebiten.DrawImageOptions struct that decides where and how the image is drawn. Passing nil keeps the default options.

Here’s the result: the player’s spaceship is drawn in the screen’s top-left corner.

The drawing options struct has a few fields; the one to focus on first is the GeoM matrix. The docs go into detail on the math behind it, and math is super helpful in making games. But if I just wanted to make a game, the last thing I’d like to read about is algebra, so I’ll try to simplify it here.

Do you know how when you copy and paste an image in a graphics editor, the pasted part stays selected, and you can keep moving and transforming it? That’s how I see DrawImage. You can think of DrawImageOptions as a “cursor” you can use to position and change the image.

Drawing uses X and Y coordinates, with the Y axis going downward. By default, everything is drawn at the (0, 0) point, so the top-left corner of the base image.

To move the image, use GeoM.Translate:

op := &ebiten.DrawImageOptions{}
op.GeoM.Translate(150, 200)
screen.DrawImage(PlayerSprite, op)

This Translate call moves the image 150 pixels right and 200 pixels down. You can use negative numbers to go left and up, respectively. (If you move too far away, you may not see the image on the screen).

Use GeoM.Rotate to rotate the image:

op := &ebiten.DrawImageOptions{}
op.GeoM.Rotate(45.0 * math.Pi / 180.0)
screen.DrawImage(PlayerSprite, op)

This rotates the image 45° clockwise. (The unit is radians. If you prefer working with degrees, use the degrees * math.Pi / 180.0 formula to get them.)

Can you see how the image is partially off-screen now? That’s because the pivot point (the point “around” which the image is being rotated) is the top-left corner of the image. I’ll show how to work around this in a bit.

Using negative values rotates the image counter-clockwise.

Finally, there’s GeoM.Scale, which lets you draw smaller or bigger images:

op := &ebiten.DrawImageOptions{}
op.GeoM.Scale(2, 2)
screen.DrawImage(PlayerSprite, op)

This Scale call makes the image twice as big horizontally and vertically.

Passing negative values flips the image horizontally or vertically. For example, to draw the image upside-down, use:

op.GeoM.Scale(1, -1)

You now know the basic tools to control images in a game: their position, rotation, and scale.

You can combine all of them in a single DrawImage call. There’s a caveat, though: the order matters. If you use Translate first and then apply the rotation or scale, you may not see what you expect because the change will be applied to the new position.

To illustrate this, let’s see how to rotate the image around its center (a common use case).

You need to move the pivot point to the image’s center. You can find it simply by dividing the image’s width and height by half. (Calculating stuff based on the image size is also super common.)

width := PlayerSprite.Bounds().Dx()
height := PlayerSprite.Bounds().Dy()

halfW := float64(width / 2)
halfH := float64(height / 2)

You need to move the image by the negative values first. This is so the image’s center aligns with the origin point (0, 0). Then, apply the rotation and move the image “back” by the same amount.

op.GeoM.Translate(-halfW, -halfH)
op.GeoM.Rotate(45.0 * math.Pi / 180.0)
op.GeoM.Translate(halfW, halfH)

It might be easier to grasp using this animation of all steps (scaled three times):

Changing colors

Another useful option is changing the sprite’s color. You can use it to draw an image that looks like the sprite but with a solid color. Of course, this makes the most sense if you use sprites with a transparent background.

The syntax is similar, although you need to use the recently added colorm package.

op := &colorm.DrawImageOptions{}
cm := colorm.ColorM{}
cm.Translate(1.0, 1.0, 1.0, 0.0)
colorm.DrawImage(screen, PlayerSprite, cm, op)

The first three arguments of Translate are red, green, and blue values and go from 0.0 to 1.0 (0% to 100%). The last one is alpha (transparency).

You can control the alpha field to make sprites transparent, creating cool effects like adding a shadow. To keep the transparent background unchanged, use Scale instead of Translate. Keep the first three arguments at 1.0, so they don’t change.

op := &colorm.DrawImageOptions{}
cm := colorm.ColorM{}
cm.Scale(1.0, 1.0, 1.0, 0.5)
colorm.DrawImage(screen, PlayerSprite, cm, op)

Combining options

You can do more with DrawImageOptions, but this should be all you need to start making games.

For example, here’s an evolution animation I created in airplanes:

I really like how it turned out. The best part is that I didn’t use graphic assets for the animation. It’s all done with DrawImageOptions and goes like this:

Change the color of the sprite to white.
Create a new sprite (also white) on top of the current one, with the scale set to (0, 0) (invisible).
Scale the new sprite to scale (1, 1) over time.
Scale the old sprite to scale (0, 0) over time.
Remove the white color.

Can you notice how the shadow evolves as well? It’s the same idea but using a gray sprite and some transparency.

Are you experienced engineer who wants to learn Go basics?
You don't become an engineer by watching videos.
Learn Go hands-on by building real projects.

The logic

To make the game do anything interesting, you need to fill in the Update method. You keep all logic there while the Draw method draws images (on the screen or on top of each other).

The basic idea of the game logic is simple: the Game struct holds some state, and the Update method changes this state. Draw reads the state and draws images based on it.

A classic example of a game state is the player’s position. In 2D games, this will usually be a pair of (X, Y) values, also known as a vector.

type Vector struct {
 X float64
 Y float64
}

type Game struct {
 playerPosition Vector
}

The zero-value of Vector is (0, 0), like any struct in Go. Let’s initialize the game state with a predefined value:

g := &Game{
 playerPosition: Vector{X: 100, Y: 100},
}

All left to do is to update the Draw method so it draws the player’s sprite at the given position. This shouldn’t be surprising:

op := &ebiten.DrawImageOptions{}
op.GeoM.Translate(g.playerPosition.X, g.playerPosition.Y)
screen.DrawImage(PlayerSprite, op)

Note how we call Translate always with the current position of the player. Each time Draw is called, the screen is empty, and the images are drawn from scratch. (Ebitengine might optimize this, but we don’t need to care now.)

Movement

To move the player to the right, keep increasing the X position in Update.

Do you remember that Update is called in an infinite loop? This becomes important now, as you need to decide how fast to move the player.

func (g *Game) Update() error {
 speed := 5.0
 g.playerPosition.X += speed

 return nil
}

speed is the number of pixels the position changes in a single tick (one Update call). With the default 60 ticks per second, the image would move 300 pixels per second to the right.

It may be easier to keep the speed as “pixels per second” instead of “pixels per tick”. In this case, calculate it like this:

// Move 300 pixels per second
speed := float64(300 / ebiten.TPS())

Controls

Let’s now allow moving the player using the arrow keys. The idea stays the same: we need to update the position. Not every tick, though, but only if a key is pressed.

func (g *Game) Update() error {
 speed := 5.0

 if ebiten.IsKeyPressed(ebiten.KeyDown) {
 g.playerPosition.Y += speed
 }
 if ebiten.IsKeyPressed(ebiten.KeyUp) {
 g.playerPosition.Y -= speed
 }
 if ebiten.IsKeyPressed(ebiten.KeyLeft) {
 g.playerPosition.X -= speed
 }
 if ebiten.IsKeyPressed(ebiten.KeyRight) {
 g.playerPosition.X += speed
 }

 return nil
}

ebiten.IsKeyPressed returns true if the given key is pressed in the current “tick” (remember, Update is called about 60 times per second by default).

The other function you might need is ebitenutil.IsKeyJustPressed, which returns true just in the tick the player presses the key. It’s the difference between “hold space to keep jumping” and “even if you keep holding space, you jump just once”. Note it’s in the ebitenutil package not ebiten, in contrast to IsKeyPressed.

Note how we don’t use if-else — thanks to this, the player can move diagonally by holding two arrows at a time.

Tip

Moving diagonally this way is naive and faster than moving in one direction at a time. If that’s not what you want, you need to adjust for it.

var delta Vector

if ebiten.IsKeyPressed(ebiten.KeyDown) {
 delta.Y = -speed
}
if ebiten.IsKeyPressed(ebiten.KeyUp) {
 delta.Y = speed
}
if ebiten.IsKeyPressed(ebiten.KeyLeft) {
 delta.X = -speed
}
if ebiten.IsKeyPressed(ebiten.KeyRight) {
 delta.X = speed
}

// Check for diagonal movement
if delta.X != 0 && delta.Y != 0 {
 factor := speed / math.Sqrt(delta.X*delta.X+delta.Y*delta.Y)
 delta.X *= factor
 delta.Y *= factor
}

g.playerPosition.X += delta.X
g.playerPosition.Y += delta.Y

Timers

A very common need in games is changing the logic with time. For example, you want the enemy object to move for two seconds to the right and then back to the left. Or you want the boss to cast a spell every five seconds.

You can’t use a regular way to measure time, using time.Now() and time.Since() due to how the Update method works (running at a constant speed of 60 ticks per second).

The idea is to keep counting ticks that have happened since the timer started. Then, you take some action and reset it.

I like to keep the Timer as a separate struct with a nice API.

type Timer struct {
 currentTicks int
 targetTicks int
}

func NewTimer(d time.Duration) *Timer {
 return &Timer{
 currentTicks: 0,
 targetTicks: int(d.Milliseconds()) * ebiten.TPS() / 1000,
 }
}

func (t *Timer) Update() {
 if t.currentTicks < t.targetTicks {
 t.currentTicks++
 }
}

func (t *Timer) IsReady() bool {
 return t.currentTicks >= t.targetTicks
}

func (t *Timer) Reset() {
 t.currentTicks = 0
}

And here’s a short example of using it:

g := &Game{
 attackTimer: NewTimer(5 * time.Second),
}

// ...

func (g *Game) Update() error {
 g.attackTimer.Update()
 if g.attackTimer.IsReady() {
 g.attackTimer.Reset()

 // Execute the attack!
 }
}

Game Objects

Keeping the player’s position directly in the Game struct is like using global variables. It would be challenging to maintain non-trivial games this way.

Encapsulation is a good improvement: having building blocks you can use instead of dealing with the details. A simple game object (or entity or whatever you want to call it) can be a struct with a position and sprite.

There are many approaches to organizing the game logic, and you can use whatever makes sense to you. An easy way to start is to have each game object expose its own Update and Draw methods, and call them in the Game’s Update and Draw.

type Player struct {
 position Vector
 sprite *ebiten.Image
}

func NewPlayer() *Player {
 return &Player{
 position: Vector{X: 100, Y: 100},
 sprite: PlayerSprite,
 }
}

func (p *Player) Update() {

}

func (p *Player) Draw(screen *ebiten.Image) {
 op := &ebiten.DrawImageOptions{}
 op.GeoM.Translate(p.position.X, p.position.Y)
 screen.DrawImage(p.sprite, op)
}

Now the Game methods don’t concern about the details of the Player:

type Game struct {
 player *Player
}

func (g *Game) Update() error {
 g.player.Update()
}

func (g *Game) Draw(screen *ebiten.Image) {
 g.player.Draw(screen)
}

Let’s finish the Player implementation for this Asteroids clone.

The player should be spawned in the center of the screen. It’s easy to calculate if we know the screen’s width and height. First, define the screen size as constants.

const (
 ScreenWidth = 800
 ScreenHeight = 600
)

Now is the time we use the third method of ebiten.Game’s interface: the Layout. It should return the size of the game window.

func (g *Game) Layout(outsideWidth, outsideHeight int) (int, int) {
 return ScreenWidth, ScreenHeight
}

Now, back to the NewPlayer constructor to set the initial position. The center of the screen is half of the screen’s width and height (X and Y). But remember, the player’s sprite top-left corner will be drawn at the given position. To keep it precisely in the center, we must move it left and up by half the sprite’s width and height.

func NewPlayer() *Player {
 sprite := PlayerSprite

 bounds := sprite.Bounds()
 halfW := float64(bounds.Dx()) / 2
 halfH := float64(bounds.Dy()) / 2

 pos := Vector{
 X: ScreenWidth/2 - halfW,
 Y: ScreenHeight/2 - halfH,
 }

 return &Player{
 position: pos,
 sprite: sprite,
 }
}

Instead of moving the spaceship, the player should be able to rotate it. We can keep the rotation as float64 in the Player struct.

func (p *Player) Update() {
 speed := math.Pi / float64(ebiten.TPS())

 if ebiten.IsKeyPressed(ebiten.KeyLeft) {
 p.rotation -= speed
 }
 if ebiten.IsKeyPressed(ebiten.KeyRight) {
 p.rotation += speed
 }
}

Remember, the rotation unit is radians. 2π is a full rotation. I’ll stick to π / TPS, which means the player can rotate 180° per second.

Now, we need to update the Draw struct to take rotation into account (using the trick of rotating the image around its center).

func (p *Player) Draw(screen *ebiten.Image) {
 bounds := p.sprite.Bounds()
 halfW := float64(bounds.Dx()) / 2
 halfH := float64(bounds.Dy()) / 2

 op := &ebiten.DrawImageOptions{}
 op.GeoM.Translate(-halfW, -halfH)
 op.GeoM.Rotate(p.rotation)
 op.GeoM.Translate(halfW, halfH)

 op.GeoM.Translate(p.position.X, p.position.Y)

 screen.DrawImage(p.sprite, op)
}

Spawning objects

Following the game object idea, let’s create a Meteor object. We can load many assets and pick one randomly to make it more interesting.

var MeteorSprites = mustLoadImages("meteors/*.png")

type Meteor struct {
 position Vector
 sprite *ebiten.Image
}

func NewMeteor() *Meteor {
 sprite := MeteorSprites[rand.Intn(len(MeteorSprites))]

 return &Meteor{
 position: Vector{},
 sprite: sprite,
 }
}

The Draw method is pretty much the same as for Player. (As you can imagine, there’s room for improvement here to keep the common code in one place — more on this later).

Now, Game needs to keep track of meteors. It’s the same idea as with Player, except we now need a slice of objects. Let’s also add a spawn timer.

type Game struct {
 player *Player
 meteorSpawnTimer *Timer
 meteors []*Meteor
}

Update and Draw iterate over all meteors and call their respective methods. And every time the timer is ready, a new meteor is added to the slice.

func (g *Game) Update() error {
 g.player.Update()

 g.meteorSpawnTimer.Update()
 if g.meteorSpawnTimer.IsReady() {
 g.meteorSpawnTimer.Reset()

 m := NewMeteor()
 g.meteors = append(g.meteors, m)
 }

 for _, m := range g.meteors {
 m.Update()
 }

 return nil
}

func (g *Game) Draw(screen *ebiten.Image) {
 g.player.Draw(screen)

 for _, m := range g.meteors {
 m.Draw(screen)
 }
}

Tip

Note that the order of drawing matters. The entities you draw later in Draw will appear on top of the ones drawn earlier (if their positions overlap).

Details of the meteor

The meteor should spawn at the edge of the screen. Here’s one approach to do it.

// Figure out the target position — the screen center, in this case
target := Vector{
 X: ScreenWidth / 2,
 Y: SreenHeight / 2,
}

// The distance from the center the meteor should spawn at — half the width
r := ScreenWidth / 2.0

// Pick a random angle — 2π is 360° — so this returns 0° to 360°
angle := rand.Float64() * 2 * math.Pi

// Figure out the spawn position by moving r pixels from the target at the chosen angle
pos := Vector{
 X: target.X + math.Cos(angle)*r,
 Y: target.Y + math.Sin(angle)*r,
}

As I mentioned, math is often handy for making games! You don’t need to know how it works in depth; just know the basic operations like “move 100 pixels from position (100, 200) at 30° angle”. (There’s absolutely no shame in having to consult the internet or AI every time you do it. 😅)

Next, the meteor should keep moving toward the player’s position (the screen center).

// Randomized velocity
velocity := 0.25 + rand.Float64()*1.5

// Direction is the target minus the current position
direction := Vector{
 X: target.X - pos.X,
 Y: target.Y - pos.Y,
}

// Normalize the vector — get just the direction without the length
normalizedDirection := direction.Normalize()

// Multiply the direction by velocity
movement := Vector{
 X: normalizedDirection.X * velocity,
 Y: normalizedDirection.Y * velocity,
}

NewMeteor can store the movement in the struct, and then the Update becomes trivial:

func (m *Meteor) Update() {
 m.position.X += m.movement.X
 m.position.Y += m.movement.Y
}

A final touch is adding a random rotation to each meteor. You can calculate it in the constructor:

rotationSpeed := -0.02 + rand.Float64()*0.04,

Then, keep moving and rotating the meteor.

func (m *Meteor) Update() {
 m.position.X += m.movement.X
 m.position.Y += m.movement.Y
 m.rotation += m.rotationSpeed
}

Shooting Bullets

Shooting bullets shouldn’t be surprising by now, so I won’t describe it in detail. You can see the complete source in the repository.

We need to:

Keep a timer with a shooting cooldown.
Spawn a new bullet when a button is pressed.
Rotate the bullet and keep it moving in the direction it faces.

p.shootCooldown.Update()
if p.shootCooldown.IsReady() && ebiten.IsKeyPressed(ebiten.KeySpace) {
 p.shootCooldown.Reset()
 // Spawn the bullet
}

One new thing is having Player spawn bullets instead of Game. But Game needs to keep track of the bullets, similar to meteors.

The simplest way is to pass Game to Player’s constructor and keep it in the struct. Then, expose an AddBullet method from Game. It’s not ideal, as it creates a cross-dependency between Player and Game, but it’s good enough for now.

I decided to calculate the spawn point on the Player side and make the Bullet’s constructor take it as an argument.

bulletSpawnOffset := 50.0

bounds := p.sprite.Bounds()
halfW := float64(bounds.Dx()) / 2
halfH := float64(bounds.Dy()) / 2

spawnPos := Vector{
 p.position.X + halfW + math.Sin(p.rotation)*bulletSpawnOffset,
 p.position.Y + halfH + math.Cos(p.rotation)*-bulletSpawnOffset,
}

bullet := NewBullet(spawnPos, p.rotation)
p.game.AddBullet(bullet)

Collisions

Detecting collisions is a complex topic, although, at the basic level, the concept is as simple as iterating over all objects and checking if they intersect.

To make things easier, let’s introduce a Rect struct that represents a rectangle and makes it easy to check intersections.

type Rect struct {
 X float64
 Y float64
 Width float64
 Height float64
}

func NewRect(x, y, width, height float64) Rect {
 return Rect{
 X: x,
 Y: y,
 Width: width,
 Height: height,
 }
}

func (r Rect) MaxX() float64 {
 return r.X + r.Width
}

func (r Rect) MaxY() float64 {
 return r.Y + r.Height
}

func (r Rect) Intersects(other Rect) bool {
 return r.X <= other.MaxX() &&
 other.X <= r.MaxX() &&
 r.Y <= other.MaxY() &&
 other.Y <= r.MaxY()
}

Next, each game object exposes a Collider() Rect method.

func (p *Player) Collider() Rect {
 bounds := p.sprite.Bounds()

 return NewRect(
 p.position.X,
 p.position.Y,
 float64(bounds.Dx()),
 float64(bounds.Dy()),
 )
}

In the Game’s Update, iterate over all objects and check for collisions.

for i, m := range g.meteors {
 for j, b := range g.bullets {
 if m.Collider().Intersects(b.Collider()) {
 // A meteor collided with a bullet
 }
 }
}

for _, m := range g.meteors {
 if m.Collider().Intersects(g.player.Collider()) {
 // A meteor collided with the player
 }
}

When a bullet collides with a meteor, we want to destroy both objects. In our case, this means just dropping them using slice operations. If they’re gone from the list, the game won’t call their Update and Draw methods, so they effectively disappear (and the garbage collector handles the rest for us).

if m.Collider().Intersects(b.Collider()) {
 g.meteors = append(g.meteors[:i], g.meteors[i+1:]...)
 g.bullets = append(g.bullets[:j], g.bullets[j+1:]...)
}

A meteor colliding with the player means the game is over. Let’s restart the game in this case.

func (g *Game) Reset() {
 g.player = NewPlayer(g)
 g.meteors = nil
 g.bullets = nil
}

UI

Drawing UI is similar to drawing other sprites. Two rules of thumb:

It’s usually the last layer you draw, so it always stays on top.
Sometimes, you want it on a separate section of the screen. In this case, you want to draw the game not directly on the screen but on a smaller canvas that you eventually draw next to the UI. (You can create an empty image with ebiten.NewImage and use it as a “panel”).

A classic UI example is the score the player gets after shooting down a meteor. We can keep the score as an integer in the Game struct, increment it when a meteor collides with a bullet, and reset it to zero on the game over.

What’s left is drawing the score on the screen. First, you need to load a font, which is similar to loading a sprite. (I’m using Kenney Fonts.)

var ScoreFont = mustLoadFont("font.ttf")

func mustLoadFont(name string) font.Face {
 f, err := assets.ReadFile(name)
 if err != nil {
 panic(err)
 }

 tt, err := opentype.Parse(f)
 if err != nil {
 panic(err)
 }

 face, err := opentype.NewFace(tt, &opentype.FaceOptions{
 Size: 48,
 DPI: 72,
 Hinting: font.HintingVertical,
 })
 if err != nil {
 panic(err)
 }

 return face
}

Then, call text.Draw in Game’s Draw:

text.Draw(screen, fmt.Sprintf("%06d", g.score), ScoreFont, ScreenWidth/2-100, 50, color.White)

If you want to be precise with the position, use font.BoundString to get the bounds. In the example above, I just estimated something that looks good enough.

If you like this article, check out our materials on topics other than games.
You can start with our e-book on modern Go applications!

Go With The Domain Three Dots Labs

🔒 We do not send spam. You can unsubscribe at any time!

Other concepts

This article is already quite long, but I want to mention some ideas you may find helpful once you start making games.

Scenes

A scene is a standalone part of your game. For example, the welcome screen, the main menu, or the game itself.

In the meteors project, there’s a single Game scene. This could be extended so it’s possible to switch between scenes, and each has its own Update and Draw methods. (And manages its own game objects.)

Camera

I was initially intimidated by the camera concept, but it turned out to be a simple idea. Your game “world” can be a big area, and you often don’t want to show all of it at once. With the camera concept, you draw only part of the bigger image. You can move the camera, make it follow the player, or scale it up and down to achieve the zoom effect.

A trivial example needs just two things:

A Camera struct that keeps the position (and optionally scale and other fields).
An offscreen image (initialized with ebiten.NewImage).

type Game struct {
 camera *Camera
 offscreen *ebiten.Image

 player *Player
 enemies []*Enemy
}

func (g *Game) Draw(screen *ebiten.Image){
 g.offscreen.Clear()

 // Draw the game on the offscreen image
 g.player.Draw(g.offscreen)
 for _, e := range g.enemies {
 e.Draw(g.offscreen)
 }

 // Draw the offscreen image on the screen
 op := &ebiten.DrawImageOptions{}
 op.GeoM.Translate(-g.camera.X, -g.camera.Y)
 screen.DrawImage(g.offscreen, op)
}

As you update the camera’s position, the part of the offscreen image will also change.

Animations

An animation means changing the sprite of an object over time. Sometimes, just two or three sprites can create an animation that makes the game look more alive.

To implement it, create a slice of sprites, a timer, and a “current sprite index”. Then, increase the index as the timer resets.

func (p *Player) Update() {
 p.timer.Update()
 if p.timer.IsReady() {
 p.timer.Reset()

 p.index++
 if p.index >= len(p.sprites) {
 p.index = 0
 }

 p.sprite = p.sprites[p.index]
 }
}

Animation sprites often come as a single file with frames next to each other. You can use the SubImage method to extract a single frame from the loaded image.

You can also “animate” other parameters of an object, like the sprite’s color or scale.

Deploying to the web

It’s easy to deploy your game with WebAssembly and host it anywhere.

See more details in the docs and on GitHub.

Entity Component System

You probably noticed many common parts between the Player, Meteor, and Bullet game objects. It’s tempting to extract the common code so it’s easier to create new objects, and it definitely makes sense.

One way to do it is by keeping a generic GameObject struct from which other game objects inherit. In Go, you could use struct embedding with similar results.

Entity Component System (ECS) is another approach that prefers composition over inheritance. The idea is to keep common components like position, sprite, and collider as separate structs and then create game objects (entities) by combining them. All the logic is kept in systems that iterate over all entities and update them.

There are many ECS libraries for Go. My pick is donburi, and my airplanes game is based on it.

Debugging

As with any code, you will sometimes get stuck with hard-to-debug bugs. While using prints for debugging sometimes helps, I like adding visual helpers in place, sort of “debug mode.” It often shows in a very explicit way where the issue is.

You can make it appear and hide after a key so it doesn’t interfere with regular gameplay. Here’s an example debug mode I introduced in airplanes:

Ebitengine provides some utils for this:

ebitenutil.DebugPrintAt — to quickly draw a text.
The vector package has functions like StrokeLine, StrokeRect, and StroneCircle.

Go build something!

All this barely scratches the surface of Gamedev. I picked topics that should let you make games and learn the basics. Let me know if you’d like to see more posts on some topics.

To learn more about Ebitengine, the official examples are a great starting point.

My example project is on GitHub, and you can play it online. Feel free to fork the repository and iterate over it!

Here are some ideas to implement:

Add support for controls using a mouse or gamepad.
Make big meteors “break” into small ones instead of disappearing.
Have meteors drop power-ups that the player can collect.
Allow switching weapons.
Add AI spaceships that follow a more complex pattern than flying towards the player.
Add more scenes like the main menu and high scores.
Add sound effects. (It really makes a difference!)

If you can’t figure out how to create some mechanic you came up with (and it will happen a lot), try to figure out each step that needs to happen. For me, sketching using pen and paper often helps.

Now, it’s time to go and build something. Have fun!

Watermill 1.3 released, an open-source event-driven Go library

Mon, 25 Sep 2023 00:00:00 +0200

Hey, it’s been a long time!

We’re happy to share that Watermill v1.3 is now out!

What is Watermill

Watermill is an open-source library for building message-driven or event-driven applications the easy way in Go. Our definition of “easy” is as easy as building an HTTP server in Go. With all that, it’s a library, not a framework. So your application is not tied to Watermill forever.

Currently, Watermill has over 6k stars on GitHub, has over 50 contributors, and has been used by numerous projects in the last 4 years. It makes Watermill the most popular open-source event-driven library in Go.

Watermill provides support for 12 different Pub/Sub implementations, including Kafka, Redis Streams, NATS, Google Cloud Pub/Sub, Amazon SQS/SNS (alpha), SQL, RabbitMQ and more. It’s also possible to provide a custom Pub/Sub implementation if needed.

We believe that the foundation of every good open-source is excellent documentation together with real-life examples showing all features.

Yesterday, we just wrapped up the v1.3 release. We included multiple features that make Watermill even more powerful and easy to use.

What we’ve been up to over the last year

In March, we kicked off the work on our second training, Go Event-Driven. To check the interest, we organized a pre-sale with the goal of selling at least 50 copies. To our amazement, 323 copies sold within a week and generated over $54k of income. Even though we didn’t have any content ready yet.

Working on the training limited our time for Watermill, but on the other hand, it created an excellent synergy. While coming up with examples for the exercises, we also found a few improvements to Watermill.

Finally, once we finished the training, we could wrap up the work on Watermill v1.3.

Celebrating the Launch

You can watch the release party recording on YouTube.

Let’s dive deeper into the changes!

What’s new?

For the complete list of changes, see the GitHub releases page.

Big thanks go to everybody who contributed to this release! 💪

New CQRS API

There are three APIs for handling messages in Watermill:

The raw Subscribe() calls on a Subscriber.
The Router and message handlers.
The cqrs component and event/command handlers.

Each is a more high-level abstraction than the previous one. There’s no best API — depending on your use case, you may want to use one or the other.

During the last release, we mentioned we wanted to introduce a generic API to eliminate conversions from the interface{} (now: any).

At the same time, we noticed some design issues with the CQRS component. The way to use it was via the Facade, which wired everything up using a big config. If you wanted to create a simple setup, it felt like too much effort compared to the Router.

Here’s what changed in v1.3.

The `cqrs.Facade` is now deprecated

We won’t remove it, so your current code will keep working as before. We recommend using the new API instead of the Facade for new use cases.

For consuming messages, you want to use the CommandProcessor and EventProcessor:

For publishing messages, you want to use the CommandBus and EventBus:

The bus and processor concepts are not new. The previous API also had them, so this change is backward-compatible. Their API is now more cohesive and easier to use.

Generic handlers

With the new API, you can use the same event or command handlers as before. There’s also the new generic API that eliminates the type assertion and is easier to write.

This is the original API (still supported):

type OrderPlacedHandler struct {}

func (h OrderPlacedHandler) HandlerName() string {
 return "OrderPlacedHandler"
}

func (h OrderPlacedHandler) NewCommand() interface{} {
 return &OrderPlaced{}
}

func (h OrderPlacedHandler) Handle(ctx context.Context, e interface{}) error {
 event := e.(*OrderPlaced)

 return nil
}

And this is the new generic API:

cqrs.NewEventHandler(
 "OrderPlacedHandler",
 func(ctx context.Context, event *OrderPlaced) error {
 // No more type assertions!
 return nil
 },
)

Of course, you can still use structs for dependency injection:

type OrderPlacedHandler struct {
 repository OrderRepository
}

func (h OrderPlacedHandler) Handle(ctx context.Context, event *OrderPlaced) error {
 return h.repository.Save(event)
}

h := OrderPlacedHandler{
 repository: NewRepository(),
}

eventHandler := cqrs.NewEventHandler("OrderPlacedHandler", h.Handle)

Event handler groups

The EventProcessor creates one subscriber for each event handler. This works great out of the box for the common use case of one event type per topic.

If you keep all your events on one topic, you can use the new AddHandlersGroup() method to create a single subscriber for all specified handlers. This is especially important if you care about the message ordering. Otherwise, your events may be processed in parallel by different handlers.

err := eventProcessor.AddHandlersGroup(
 "events",
 UserSignedUpHandler{},
 OrderPlacedHandler{},
 OrderCancelledHandler{},
)

In this setup, set the AckOnUnknownEvent config option to true so the processor skips events that don’t have a handler assigned.

Circuit Breaker Middleware

A new CircuitBreaker middleware implements the circuit breaker pattern. It uses the gobreaker library.

Like the Throttle middleware, it can help to avoid overloading your downstream services in case of a failure.

If processing the message fails several times, the middleware immediately returns an error without calling the handler for the following messages. After some time, it tries to rerun the handler. If it succeeds, it returns to normal. Otherwise, it keeps failing.

The number of fails, the time it needs to fail within, and the time to wait before retrying are configurable, among other things.

Request-Reply support for commands

While communication over messages is asynchronous most of the time, it sometimes makes sense to receive a confirmation or result from a command. Many Pub/Subs support this use case with the request-reply pattern. We introduced a similar feature that works on top of any Pub/Sub.

The new requestreply package works on top of the cqrs.CommandBus. It allows you to send a command and wait for a reply. The Reply can contain an error or, optionally, a (generic) result. If you are returning just an error from the handler, the function signature is compatible with the generic handler signature.

err := commandProcessor.AddHandlers(
 requestreply.NewCommandHandler(
 "hotel_room_booking",
 ts.RequestReplyBackend,
 func(ctx context.Context, cmd *BookHotelRoom) error {
 return fmt.Errorf("some error")
 },
 ),
)
// ...

After running the command processor with the handler, you can send a command and wait for a reply:

reply, err := requestreply.SendWithReply[requestreply.NoResult](
 context.Background(),
 ts.CommandBus,
 ts.RequestReplyBackend,
 &BookHotelRoom{ID: "1"},
)

// ...

fmt.Println(reply.Error) // it's equal to `fmt.Errorf("some error")`

You can also send a result from the handler:

err := commandProcessor.AddHandlers(
 requestreply.NewCommandHandlerWithResult[PayForRoom, PayForRoomResult](
 "pay_for_room",
 ts.RequestReplyBackend,
 func(ctx context.Context, cmd *PayForRoom) (PayForRoomResult, error) {
 return PayForRoomResult{PaymentReference: "1234"}, nil
 },
 ),
)
// ...

After running the command processor with the handler, you can send a command and wait for a reply:

reply, err := requestreply.SendWithReply[requestreply.NoResult](
 context.Background(),
 ts.CommandBus,
 ts.RequestReplyBackend,
 &TestCommand{ID: "1"},
)

// ...

fmt.Println(reply.Result.PaymentReference) // it's equal to "1234"
fmt.Println(reply.Error) // it's nil

We have also used generic types here, so your code is type-safe.

By default, we don’t recommend using the request-reply pattern because it introduces coupling between services. Usually, using RPC is a better choice. But if you have a good argument for using request-reply (for example, you want to ensure that the request is processed, even if there was a transient error, and you wish to return a response to the customer), we’ve got you covered.

Slog adapter

Watermill now supports a logger backed by the new slog API.

watermill-sql: v2 released

We discovered a low chance of losing messages in watermill-sql in some specific cases. This is due to how transactions work — if you’re interested in the details, see this blog post.

We had to change the default schema and the public API to fix this, so this is a major version bump. We recommend using watermill-sql v2 from now on.

Note the minimal PostgreSQL version to use it is now 13.

Migrating the existing database schema would be pretty complicated. Instead, we recommend using new tables.

If you use watermill-sql just for the outbox pattern and there is one topic you care about, you can rename it. For example, if you’ve used the topic events-to-forward, you can rename it to events-to-forward-v2 while bumping watermill-sql to v2. This will create new tables with the new schema.

If you use more topics, you can use the GenerateMessagesTableName option to customize the table name. The default prefix is watermill_. You can change it to something else, like watermill_v2_, while bumping watermill-sql to v2.

GenerateMessagesTableName: func(topic string) string{
 return fmt.Sprintf("watermill_v2_%s", topic)
}

Remember to pass this option to both the schema and the offsets schema!

Usually, these tables are used just for delivering the messages and are not needed after it’s done. But you need to ensure all messages are processed before switching to new tables. In high-traffic environments, you may need to run both versions of watermill-sql for the migration period.

Amazon SQS/SNS — looking for testers!

There’s some work going on within PRs of watermill-amazonsqs (thank you, @Mariscal6!). If you’re familiar with this Pub/Sub or are running a project using it, we’d love to hear your feedback!

Live Event Recording

You can watch the release party recording on YouTube.

Watermill v1.2 released

Thu, 16 Feb 2023 00:00:00 +0100

After almost three years since the last stable release, Watermill v1.2 is finally out!

If you’re new here, Watermill is our open-source library for building event-driven applications in Go, the easy way.

A lot has happened since v1.1. We’ve been running Watermill on production in many projects, using it in new ways and adding features to support new use cases. We also received a lot of feedback and contributions from the community.

Watermill grew to over 5.2k stars on GitHub and 39 contributors, counting just the main repository. We’ve added 4 new official Pub/Subs, and there are many unofficial integrations.

Celebrating the Launch

Meet the Gopher at our live event!

To celebrate the release, we plan to host a live event on March 1st at 5pm UTC. Join us on YouTube.

The planned agenda is as follows:

Watermill 1.2 Showcase.
Using Consumer Groups in Watermill - a live demo with examples.
Announcing our Event-Driven Training.
Coming next to Watermill (generics!).
A chance to win a unique crotchet Gopher.

What’s new?

You can find the full list of changes on the releases page. It’s long and includes many bug fixes and minor improvements, often from outside contributors (thank you, everyone!).

Below are highlights of the bigger changes.

The Forwarder Component

One of the best use cases for Watermill is publishing messages in transactions (AKA the outbox pattern). I wrote about it before in When an SQL database makes a great Pub/Sub featuring a Watermill example.

The Forwarder component provides a universal mechanism for this behavior: forwarding messages from one Pub/Sub to another (e.g., from SQL to Kafka).

It’s a high-level API on top of the Router, so it’s even easier to forward messages.

Forwarder has its own documentation entry and a complete example. It was contributed by Grzegorz Burzyński.

The FanIn Component

We already had the FanOut component before. It takes a single stream of messages and re-publishes them on a few topics.

FanIn does the opposite: it takes a collection of topics and merges the messages from them into a single stream.

FanIn has its own documentation entry. It was contributed by Michał Sokołowski.

I’m thrilled to see it’s possible to write such components on top of existing structures. It shows that Watermill’s abstractions follow the UNIX philosophy of modular components that work well together.

New Supported Pub/Subs

Firestore - a NoSQL document database from Google. Contributed by Grzegorz Burzyński and Filip Borkiewicz.
Bolt - an embedded key/value database for Go. Contributed by Filip Borkiewicz.
NATS Jetstream - NATS’s built-in distributed persistence system. We released v2.0.0 in the existing NATS repository. Contributed by Alex Ullrich.
Redis Stream - Redis-based Pub/Sub. Contributed by Hao-Ming Hsu.

New Examples

Server-Sent Events example

SSE is our preferred way of pushing updates from the server. It’s simpler than WebSockets and plays nicely with Watermill’s abstractions.

See the example on the Watermill repository.

Exactly-once delivery example

Exactly-once delivery is not possible in most setups, but there’s one specific scenario where you can achieve it.

See the example on the Watermill repository.

Adding handlers while the Router is running

It’s now possible to add handlers to a running Router. Previously, all handlers had to be set up before starting it.

New CI

We moved the CI from CircleCI to GitHub Actions. It’s better integrated with GitHub’s PRs, and the configuration is simpler. All Pub/Subs now use the workflows defined in the main repository.

All repositories now use golangci-lint for static analysis.

Awesome Watermill

We’ve added the Awesome Watermill page with a list of unofficial libraries. Please add a PR if you know a repository that should appear there!

Watermill Contributors

Without contributors, Watermill wouldn’t be where it is today.

Writing a Pub/Sub integration is an especially high-impact project. To support the authors, we now keep a dedicated private channel on our discord server where they can reach us directly. The authors of Pub/Sub integrations keep write access to the repository after it’s transferred to the ThreeDotsLabs organization.

Let us know if you want to join this community and have your Pub/Sub integration officially supported.

Support

As always, if something’s not clear, check out watermill.io. If you can’t find the answer there, join our discord server and ask your questions on the #watermill channel.

If you’d like to get some open-source experience and contribute to Watermill, we’ve prepared a list of good first issues.

The Go libraries that never failed us: 22 libraries you need to know

Tue, 13 Dec 2022 00:00:00 +0100

Did you have a situation when you lost a ton of time finding a Go library for your need? In theory, you can check lists like Awesome Go or make a choice based on GitHub stars. But Awesome Go contains over 2600 libraries, and popularity is not always the best indicator of library quality. I often thought that it would be great to have a place where I could find just the best and battle-tested libraries I could use in my project. Because we didn’t find such a place with Miłosz, we decided to create it.

Based on our experience leading multiple Go teams and working on various projects, including complex financial, health, and security, we will recommend tools that could work well for different projects.

In addition to providing a list of libraries, we also want to show you some non-obvious uses for those tools and libraries. However, it’s important to note that most of these tools can be misused. We’ve included some common anti-patterns to help you avoid making those mistakes.

Note

This list is intended to be opinionated. We only wanted to include libraries we used on real production systems. Thanks to that, we recommend just libraries that we are 100% sure about. Unfortunately, our day is limited to 24 hours, so checking all available libraries is impossible.

If you know of any libraries we should include on this list, please let us know in the comments! We will continue to update the list with new findings over time.

Table of Contents

HTTP

Routers

As I mentioned in my previous article, it’s generally better to use libraries instead of frameworks for long-term projects. One of the most fundamental components of any service is an HTTP router. While it’s technically possible to build an application without one by using the standard library’s http package, its routing capabilities are limited. Using a dedicated router will make your life much easier.

❌ Anti-pattern: Frameworks in Go

If the library you consider using impacts how you write your domain models, it’s probably a framework, not a router.

We recommend using lightweight routers instead. Learn more about the risks of using the framework in my previous article.

By design, router functionality is limited to routing requests to a proper handler. All non-standard functionalities like CORS, CSRF, error handling, HTTP logging, and authorization (that frameworks usually provide) are provided by reusable middlewares. I recommend some in the section on middlewares.

I use one of two router libraries in most projects: Echo or chi. Both of them are great routers, with different characteristics. They work perfectly with OpenAPI code generation.

✅ Echo [GitHub] [Docs] [Examples]

Compared to chi, Echo does offer a custom *http.Request handler signature. Some people may find it a downside, but I think it helps to write less error-prone HTTP handlers.

If you have been writing Go for a while, you probably made this mistake at least once:

func someHandler(w http.ResponseWriter, r *http.Request) {
 err := foo()
 if err != nil {
 w.WriteHeader(http.StatusBadRequest)
 // you forgot the return here, bar() will be executed
 }

 bar()
}

Echo makes you return an error:

func someHandler(c echo.Context) error {
 err := foo()
 if err != nil {
 return err
 }

 bar()

 return c.NoContent(http.StatusNoContent)
}

The advantage of Echo is the ability to define a custom error handler. It’s not possible to do it in the same way with chi.

For detailed usage and examples, please check Echo docs.

✅ chi [GitHub] [Docs] [Examples]

Compared to Echo, chi’s handler functions are compatible with the standard library. For some people, it may be an upside; for some, it may be a downside – you should make your own judgment.

What chi does better than Echo is the format of defining routes and grouping. It gives you better control over middleware per path or sub-path.

r.Route("/articles", func(r chi.Router) {
 r.With(paginate).Get("/", ListArticles)
 r.Post("/", CreateArticle) // POST /articles
 r.Get("/search", SearchArticles) // GET /articles/search

 r.Route("/{articleID}", func(r chi.Router) {
 r.Use(ArticleCtx) // Load the *Article on the request context
 r.Get("/", GetArticle) // GET /articles/123
 r.Put("/", UpdateArticle) // PUT /articles/123
 r.Delete("/", DeleteArticle) // DELETE /articles/123
 })

 // GET /articles/whats-up
 r.With(ArticleCtx).Get("/{articleSlug:[a-z-]+}", GetArticle)
})

Full source

❌ Anti-pattern: You should not choose tools based just on benchmarks

Some developers choose libraries based on the benchmark results. It’s a risky approach because extreme performance optimizations lead to worse API and limited functionalities set. In most cases, performance differences are negligible in real-life use cases.

Even if, for some applications, it may make a difference, for most applications, it doesn’t matter that much. Making just one extra database query or up-scaling a service can make a much more significant difference in performance.

If performance is not absolutely critical for you, you should prefer other characteristics, like the ease of use and number of features.

Middlewares

HTTP middlewares can give you functionalities like CORS, CSRF, error handling, HTTP logging, authorization, etc.

Echo and chi provide their set of middlewares:

Echo middlewares have a different interface, so they can’t be used in chi. Generally speaking, all standard-library compatible middlewares are compatible with chi and echo.

To use standard library-compatible middleware with echo, you need to call echo.WrapMiddleware:

package main

import (
 "github.com/go-chi/chi/v5/middleware"
 "github.com/labstack/echo/v4"
)

// echo version
func main() {
 e := echo.New()

 // You can use a middleware from chi with echo.
 e.Use(
 echo.WrapMiddleware(middleware.BasicAuth("realm", map[string]string{
 "admin": "password",
 })),
 )

 e.Logger.Fatal(e.Start(":8080"))
}

If none of them provides the middleware you are looking for, you can check the Awesome Go list. All of them will be compatible with chi, Echo, and servers built just with the standard library. You can also write your own middleware. Check example middlewares for inspiration!

Serving static content

You don’t need any library to serve static content in Go. Since Go 1.16, you can easily embed static files into your Go binary.

Here’s how to do it for Echo and chi:

package main

import (
 "embed"
 "net/http"

 "github.com/go-chi/chi"
 "github.com/go-chi/chi/v5"
 "github.com/labstack/echo/v4"
)

// your static files should be in the static/ directory, for example static/index.html, static/main.js etc.
//
//go:embed static
var staticFs embed.FS

// chi version
func main() {
 r := chi.NewRouter()

 r.Handle("/static/*", http.StripPrefix("/", http.FileServer(http.FS(staticFs))))
 log.Fatal(http.ListenAndServe(":8080", r))
}

// echo version
func main() {
 e := echo.New()
 e.GET("/static/*", echo.WrapHandler(http.StripPrefix("/", http.FileServer(http.FS(staticFs)))))
 e.Logger.Fatal(e.Start(":8080"))
}

After running the server, assets will be available under http://localhost:8080/static/index.html, http://localhost:8080/static/main.js etc.

❌ Anti-pattern: Do not use no-name libraries for trivial functionalities

Do you remember the leftpad JavaScript library? It was 11 lines of code adding padding on the left side of a string.

Some day, the author decided to remove that library. It wouldn’t be a big problem if it wasn’t a dependency of thousands of projects, including Node and Babel.

Serving static content from your web server is one of such trivial functionalities.

Are you experienced engineer who wants to learn Go basics?
You don't become an engineer by watching videos.
Learn Go hands-on by building real projects.

OpenAPI

Nobody likes to maintain API contracts manually. It’s annoying and counterproductive to keep multiple boring JSON’s up-to-date. OpenAPI solves this problem with a JavaScript HTTP client and Go HTTP server generated from the provided specification.

This is how an example specification looks like. If you didn’t work with OpenAPI before, you can read more details in my previous article. Here, I focus on tools that we recommend for OpenAPI spec-generated code.

Generating Go server and clients

We do not recommend using the official OpenAPI generator for the Go code. We recommend the oapi-codegen tool instead because of the higher quality of the generated code. It also has more functionalities.

❌ Anti-pattern: Don't try to generate OpenAPI spec from Go code

There are tools that can generate an OpenAPI spec from Go code. We don’t recommend using them.

The entire OpenAPI specification is very rich, and it will be hard to generate everything from Go code. It’s likely that you will need to add something to the OpenAPI spec at some point, and it may be impossible to do it from the Go code.

It’s much easier to generate it the other way around: Go code from OpenAPI spec.

✅ deepmap/oapi-codegen [GitHub] [Docs] [Example]

oapi-codegen is a great tool that doesn’t just generate models but also the entire router definition, header’s validation, and proper parameters parsing. It works with chi and Echo.

To generate a server, run the following:

oapi-codegen -generate types -o "<OUTPUT DIR>/openapi_types.gen.go" -package "<GO PACKAGE>" "api/openapi/service.yml"
oapi-codegen -generate <TYPE> -o "<OUTPUT DIR>/openapi_api.gen.go" -package "<GO PACKAGE>" "api/openapi/service.yml"

Where <TYPE> for chi should be chi-server, and for Echo just server.

To generate clients:

oapi-codegen -generate types -o "<OUTPUT DIR>/$service/openapi_types.gen.go" -package "<GO PACKAGE>" "api/openapi/service.yml"
oapi-codegen -generate client -o "<OUTPUT DIR>/$service/openapi_client_gen.go" -package "<GO PACKAGE>" "api/openapi/service.yml"

Don’t forget to change <GO PACKAGE> to the desired Go package name and <OUTPUT DIR> to the desired output dir. 😉

Your job on the server side is just to implement the ServerInterface interface, like:

// ServerInterface represents all server handlers.
type ServerInterface interface {

 // (GET /trainer/calendar)
 GetTrainerAvailableHours(w http.ResponseWriter, r *http.Request, params GetTrainerAvailableHoursParams)

 // (PUT /trainer/calendar/make-hour-available)
 MakeHourAvailable(w http.ResponseWriter, r *http.Request)

 // (PUT /trainer/calendar/make-hour-unavailable)
 MakeHourUnavailable(w http.ResponseWriter, r *http.Request)
}

Full source

You can see it in action in the Wild Workouts project.

Bonus: Client for JavaScript/TypeScript

Even if it’s the list of recommended Go libraries, you may need to generate code for the browser.

✅ openapi-generator-cli [GitHub] [Docs]

In that case, we also recommend a non-official library instead of the official one.

In contrast to oapi-codegen, openapi-generator-cli is a Java tool. To avoid any JVM-related issues, we recommend generating clients using Docker:

docker run --rm --env "JAVA_OPTS=-Dlog.level=error" -v "${PWD}:/local" \
 "openapitools/openapi-generator-cli:v6.2.1" generate \
 -i "/local/api/openapi/service.yml" \
 -g javascript \
 -o "/local/web/src/clients/service"

It assumes that the spec is available locally in api/openapi/service.yml.

You can use openapi-generator-cli for TypeScript and other languages as well.

Alternative types of communication

gRPC

gRPC is a technology that can help you with building robust, internal communication between your services (but not only!).

I already described in detail why it’s worth using gRPC for internal communication and how to do it.

I’ll not repeat it here and will focus on the tooling you need.

With gRPC, you have little choice for generating server and client: you should use official tooling. The good news is that you don’t need anything more because it does its job!

✅ protoc [Docs]

To generate Go code from .proto files, you need to install protoc and protoc Go Plugin.

A list of supported types can be found in Protocol Buffers Version 3 Language Specification. More complex built-in types like Timestamp can be found in Well-Known Types list.

Messaging

✅ Watermill [GitHub] [Docs] [Examples]

About four years ago, when working on one of our projects, we found that there is no library that can simplify building message-driven or event-driven applications easily. To make our lives easier, we decided to write a library that will allow us to write event-driven code as easily as writing HTTP services. This is how Watermill was born.

Today, Watermill is one of the most popular Go libraries with almost 5k GitHub stars, +35 contributors, and 10 officially supported Pub/Subs.

Usually, message-broker libraries are very low-level. With Watermill, publishing messages may be as simple as:

publisher.Publish("example.topic", msg)

And subscribing like:

messages, err := subscriber.Subscribe(ctx, "example.topic")
if err != nil {
 panic(err)
}

for msg := range messages {
 fmt.Printf("received message: %s, payload: %s\n", msg.UUID, string(msg.Payload))
 msg.Ack()
}

Compared to using just the message broker’s library, Watermill provides support for some higher level functionalities like middlewares, CQRS support, or event-forwarder component (that can be used to stream your messages from an SQL database to the message broker).

Today, Watermill officially supports Kafka, GCP Pub/Sub, NATS, RabbitMQ message brokers (Pub/Subs). It can also listen to and emit events as HTTP hooks, from databases like MySQL/Postgres, BoltDB or Firestore. It can also work with in-memory Go-channel based Pub/Sub.

Don't miss new posts.
Join over 18k subscribers of our newsletter and get a free e-book!

Go With The Domain Three Dots Labs

🔒 We do not send spam. You can unsubscribe at any time!

Database

SQL

There is no golden hammer solution for interacting with SQL databases. The reason is simple: it depends greatly on what kind of data you store.

In some projects, data models are relatively simple. In some, they are very complex. Because of that, I have two libraries to recommend. You should choose one of them based on the requirements of your project.

For projects with straightforward data models, you should check sqlx. For a bit more complex, you should look at SQLBoiler.

✅ Tactic: Using ORM

I hear more and more that using ORM is not a good idea. I understand a reason for such thinking: many people are hurt by the improper use of ORMs.

It’s like with a knife: I have an extremally sharp Japanese knife without which I can’t imagine cooking. On another side, I need to be very careful with using it. But that fact doesn’t make this knife a bad tool! If you are using it properly, it’s making your life much easier. It’s the same situation with ORMs. Writing queries by hand may be time-consuming and error-prone when your models are complex. ORMs were invented to solve that problem.

If you have a bad experience using ORMs, you should check Things to know about DRY article. The tactics presented in that article will help you to avoid all common problems with ORMs.

❌ Anti-pattern: Avoid weakly typed ORMs

Most ORMs depend heavily on reflection and interface{}/any. The type system is one of the biggest strengths of Go. It helps you build applications efficiently. Resigning from strict typing makes your application more error-prone.

✅ sqlx [GitHub] [Docs]

The standard library’s database/sql package is rather a low-level one. sqlx provides a more convenient and powerful API to work with databases. It includes helper functions for common tasks like inserting and querying data and support for more advanced features like prepared statements and transactions. sqlx also has more advanced support for data unmarshaling (for example to structs, lists of structs, json data, etc.). As a nice bonus, sqlx’s interface is compatible with interfaces from database/sql.

But even if sqlx is a great library, it works well for relatively simple database models. At some level of complexity, you should consider migration to an ORM.

✅ sqlc [GitHub] [Docs] [Examples]

sqlc is now our default choice for new projects. Like SQLBoiler, it generates type-safe Go code, but from your SQL queries rather than just the schema. You write standard SQL, and sqlc generates fully type-safe Go code for your queries.

The key advantages of sqlc:

Query-first approach: You write SQL queries directly, and sqlc generates Go functions for them. No new query DSL to learn.
Compile-time safety: Catches SQL errors before runtime, including column mismatches and type errors.
Excellent PostgreSQL support: First-class support for PostgreSQL features like CTEs, window functions, and arrays.
Active development: The project receives regular updates and has a growing community.

Here’s an example workflow.

-- queries/users.sql
-- name: GetUser :one
SELECT id, name, email FROM users WHERE id = $1;

-- name: ListUsers :many
SELECT id, name, email FROM users ORDER BY name;

sqlc generates type-safe Go functions you can call directly:

user, err := queries.GetUser(ctx, userID)
users, err := queries.ListUsers(ctx)

sqlc supports PostgreSQL, MySQL, and SQLite.

✅ SQLBoiler [GitHub] [Docs] [Examples]

Note

Update (2026): SQLBoiler is now in maintenance mode. It still receives updates and works well in existing projects, but we recommend sqlc for new projects.

SQLBoiler is an ORM that generates Go models from your database schema. At first, how you define SQLBoiler models may surprise you. Most ORMs generate the database schema out of your Go models. SQLBoiler does the opposite: it generates Go models from your database schema.

This approach has multiple advantages. One of the most important features is stricter typing than other libraries. Thanks to that, many checks are done during compilation. You don’t need to depend on a ton of reflection and magic struct tags. In most cases, as long as the code compiles, it will work correctly.

Generating code from the database schema helps with migration from an existing database because you don’t need to re-write DB models: SQLBoiler generates them for you. So if you start with sqlx and move to SQLBoiler later, the migration should be pretty easy.

SQLBoiler supports PostgreSQL, MySQL, MSSQLServer 2012+, SQLite3, and CockroachDB.

❌ Anti-pattern: Using database models in the API responses

As long as you’re not writing a stupid simple CRUD application (and the chances are you’re not), you should not couple your database models with the API responses.

At some point, requirements will force you to return data in a format different from the format you have in the database. Instead of trying to follow DRY at all costs, it’s time to split your models.

Migrations

SQLBoiler and sqlx don’t provide out-of-the-box support for migrations. It’s okay because you are not forced to use any particular solution.

In my recent projects, I used both sql-migrate and goose, and I was happy about them.

✅ sql-migrate [GitHub] [Docs]

✅ goose [GitHub] [Docs]

We like sql-migrate and goose because of their simplicity and flexibility. sql-migrate and goose can be executed as CLI tools and as part of your service.

I like to embed it into the binary of the service. Thanks to that, the migration is executed when the service starts, and it keeps the setup simple. It’s also much less complex to run. For example, sql-migrate with go:embed:

// migrations/run.go
package migrations

import (
 "database/sql"
 "embed"

 migrate "github.com/rubenv/sql-migrate"
)

//go:embed *
var migrationsFiles embed.FS

func Run(postgresConn string) error {
 db, err := sql.Open("postgres", postgresConn)
 if err != nil {
 return err
 }

 migrations := &migrate.EmbedFileSystemMigrationSource{
 FileSystem: migrationsFiles,
 Root: ".",
 }

 if _, err := migrate.Exec(db, "postgres", migrations, migrate.Up); err != nil {
 return err
 }

 return nil
}

Put your migrations in migrations/, for example: migrations/1_init.sql, migrations/2_alter_some_table.sql, etc. Then run Run in your main.

Observability

Logging

The standard library’s logger doesn’t provide essential features like log levels and output formatting.

For logging, we can recommend two libraries: Logrus and zap. In contrast to zap, Logrus provides a bit nicer user API, but zap is faster.

You can check detailed benchmarks in zap’s readme.

❌ Anti-pattern: You should not choose tools based just on benchmarks

Some developers tend to choose libraries based on the benchmark results. It’s a risky approach because extreme performance optimizations lead to worse API and limited functionalities set. In most cases, performance differences are negligible in real-life use cases.

If performance is not absolutely critical for you, you should prefer other characteristics, like the ease of use and number of features.

✅ Logrus [GitHub] [Docs]

✅ zap [GitHub] [Docs]

Metrics and tracing

✅ opencensus-go [GitHub] [Docs]

OpenCensus Go is a library that helps you add metrics and tracing to your endpoints or database queries. The integration uses middleware/decorator patterns, and it doesn’t require a lot of custom code. It supports HTTP endpoints, gRPC endpoints, SQL databases, MongoDB, etc.

You can export traces and metrics to Prometheus, OpenZipkin, GCP Stackdriver Monitoring, Jaeger, AWS X-Ray, Datadog, Graphite, Honeycomb, or New Relic.

Configuration

Go’s standard library doesn’t support much more configuration options than the flag package. Even if it’s enough for simple CLI tools, you may need a bit more for building services.

Env variables

✅ caarlos0/env [GitHub] [Docs]

This library should provide everything you need for configuration for most applications. Compared to the standard library, it does support loading envs to structs and setting env defaults. It helps to save a lot of boilerplate for bigger configurations. It also supports embedded structs, so you can compose bigger a configuration from independent components.

✅ Tactic: Use env variables for your services configuration

For most applications, environment variables should be good enough as configuration.

Configuration is where you should keep secrets and things that differ between environments. If your configuration is massive and does not change often, it may be worth hardcoding it instead. It’s much more pragmatic than having tens of never-changing configuration options.

Multi-format configuration

✅ koanf [GitHub] [Docs]

Koanf is an excellent tool if your project requires multiple configuration formats. It’s often the case when you write tools that are used externally (for example, CLI tools).

This is my most recent finding. Compared to other more popular libraries, koanf just does loading multi-format configuration right. Bonus points for a nice abstraction that allows extending parsing and loading.

Koanf does support the most important configuration formats, like json, yaml, dotenv, env vars, or hcl. They can be loaded from the filesystem, flags, and multiple external sources like s3, vault, etcd, or consul.

Building CLI

Building CLI libraries

✅ urfave/cli [GitHub] [Docs] [Examples]

We like urfave/cli because of its simple interface and extensibility. We used it in multiple projects without any issues.

Compared to other alternatives, it offers a big-enough feature set while keeping the library lightweight.

Testing

Assertions

✅ testify [GitHub] [Docs]

testify became the standard assertion library, and I’ve seen it in every project I worked on. It provides asserts for the most common cases and also some more complex. One of testify’s key features are friendly messages for all failed asserts. It makes writing and debugging tests much faster.

The library provides two ways of asserting:

assert from github.com/stretchr/testify/assert - the test continues after failure. You should use it when you want to see multiple errors (not just the first one). Works when called in a goroutine.
require from github.com/stretchr/testify/require - the test is interrupted after failure. You should use it when some critical condition was not met and continuing doesn’t make any sense (for example: storing to database failed). Doesn’t work when called in a goroutine.

Some example asserts:

Equal - good enough in most cases
Eventually - useful for asserting asynchronous conditions
ElementsMatch - useful for unsorted slices
WithinDuration - useful when comparing time that is not exactly equal
ErrorIs
JSONEq
Panics

✅ Tactic: Use assert messages just if it is really needed

I’ve seen people who obsessively write messages for all failed assert.

For example:

assert.Equal(t, 123, 321, "123 is not equal to 321")

will give output:

Error: Not equal:
 expected: 123
 actual : 321
Test: TestEqual
Messages: 123 is not equal to 321

As you can see, the message doesn’t add anything more than testify would figure out. It can even be harmful because with time, you will need to spend a lot of time to keep the message up to date.

In most cases, the message provided by testify will be good enough. If the test fails, the person who sees the failure will navigate to this test and will understand the reason from the surrounding code.

✅ Tactic: Do not write basic asserts by hand

Many people advocate for writing all asserts by hand. It won’t give you much advantage in the end.

testify is also very smart in showing the difference between the expected and actual value.

For example:

assert.Equal(t, []byte("foo bar baz"), []byte("foo bar 42"))

prints:

 Error: Not equal:
 expected: []byte{0x66, 0x6f, 0x6f, 0x20, 0x62, 0x61, 0x72, 0x20, 0x62, 0x61, 0x7a}
 actual : []byte{0x66, 0x6f, 0x6f, 0x20, 0x62, 0x61, 0x72, 0x20, 0x34, 0x32}

 Diff:
 --- Expected
 +++ Actual
 @@ -1,3 +1,3 @@
 -([]uint8) (len=11) {
 - 00000000 66 6f 6f 20 62 61 72 20 62 61 7a |foo bar baz|
 +([]uint8) (len=10) {
 + 00000000 66 6f 6f 20 62 61 72 20 34 32 |foo bar 42|
 }
 Test: TestEqual
--- FAIL: TestEqual (0.00s)


Expected :[]byte{0x66, 0x6f, 0x6f, 0x20, 0x62, 0x61, 0x72, 0x20, 0x62, 0x61, 0x7a}
Actual :[]byte{0x66, 0x6f, 0x6f, 0x20, 0x62, 0x61, 0x72, 0x20, 0x34, 0x32}

It makes no sense to reinvent the wheel and write it from scratch.

❌ Anti-pattern: Do not use test suites from testify

Testify is an excellent library for assertions, but we don’t recommend its test suites. They don’t support parallel sub-tests. They may be fine for unit tests, but for integration/API/E2E tests it’s a deal-breaker.

The standard library can achieve most of the functionalities provided by testify’s test suites. You can see specific examples in this article on testing microservices.

✅ go-cmp [GitHub] [Docs] [Examples 1] [Examples 2]

Sometimes, you must assert a complex struct in your tests skipping some fields. Or the struct contains fields that should be compared in a specific way. Or you need to ignore the slice order or time delta. It’s where go-cmp can help you!

import (
 "github.com/google/go-cmp/cmp"
 "github.com/google/go-cmp/cmp/cmpopts"
)

diff := cmp.Diff(
 want,
 got,
 // FieldToIgnore and AnotherFieldToIgnore will be ignored in SomeStruct
 cmpopts.IgnoreFields(SomeStruct{}, "FieldToIgnore", "AnotherFieldToIgnore"),

 // when comparing time, truncate it to one second
 // can be written for any type
 opt := cmp.Comparer(func(x, y time.Time) bool {
 return x.Truncate(time.Second).Equal(y.Truncate(time.Second))
 })

 // sort all []int
 cmpopts.SortSlices(func(x, y int) bool {
 return x < y
 }))
)

// cmp returns diff if two objects are different
// to check if objects are equal, you can assert if the diff is empty
assert.Empty(t, diff)

To see the list of all available options, I recommend checking the godoc of cmp and cmpopts package.

go-cmp can also be used outside of tests, but be careful – it’s another tool that, used irresponsibly, may hurt your project.

✅ gofakeit [GitHub] [Docs]

If you need more realistic data for your tests, gofakeit helps.

Mocking

Writing mocks by hand

Initially, I recommended one popular mocking tool here. But after some thinking, we decided that the tool is not good enough to recommend. Instead, consider an alternative mocking strategy 👇

✅ Tactic: Consider writing mocks by hand

Even if it sounds like a waste of time, writing mocks yourself may be good enough. Objectively speaking, writing them by hand doesn’t require much more code and time. As a bonus, it gives you much more flexibility.

This is how an example mock can look like:

type BalanceUpdate struct {
 UserID string
 AmountChange int
}

type UserServiceMock struct {
 BalanceUpdates []BalanceUpdate
 balanceUpdatesLock sync.Mutex
}

func (u *UserServiceMock) UpdateTrainingBalance(ctx context.Context, userID string, amountChange int) error {
 u.balanceUpdatesLock.Lock()
 defer u.balanceUpdatesLock.Unlock()

 u.BalanceUpdates = append(u.BalanceUpdates, BalanceUpdate{userID, amountChange})
 return nil
}

It took me literally 1 minute to write it.

✅ Tactic: Keep your interfaces small, so it's easier to mock them

It’s hard to mock complex types by hand. But if your interface is so complex you can’t write a mock for it, you should reconsider if it needs to be that big. Using mocking libraries obfuscates the real problem.

Try to simplify the type that you are mocking. Maybe the interface segregation principle will help? It could be possible to split this type into multiple smaller types.

It will not only simplify your mocks but will improve your codebase.

Misc

Extra types support

✅ google/uuid [GitHub] [Docs]

This library generates UUIDs.

✅ oklog/ulid [GitHub] [Docs]

UUIDs may be slow to store at a larger scale in relational databases. A solution may be using Universally Unique Lexicographically Sortable Identifier: ULIDs. ULIDs are compatible with UUIDs, are unique enough for large scale, and have shorter string representation (Crockford’s base32). ULIDs are lexicographically sortable, thanks to what building indexes should be much faster.

Note

It’s worth mentioning that UUID v6, v7, and v8 will also be lexicographically sortable. But its spec is still draft when during the release of the article. If you want to try UUID v6 or v7, you can check github.com/gofrs/uuid which does already implement them.

✅ shopspring/decimal [GitHub] [Docs]

Go doesn’t have built-in support for decimals. shopspring/decimal does the job. We have used this library for a couple of years to build a large financial system.

✅ Tactic: Use decimals for monetary values

Floats are not designed to accurately store decimal numbers.

For example:

fmt.Printf("%.16f", 12.1+0.03)

> Output: 12.1300000000000008

To make sure your money calculations are correct (and you are not losing or getting extra cents in calculations), we recommend using a decimal type.

It’s also a good idea to use the string representation of decimals instead of floats in transport (in events, API requests and responses, etc.).

Errors

✅ hashicorp/go-multierror [GitHub] [Docs]

Did you ever need to handle an error while you were handling another error? hashicorp/go-multierror is here to help you!

It’s also helpful if an operation can return multiple errors, and you don’t want to return just the first one (for example, validation).

Example use cases:

func validate() {
 var resultErr error

 if err := validateFoo(); err != nil {
 resultErr = multierror.Append(resultErr, err)
 }
 if err := validateBar(); err != nil {
 resultErr = multierror.Append(resultErr, err)
 }

 return resultErr
}

func ExecuteStuff() error {
 if err := makeStuff(); err != nil {
 if cleanupErr := cleanup(); cleanupErr != nil {
 err = multierror.Append(err, cleanupErr)
 }

 return err
 }

 return nil
}

Note: Go 1.20 will introduce errors.Join function. After release of Go 1.20 you should consider using it instead.

Useful tools

Misc

✅ samber/lo [GitHub] [Docs]

Lodash-style Go library based on Go 1.18+ Generics. It may be especially useful for you if you are coming to Go from Python and missing some basic slice/map functions.

Some functions that I’m using the most:

Filter
Map
Keys
Values
Find
Max
Must 😈 please use it just for tests or if you really have a good reason

Even if some may find it “non-idiomatic”, I find it useful in some cases. It’s similar to using an ORM – as long as such libraries are used responsibly and don’t obfuscate code, they are useful.

So if you find yourself writing code like:

lo.Map(
 lo.Filter(someSlice, func(v SomeType, _ int) bool {
 return v.IsSpecial
 }),
 func(t SomeType, _ int) string {
 return t.SpecialName()
 },
)

…it’s just better to convert it to a simple, more readable loop. 😉

✅ Task [GitHub] [Docs]

Task is not really a Go library, but it’s a tool written in Go that may be useful for your projects.

It’s an excellent alternative to Makefile. The most important features that it offers are:

Parallel tasks execution (supported by task dependencies)
Preventing unnecessary work
Loading .env
Dynamic variables
Forwarding CLI arguments
Templating

It’s a must-have for each of my new projects.

Live code reloading

✅ reflex [GitHub] [Docs] [Example]

Go doesn’t provide code live-reloading out of the box. But you can achieve it quickly with the reflex library.

Some time ago, Miłosz wrote an article that shows how to create a local environment with Docker and reflex.

Linter

✅ golangci-lint [GitHub] [Docs]

golangci-lint is a linter that aggregates multiple linters and runs them in parallel and does it very fast.

Here’s an example configuration that we use in our projects.

✅ go-cleanarch [GitHub] [Docs]

If you use Clean/Hexagonal Architecture in your project, you can use this linter to ensure that The Dependency Inversion Rule and interaction between packages are kept.

Formatters

✅ go fmt

The standard formatter provided by Go toolchain.

✅ goimports [Docs]

Goimports does all that go fmt does, but it also sorts imports of your Go files. It’s one of the tools that you will see widely adopted in most Go projects.

Not everybody knows, but you can also separately group your local imports with the -local flag.

goimports -local "github.com/ThreeDotsLabs/some-repository" -l -w .

✅ gofumpt [GitHub] [Docs]

Just for the biggest formatting freaks! Does all that go fmt and goimports do and more!

Personally, I like gofumpt’s formatting decisions.

Example projects

DDD & Clean Architecture

✅ Wild Workouts Go DDD Example application [GitHub]

Wild Workouts is an example Go DDD project that we created to show how to build Go applications that are easy to develop, maintain, and fun to work with, It shows a project developed over time and with complex problems to solve. In contrast to other example projects, it was not blindly copied from other languages.

This is the way how we build our services daily. Highly recommended if you are looking for patterns that will allow you to build more complex projects!

❌ Anti-pattern: Low-quality example repositories

Avoid projects that look like over-engineered copies from other programming languages.

People who write such “DDD” projects often just read a couple of articles about it without understanding it correctly and without using it in real-life projects. If you see DDD/Clean Architecture examples without encapsulated domain models (with public fields) and json tags: run! It’s definitely not DDD nor Clean Architecture.

General purpose

✅ Modern Go Application by Márk Sági-Kazár [GitHub]

Another example repository that we can recommend. It doesn’t cover patterns like DDD or Clean Architecture but emphasizes infrastructure beats like observability.

Summary

Should we check some library that is not listed here? Please let us know in the comments!