The Problem

We discovered that invoice numbers were being generated out of sequence for shipments containing multiple orders.

Instead of clean sequences like:

inv-1, inv-2, inv-3

we occasionally saw results such as:

inv-1, inv-2, inv-125574

This only happened for shipments with multiple orders, and the pattern was inconsistent across environments, sometimes all invoices were wrong, sometimes only one, and sometimes none at all.

How Invoice Numbers Were Generated

At the time, invoice numbers were generated using a custom sequence table and MySQL’s LAST_INSERT_ID() function.

The flow looked like this:

1. Increment the sequence:

   update invoice_sequence
   set next_value = LAST_INSERT_ID(next_value) + 1
   where id = :id;

2. Fetch the generated value:

   select LAST_INSERT_ID();

3. Insert the invoice number into InvoiceNumber

4. Insert an outbound message into another table (to notify other consumers of invoice event)

The Key Assumption

This logic only works if steps 1 and 2 execute atomically, i.e., with no inserts happening in between. Otherwise, LAST_INSERT_ID() can change.

What Went Wrong

Hibernate Flush Behavior

For shipments with multiple orders, this logic ran inside a loop.

Hibernate (JPA), for performance reasons, does not immediately execute insert statements. Instead, it queues them and flushes later. Either:

• at transaction commit, or

• automatically before executing certain queries.

This led to a subtle but fatal interaction.

Why LAST_INSERT_ID() Broke

In MySQL:

• If an insert happens on a table with an AUTO_INCREMENT column, it overwrites LAST_INSERT_ID().

• If the table has no auto-increment, LAST_INSERT_ID() becomes 0.

Hibernate sometimes flushed pending inserts between:

• the sequence update, and

• the select LAST_INSERT_ID() call.

When that happened, the invoice number suddenly came from an entirely different table.

Observed Behavior in Different Environments

Because Hibernate’s default flush mode is AUTO, the timing was unpredictable.

We observed several patterns:

a) Local Environment

For a shipment with 3 orders:

• The first invoice was correct

• The next two were wrong

Hibernate flushed inserts right before the next iteration’s select LAST_INSERT_ID(), corrupting the value.

b) Production – Flush Only at Commit

Some shipments flushed only when the transaction closed.

• No errors occurred

c) Flush at the Last Order

Hibernate flushed inserts only during the final iteration.

• Only the last invoice was corrupted: in 10 orders, only the last one was corrupt

d) Flush Mid-Loop

Hibernate flushed somewhere in the middle.

• One random order was corrupted

• Example: 4 orders, 2nd one corrupted

There was no fixed pattern.

Why This Happened

Hibernate always flushes before executing a native query.

Since select LAST_INSERT_ID() is a native query, Hibernate triggered a flush—but only if there were pending inserts.

This behavior is documented and consistent with Hibernate’s implementation:

• Default flush mode: AUTO

• Meaning: Hibernate may flush “sometimes”

Impact

• Shipments with multiple orders generated invalid invoice numbers

• Invoice sequences jumped unexpectedly

• Downstream systems were affected due to incorrect invoice references

The Fix

We made two key changes:

1. Removed LAST_INSERT_ID() Entirely

• Replaced the update/select logic with a SELECT … FOR UPDATE on the sequence row

• Ensured correctness without relying on session-level side effects

2. Generate Invoice Numbers in One Shot

• Instead of generating numbers inside a loop

• We queried and reserved all required invoice numbers once per shipment

This eliminated flush-related timing issues completely.

Why Tests Didn’t Catch It

• Unit tests used mocked databases

• Mock DBs don’t replicate:

  • LAST_INSERT_ID() semantics

  • Hibernate flush timing

• The issue was non-deterministic, making it nearly impossible to catch with small test runs

• Integ tests missing multiple orders in one shipment

Lessons Learned

1. Never rely on LAST_INSERT_ID() across multiple statements unless they are truly atomic

2. ORM flush behavior matters, especially with native queries

3. Mocked DB tests are insufficient for sequence and concurrency bugs

4. Prefer explicit locking (FOR UPDATE) or database-native sequences

5. Generate identifiers outside loops whenever possible

6. Run integration tests with all possible scenarios

Back in the days, say 1990s, internet and information was the turning point. Within a decade, every office, bureaucracy was digitized. Everywhere cyber-cafes popped up. Where they didnt, dial-up modems came in handy. Everyone got their first embarrassing emails. It fundamentally changed how people lived and did their work.

Exams were held on computers, results came in on internet. Banks were digitized. Govt. offices moved to Microsoft Office. Windows was on everyone’s vocabulary. Bill Gates became the richest person in the world.

That was indeed a turning point.

What happened after that, was no more a ‘turning point’ but a natural progression. Dial-up modems were discarded in favor of broadband connections. 40GB HDDs gone. Bulky desktops gone and everyone got down to a laptop, much more powerful than any bulky desktop. CD DVDs were not required anymore. Ahead Software, are you still there with your Nero?

None of it changed the way people lived their lives or worked. Sure, it modified and influenced their way of working in nooks and corners, but that wasn’t a modification to the fundamental approach of how life was being lived.

The next big thing that came up was probably cloud computing. But it didn’t effect the end consumers as much as the software companies. Probably the end users didnt even notice something changed in the backend of the products they were using. At least in the beginning. What end users would have certainly noticed is the mushrooming of delivery apps and eCommerce sites, which wouldn’t have been possible without the ease of access to quick infrastructure through cloud computing.

Fast forward to 2023, or maybe 2020+.

AI is suddenly the new thing that everyone is talking about. Phrases like ‘agentic, LLM, AI powered, AI integrated, vibe coded’ etc., are everywhere. LinkedIn is full of AI flex. And so is the ‘AI will take my job’ paranoia. Just like delivery apps, this time an avalanche of AI startups. AI powered customer support, AI powered IDEs, AI integrated document system, AI in emails etc etc.

Why is it a turning point ? Well, maybe it isn’t, maybe it is a huge bubble waiting to burst. But it doesn’t appear like that to me. To me it appears more like a turning point, with a lot of exaggeration on the sides, of course. This is a turning point because it changes how we live and work.

The state of AI is now at a place where everyone has an assistant by their side. Things have become ‘easy’, or probably too easy sometimes. Writing some boilerplate code, restructuring some code portions, or writing docs - they are all too easy now.

And talk about learning. Ask AI to generate a learning plan for some topic you’d want to learn, maybe economics during the Aztecs, and AI will generate a plan for you with references!

Every industry, and every age group is effected by this. Students don’t write thesis no more! Movies refine, or maybe just write their scripts through AI. News articles edited through AI, sometimes with footers from ChatGPT still in there! Doctors take notes through AI. Software engineers are now becoming prompt engineers; yes, one abstraction level up!

With all that said, it is clearly changing the way we think and work. I am not sure I believe it is a great change, though. Like anything else, as long as we keep it in balance and equilibrium, we should be fine. The more we rely on AI, and the more we offload our “power of thought” to it, the more it will create a state of disturbance.

A few things, out of many, that I find particularly concerning are the environmental impacts of using such widespread AI and the energy-guzzling, GPU-powered data centers. End users, while generating funny videos or memes or just having fun with a prompt—do they know how much energy is spent behind the scenes?

Second, the impact on creativity and overall brain health. Humans are what we are because of our cognitive skills. What happens when we offload every bit of that to AI? Take the example of students writing their thesis with AI. Getting help with refinements or corrections is one thing, but writing the whole thing through AI is something that scares me.

Third, the cookie-cutter style prose. I can clearly identify what is written through AI and what’s not (or what is cleverly hidden). The dashes —, bullet points, and the same repeated emojis give it away. The style of being super neutral, without any hint of bias, is another sign. But think of a world where everything you read sounds similar. If that’s not boring, what is?

Fourth, I see it as a threat to both the people whose careers depend on writing and to those who simply consume whatever the AI throws at them—because the AI said so.

In conclusion, I believe this is indeed a turning point because it is changing our behavior, lifestyle, and working style. Now it is on us to make judicious calls on when and how to use it, and how much we should offload onto it.

The Berlin Days

I had joined Amazon in Berlin. A super sleepy team with extremely boring and meaningless work. I didn’t even get to commit my first code until well into 3 or 4 months into the job.

I was in a team that was tasked with re-architecting and implementing a ‘review solicitation engine’. In simple terms, it was something that would find out who bought stuff from Amazon in the last X weeks, and then send them emails to write a review of the product on the Amazon site. The legacy workflow was implemented as a few python scripts that used to run on cron triggers. The new service was to be implemented in ‘modern’, ahem, ‘Java8‘ and as a workflow, instead of a bunch of scripts tied together.

It seemed like a normal software engineering project in the beginning. But turned sour quite soon. A toxic team was to blame. The team had two additional SDEs who would never stop arguing. It was painful to work with the team. Even the SDM was helpless. Every single pull request ended in heated discussions, even over simple stuff like, ‘why tag code comments as [nit]’ or ‘how much detail would you want to put in an acceptance criteria of a ticket’. Getting them to do something as a team, was next to impossible. Many times, PEs were pulled into the discussions to just tell them ‘shut the fuck up, and do this’.

And that turned the project from super frustrating to extremely boring. The PE came in, and made a unilateral decision to work with a workflow already developed by some other team! And we were just expected to write ‘yaml’ configurations to tell how the workflow should behave. So, essentially, months of ‘heated discussions‘ on the potential Java service, reduced to a maintaining a number of yaml configuration files.

We still figured some good engineering things to do, afterall we liked coding! So we came up with templatizing the yaml files and generating those. Basically, use config files to generate more config files.

Later on, we started with experimenting reducing the spams sent to Amazon’s customers by using ML to identify most likely customers to write some reviews. And we did some additional enhancements and expansions among other things.

But overall, it was pretty boring. Meaningless too, as I despised the whole ‘send spam notifications’ to customers

The SCOT Years

I moved to Seattle to join a team in the gigantic supply chain organization. It was a team that identified vendors or suppliers from whom Amazon should replenish the inventories.

The work was better. Regular java microservices. People weren’t fighting much, which was good. But boy, was I in for a surprise! The abysmal state of code or testing or general engineering practices, were embarrassing. Code that was written sometime in 2008 or so, and had no clear owners and kept on passing around to multiple teams. Folks just came along, dumped their shit, and moved on. Design patterns? What’s that?

I soon learnt that Seattle had a completely different work culture. SDEs were more focused on ‘doing the most visible thing’ as opposed to ‘doing engineering things‘. Code optimizations, design patterns, testing strategies etc., always took a backseat. Folks were doing things that solved business problems ASAP no matter how it was done. Just add some patch work to already existing spaghetti mess and call it a day.

No wonder things were super messy. A deprecation of a service went on and on for years with no result. Changing an API signature would take 10 months of work. Tickets will remain open for months and then some ‘senior engineer’ will close them with an embarrassing comment like ‘if the issue still remains, please create a new one’. Of course, the ticket requester would have forgotten what was the issue in the first place.

I wouldn’t say it was the fault of the SDEs. It was the ‘culture’. SDEs who worked on business heavy projects were getting more visibility. Better chances of promotion if you learnt supply chain instead of engineering skills. Praises were showered on folks who wore multiple hats, i.e., SDEs who also did the work of a supply chain manager! SDEs would get promoted for writing a tiny lambda that created some business tickets, rather than engineers who spent much effort in rearchitecting in optimized way over AWS.

I realized I had to leave the org if I wanted to stay as an ‘engineer’. If I have to roughly estimate, 70% of my time was going in learning and making decisions related to supply chain and probably 30% was going towards real engineering work. I hated working with the decade old spaghetti code that nobody wanted to cleanup. If I would take the initiative to clean up that mess, it would have been a thankless job.

I tried to make some initiatives. Like, introducing Kotlin to the stack, or developing a generic test framework. I was shut down. ‘We are not in the business of making platforms’ or ‘Kotlin will be harder to move from Java’ (seriously?) were some of the remarks.

On top of that, the constant ass licking of upper level was nauseating. A senior manager literally said in an org meeting that ‘PEs/Sr. SDEs are great gods’. PEs or Senior SDEs were more Supply chain engineers rather than ‘software engineers’. I was disgusted by the incompetence of those. They would ask questions like ‘whats the benefit of Java 17 over 8 apart from being cool’, or ‘why do we need CDK when we can just have the resources created through console’.

So I left. I wanted to do engineering work. And I realized, the hard way, that I should stay at a place where ‘technology’ is the business.

Experiments at AWS

I first joined Route53. My team worked on the ZonalShift product that customers use to move traffic away from an availability zone to some other zone. Usually they use it during AZ impairment events. Like, IAD1 is down, then get out of that zone and move traffic to IAD6 or something.

It was what I wanted. Pure technology business. No bullshit supply chain decisions or sending spam emails to folks. I loved it. The focus on technology was much better than any other team before. We talked tech, instead of random business shit. People knew more than me. So it was an opportunity to learn from them, and I did a good deal.

Also the team was working heavy on Kotlin. CDK was given. I was doing a lot of those. Custom CDK constructs in typescript, developing features in Kotlin, talking to IAM teams, ASG teams etc., to onboard additional AWS services to ZonalShift. It was good work, and I liked it.

But, I still wanted something better. My learning in that team soon plateued. There wasn’t more fun things to do. And I wanted to get into rust, databases, system programming. Everything seemed very ‘high level’ now. It was much better than SCOT, but it wasn’t that fulfilling from my learning point of view.

So I moved again. This time to AuroraDSQL. The fancy, brand new database that was under development at that point of time.

I wasn’t in the core DB internals team. I joined the peripheries. I was in a team that worked closely with the control plane of the database. We owned the billing and metering of the database. When customers create a cluster, write some data, our service made sure that the db metrics (e.g., storage size, number of DPUs etc.,) are visible in their cloudwatch.

Everything was in rust. It was great learning experience. I also took part in some work that involved in getting a bit inside the storage engine. The team had very experienced developers who knew a great deal of DB internals, query engines, rust programming and so on. The team worked closely with rust team that actually develops some well known open source libraries and frameworks like turmoil, duchess, tokio etc.

This was everything that I wanted: DB, rust, system programming, distributed systems. I was already shining in the first few months of my joining. I looked forward to so much more! Formal methods, query engine optimizations, simulations, what not!

Until I wasn’t anymore.

The 5 day RTO

January 2025. Amazon declared war on any employee who had a life. Remote or 3-day RTO worked great for me. But 5-day RTO, that came as a kick on the balls.

Everyday, I lost about 2 hours in just driving. Add the time to get dressed, prepare things, stretch and breath, it was easily about 2.5-3 hrs of meaningless wastage of time and energy.

And it wasn’t just that. I had to make sure to at least reach office by 9:30 or so. Which meant, leaving home by 8:15 or at least by 8:30. But kids school timings were 9:00-3:15, which meant I had to arrange extra-hours for my eldest. Similarly my youngest, whose daycare timings were 9:00-5:00, had to be extended to 8:00-6:00.

This came as a big lifestyle and financial change. My kids, who eat their food slower than a snail crossing a mile, almost stayed fasting the whole time. The eldest, who used to come at 3:30 and have a late lunch at home, stayed hungry till 5:30. And both the kids had to wake up early, which effected their sleep quality every single day. They are not early risers, and everyday, I had to see them waking up with eye bags. It was distressing. Why they had to suffer because of a stupid rule from my employer?

Financially, extra hours at day care and school meant shedding extra money. Every month, I paid around 700 extra for that. And on top of that, I had to pay for the fuel and parking. I was losing money on things that I never wanted in the first place.

And all of it started to reflect on my work front. I was tired and exhausted by the time I’d reach home. I hardly worked for 4-5 hours in office and then, even if I wanted to to, I just couldn’t do much productive work at home. Meanwhile, fresh grads, or below-30 folks, staying near the office, easily spent their whole day in the office churning code.

There was no match.

I also thought of moving to Seattle. Maybe that would work? I spent a few weekends hunting rentals in that region. And I was of course disappointed. The lack of good amenities, super high rents, not-so-good schools, drug addicts and homelessness, shooting incidents etc., just didn’t feel right. Kids were having a good time where they were staying, with nice school and friendly atmosphere. Going to a drug infested zone, clearly would have been a wrong move for them.

Also, there was no assurance that Amazon wouldn’t lay me off or put me in focus or something after I move there. Amazon wouldn’t change anyway.

The Grind

Meanwhile, the grind at AuroraDSQL wasn’t getting any better. Deadlines after deadlines. No additional resources. Many folks, including seniors, left the org. Some left Amazon, some left to other teams. No-lifers could still sustain, working 14-16 hours a day. People worked over weekends like it was nothing. Do more, and more, and more - for some artificial deadline. I never understood who was waiting for those features holding their breath? But it was pushed. Get it done somehow!

I checked randomly. There wasn’t a single hour when I didn’t get an email from the code review system. Which comes when someone publishes a code review, or someone comments in a code review. I checked for a longer time window, like 2-3 weeks, and every single hour had at least a few mails: XYZ added some comments, ABC published a CR. 2 AM, 3 AM, 4 AM - every single fucking hour.

I wondered, what the fuck are these folks doing in their lives?

I obviously couldn’t do it. I liked databases, rust, distributed systems. But I liked my life more. I would rather read something and sleep, or maybe work on my personal projects, than work for Amazon.

I still could have done much, much more with remote or 3 day RTO setting. But with 5-day RTO, there was no way in hell I could have kept up with the no-lifers.

So, there went my ambition of getting into storage engine or query planner down the Nitro North’s toilet. I stayed in the peripheries and didn’t even bother to try to get into the core. It was just not worth it.

I finally found something that I thought would work better for me. So I left. Time will tell if that was a good decision. But I had to do it anyway. I moved to Aerospike.

The Goods and Bads of Amazon

There is no doubt Amazon has some great minds working for them. It’s a great place to learn engineering skills, explore diverse areas and domains of tech. The attention to detail, particularly in AWS, is something to be proud of. The deep dives, COEs, the microsecond level optimizations - all are great examples of high quality engineering.

But the bads, are just too many! Its a deeply hierarchical company. And its very frustrating to see incompetent managers or PEs or Sr. SDEs doing random and stupid shit. More so in Amazon Retail, compared to AWS. I have seen Sr. SDE asking question like ‘what is an integ test‘!

And with hierarchies, comes ass-licking. Amazon doesn’t lack asslickers at any level. SDEs try to keep a ‘friendly’ PE or Sr. SDE on their side, for promotions. Which is not a bad thing in general, but becomes toxic when they just do whatever thy lord says without questioning.

Same thing with quality. PEs get away from writing stupid, unorganized docs, with the excuse of ‘I put together this quickly to give some pointers to the team’. The same doc, if written by an SDE would require 5 rounds of refinement. There was clear pattern of ‘rules apply differently to different levels’.

The operational issues, they speak volumes. Amazon, particularly AWS, takes very seriously on ops. SDEs, oncalls, work like hell when a sev-2 knocks. But then, sometime, comes the holier-than-thou comments from the upper echelons. A PE would ask, “Why it wasn’t deployed on so-and-so region ? Bake-time? Why do we need bake time?”. The same PEs would then ask “Why there was a deployment on so-and-so region without a bake time?”

There was no winning here. Hierarchy wins. I am not saying that PEs are dumbasses. But the lack of empathy, high arrogance, farther away from actual ground reality, was very de-motivating. Amazon PEs need to learn that there are great engineers outside of Amazon too, or that, they can lead from the front while being a team player and not necessarily being ‘unavailable’.

I am sure there’s a good number of PEs who just stay unapproachable, fake busy, just to show that they are important! Power play, anyone?

I wouldn’t get into the crappiness of SDMs. They are just too many in Amazon. Its enough to say that I would try to stay away from SDMs who ever worked at Amazon.

Anyway, all that said, I am done with that. I wish it had worked out better. But it didn’t.

If you are an amazonian, and disagree with whatever is written above, shove it up your arse, and don’t reach out.

A test case was measuring ‘timestamp proximity’.

Sample code under test:

long timestamp = System.currentTimeMillis();

return SomePojo.<O>customBuilder()
       .requestId(UUID.randomUUID() + "-" + timestamp)
       .timestamp(timestamp);

As seen, it generates a UUID concatenated with a timestamp. The unit test was validating that the timestamp was ‘freshly generated’ and is within an acceptable range (=20ms):

String[] tokens = requestId.split("-");
        assertThat(Double.valueOf(tokens[tokens.length -1]), IsCloseTo.closeTo(System.currentTimeMillis(), 20));

So far so good. It just worked. And 20ms is a wide enough duration.

But it started failing after migrating to JDK15.

Suddenly, the duration 20ms was not enough anymore.

Turns out, the order of execution of the tests was no longer the same as before.

To validate what’s the order of execution, by default it relies on the order of the methods that the JDK provides. This can be tested out with the below sample code:

import java.lang.reflect.Method;

public class ReflectDemo {
    public static void main(String[] args) {
        Class<?> clazz = SampleClass.class;

        Method[] methods = clazz.getDeclaredMethods();

        System.out.println("Methods of class: " + clazz.getName());
        for (Method method : methods) {
            System.out.println(method.getName() + " - Return type: " + method.getReturnType().getSimpleName());
        }
    }
}

// A sample class with some methods
class SampleClass {
    public void sayHello() {
        System.out.println("Hello");
    }

    private int add(int a, int b) {
        return a + b;
    }

    protected void doSomething() {
        // do something
    }
}

When run with JDK 17:

Methods of class: SampleClass
add - Return type: int
doSomething - Return type: void
sayHello - Return type: void

When run with JDK 8:

Methods of class: SampleClass
add - Return type: int
sayHello - Return type: void
doSomething - Return type: void

Plus, this doesn’t stay static and it may depend on JVM implementation.

But, we weren’t using any ‘state’ that carries over from one test to another, so how can it effect?

Well, maybe not so obvious, but we still do have a ‘state’ that carries over. And thats the classloader.

So, what was happening was, when this particular test was not the first test to be executed, the UUID class was well loaded. However, when this became the first test to be executed, it did take some extra time to load the class and hence, sometimes it failed that 20ms bound check.

The fix

I could fix it with enforcing the tests execution order. But I guess that wasn’t the cleaner way to do, as that can thrown subtle bugs like these under the carpet.

The other way to fix it was obviously letting the timestamp get recorded after the class is loaded.

So, the changed code looked like this:

UUID uuid = UUID.randomUUID();
long timestamp = System.currentTimeMillis();

return SomePojo.<O>customBuilder()
       .requestId(uuid + "-" + timestamp)
       .timestamp(timestamp);

There! Fixed it.

Here is a similar case which I had found and fixed recently.

The Symptoms

1. Unit test started to fail suddenly - no code changes done in recent times.

2. Tests didn’t fail on build servers - only failing in local runs.

3. Tests failed on time assertions. Meaning, tests were able to compile, initialize and run.

Diagnosis

At the first glance, the difference between the local runs and the build servers was the Operating System. Build servers were Linux based, while the developers used Macs.

A difference in the OS shouldn’t cause errors on time assertions.

On a detailed look, I realized the build servers were running in UTC, while local setups were running, with local timezones (PST).

To validate the theory, I changed the timezone of my mac to UTC, and re-run the tests. Voila. Went smooth.

So, we have a case where unit test is dependent on the system timezone.

What did the test do

1. Take currentTime

2. Randomly take a duration range : e.g., 170 days - mocked as external input

3. Add it to currentTime to get an endTime

4. Assert that the endTime matches with the duration range that we got from external source.

Pretty simple. So, why this ‘addition’ in step 3 resulted in a wrong value in PST?

I found that test cases used DateUtils from JakartaCommons package. The add method of DateUtils uses Calendar fields to add days, without taking timezone into consideration. Normally this works, but it fails when there is a Daylight Saving Time.

Which they also mention in the JavaDocs, which was missed whenever that code was added:

It is important to note these methods use a Calendar internally (with default time zone and locale) and may be affected by changes to daylight saving time (DST).

For example, assuming daylight saving time changes on 01/01/2023 (mm/dd//yyyy), advancing the clock by 1 hour

We would expect this behavior : 

startDate = 01/01/2023 00:00:00
endDate = startDate + 1 Day = 01/02/2023 01:00:00

Days added: 1 + delta (delta = 1 hour)
Actual hours added: 24

That's because, normally, the expectation is we add 86400 seconds to the starting timestamp. So, with Daylight saving time, even when we add 24 hours, the clock is advanced by 25 hours.

However, Jakarta commons library's behavior is different. Instead of adding timestamps, it adds 1 Day to Calendar field. That means, for the same example above, it does this:

startDate = 01/01/2023 00:00:00
endDate = startDate + 1 Day = 01/02/2023 00:00:00

Days added : 1
Actual hours added: 23

With that in the test case, it became timeZone dependent and it failed in PST for a certain date range. If no fix is being done, it will start working again on its own, whenever the date range moves past the daylight switching date.

The fix

The fix was simple. Just don’t depend on the DateUtils library for this trivial task!

The service under question reads data constantly from a source, applies certain transformations, and writes to a sink (S3). Quite simple actually.

Normally, the data read is under a manageable volume that kept in sync with the S3 write operations. At least till now.

And then, suddenly the service encountered a large chunk of data, and S3 write couldn’t keep up with the pace it was reading from the source. It showed signs of frequent crashes and high heap mem usage and GC activities.

After much digging around, we got some heap dumps. (Yeah, its not that easy to get the heap dumps from within the docker containers.)

Similar to a previous encounter, I got the heapdump to analyze with Eclipse MAT.

It looked something like this:

So, a single thread, a single kotlin channel, is consuming 95% of the heap? But why?

Turns out, the ‘buffer’ that was plugged between the producer (component that reads the data) and the consumer (that writes to S3) was not applying backpressure. It was thought that it did.

Here is a sample code:

fun <T> Flow<T>.buffer(
    maxCapacity: Int,
    sizeFn: (T) -> Int
): Flow<T> = flow {
    coroutineScope {
        var capacity = maxCapacity
        val consumeTrigger = Channel<Unit>(capacity = Channel.CONFLATED)
        val channel = produce(capacity = Channel.UNLIMITED) {
            collect {
                val size = sizeFn(it)
                while (size > capacity) { consumeTrigger.receive() }
                capacity -= size
                send(it)
            }
        }
        channel.consumeEach {
            val size = sizeFn(it)
            capacity += size
            consumeTrigger.send(Unit)
            emit(it)
        }
    }
}

fun <T> Flow<T>.backPressuredBuffer(
    scope: CoroutineScope,
    maxItemRunningSize: Int,
    sizeFn: (T) -> Int
): ReceiveChannel<T> =
    buffer(maxItemRunningSize, instrumentationFn, sizeFn)
        .buffer(capacity = Channel.UNLIMITED)
        .produceIn(scope)

So, this thing, while applied a steady flow from the first buffer, it kept on consuming even when the buffer size was limited to say 1.

As learnt the hard way, a Channel.UNLIMITED essentially creates an infinite linkedList that takes whatever you offer it. And of course, heap memory is the limit.

Fixing it was quite simple. Just dont make it unlimited, and have a reasonable size, based on service behavior.

Here is a small sample unit test that validates if it works, or not:

@Test
    fun `test buffer doesn't overflow the max size`() {
        val totalItems = 1000
        val maxCapacity = 1000 // bytes

        // inner buffer is limited at 1k bytes
        // outer buffer is limited at 10 entries = 1k/100

        val randomByteArrays = List(totalItems) {
            ByteArray(Random.nextInt(100, 201)).apply {
                Random.nextBytes(this)
            }
        }

        val cursor = AtomicInteger(0)
        val receiveChannel = randomByteArrays.asFlow()
            .produceBuffered(
                scope = this,
                maxItemRunningSize = maxCapacity,
                sizeFn = { it.size }
            )
        // cursor is advanced by 1 which is not yet consumed by the buffer because the buffer is full
        assertTrue(cursor.get() <= 22)
        assertTrue(cursor.get() >= 11)
        receiveChannel.cancel()
    }

One such case was discovered a couple of days ago at my workplace, and interestingly, the bug was there for almost 4 years, and nobody noticed it. Which also means that the bug was not impacting the business much, thankfully. Or else, it would have made a lot of noise right when the bug got introduced.

The Symptoms

• Bad data is returned randomly from service fleet

• Some hosts return correct data, some not

• Bad hosts when restarted, start behaving correctly

• Once a host returns bad data, it always returns bad data

Issue diagnosis

From the first 3 symptoms, we can think of issues due to multi-threading with mutable data, like a cache. But the 4th one suggests that it could be something related to static initialization, or a singleton with mutable data. Because, with multi-threading concurrency issues, the occurrence of the issue is still expected to be random. And with cache, generally when the TTL gets over, it should auto-correct itself.

Since a bad host stays permanently damaged, it indicates towards a singleton or static object which is supposed to stay untouched, but somehow got changed.

Consider this:

public enum DaysGroup {
  WeekDays(
           Stream.of("Mon", "Tue", "Wed", "Thu", "Fri")
             .collect(Collectors.toSet())
           ),
  Weekends(Stream.of("Sat", "Sun")
             .collect(Collectors.toSet())
           )

  private Set<String> days;

  DaysGroup(Set<String> days) {
        this.days = days;
  }

  public Set<String> getDays() {
       return this.days;
  }

}

The above defines an enum which has a set of strings that gives meaning to the enums. The set is stored as an enum instance variable.

It is worth noting that enums are singleton by design, and hence there will always be a single copy of the defined enums : WeekDays and Weekends

With this defined, consider a service code portion:

...
Set<String> days = WeekDays.getDays();
if (condition) {
   days.remove("Fri");
}
...

This in the first look, seems okay. But, if seen with a slight detail, we notice that if the condition is true, the actual set from the enum is altered.

So, whats the issue?

The problem is that the enum is changed permanently, and next time if the code is run, or any other thread accesses it in parallel, it will never find Fri in the set anymore.

The problem is obviously temporarily fixed if the service is restarted, and it will stay fixed till that condition is satisfied again.

The problem also doesn’t occur enough if that condition is infrequent. Like, condition == is this year a leap-year.

The problem will occur only in those machines where this condition was ever met. So, if you have a 100 machine fleet for your service, and only 2 of them ever got this code path executed, then those 2 are broken, not the rest. So, the impact of it stays pretty unpredictable.

The Fix(es)

Return a copy

One way to fix this is to return a copy of the set, instead of the set itself.

public Set<String> getDays() {
       return new HashSet<>(this.days);
}

This works. But if this method is called at many places, and mostly they just read the items and dont change anything, then we are unnecessarily creating quite a lot of memory usage.

Keep an immutable set and return that

The other way is not to keep a mutableSet, but rather keep an immutable set as part of the enum construction.

DaysGroup(Set<String> days) {
   this.days = ImmutableSet.copyOf(days);
}

This will ensure that the clients of this code will handle the returned set as they see fit.

The above uses Guava’s ImmutableSet. In modern JDKs, this can simply be replaced with Set.of(…).

Preventions

1. The above was preventable with proper static analysis in place. For example, SpotBugs provides out of the box support to check such cases.

2. Best practices in code reviews can avoid such issues leaking to the prod.

3. Some elaborate Junit tests can capture such errors.

4. Use modern JDK/Kotlin to use immutability as first class construct rather than having the need to use external libraries or doing something special to get the immutability niceties.

A simplistic view of service cache

Caching of frequently used data to increase application throughput and reduce latency is a very common pattern.

Consider a case of services talking to each other. Service A fetches certain data from service B. This obviously incurs network latency, serialization-deserialization latency and CPU work, cost due to payload transmitted over the network and so on. If we know that the data from service B is not super real-time and we can assume some level of stability or immutability of the data, say, 1 hour, then we can essentially cache service B’s response for an hour and reduce our cost, latency and improve throughput.

Of course, there are many things to consider in a cache: invalidation strategies, what to cache, how much to cache, how long to cache, architectural requirements and so on. Many of these parameters are dependent on the business logic (for example we can’t and shouldn’t cache account balance maybe?) and some are service infrastructure dependent (how much memory can we assign for the cache?). Then there is completely separate discussion of distributed cache or local cache (an architectural question).

For now, we are simply talking about in-memory cache here. A HashMap!

Implementation of an InMemory Cache

There are off-the-shelf libraries which provide ready to use in memory caches. I am talking about JVM caches; but its the same in any other language or runtime. Some of the widely used caches include Guava or Spring.

Then there is always the option of doing it in-house for better control and with the philosophy of implementing only the required features instead of bloating the service up with some external library like Guava.

A sample AOP styled cache looks like:

(Taken shamelessly from spring docs.)

@Cacheable("books")
public Book findBook(ISBN isbn) {...}

None of the default implementations will provide any de-duped strategies. And that makes sense. Deduplication comes with its own set of problems.

What is de-duped cache and why we might need it?

A cache is just a key-value store, and naturally a cache can have only unique keys. But consider a case where values are duplicated across many many keys.

A hypothetical example (probably not a great one): say we have a bunch of vendors, and the currencies they can transact in.

Now, of course, if the query pattern is something like, get all the vendors for a given currency, we would need the currency as the key.

But if we have a query pattern that says something like: get all the transact-able currencies for a given vendor, the vendorId becomes the key.

And imagine a globalized economy (where we already are, kind of), most of the vendors can transact in most currencies. So, the value part gets duplicated over here:

			KEY           |            VALUE
         	------------------------------------------------
			vendor1       |         CAD,USD,EUR,GBP....
			vendor2       |         CAD,USD,EUR,GBP....

The problem is quite apparent here: duplicate data being repeated over and over again. The situation becomes worse with large number of keys with the same duplicated data.

If you are aware of JVM string DeDuplication, this is exactly the same case but above the JVM layer and inside the application.

Implementation of a Value Deduped Cache

A value DeDuped LRU cache is essentially a Cache over another cache. Simply put, we make the below change:

BEFORE:

key -> value

AFTER:

key -> hash(value)
hash(value) -> value

This clearly reduces the number of bulky value entries to their hashes, which do not eat up as much space.

A pseudocode for such an implementation might look like:

(The below code is also available here at github gist.)

class LeastRecentlyUsedDeDupedExpirationCache<K, V> {
    // To keep null values and distinguish null against cache miss
    private static final Object NULL = new Object();
 
    /**
     * Ordered map so that we can implement LRU.
     * This is the original LRU built on top of something like a LinkedHashMap
     * This keeps the "key -> hash(value)" entries
     */
    private final LeastRecentlyUsedMap<K, ExpirationValue<Integer>> map;
 
    /**
     * Map to hold dedup values. This need not be ordered as this is a map to backup the ordered
     * keys stored above.
     */
    private final Map<Integer, V> valueMap;
    
    /**
    * Other attributes such as capacity, ttl etc., aren't shown for brevity.
    **/
		
	public Value<V> get(final K key) {
	ExpirationValue<Integer> hash;
        Value<V> res = null;
 
        synchronized (this.map) {
            hash = this.map.get(key);
        }
 
        if (hash != null) {
            V value = valueMap.get(hash.getValue());
            if (value == null) {
            /**
             * This can happen when we have references like:
             * key1 -> hash -> value
             * key2 -> hash -> value
             *
             * Now, if key1 is expired, it will remove the the entries from both the maps.
             * That results in having : key2 -> hash -> null
             *
             * To avoid this, we can do the following options
             *
             * 1. Keep a ref-count of the hash :  key1 -> hash -> value, ref-count
             * We would need to update the ref-count whenever the key is added/expired
             *
             * 2. Keep another map which is revert of 1st map: hash -> key
             * That doesn't look so efficient
             *
             * 3. Just return null and make it look like the value didn't exist, and so the MethodProxyHandler will
             * fill up the cache
             */
                return null;
            } else if (value.equals(NULL)) {
                res = new DeDupedValue<>(null);
            } else {
                res = new DeDupedValue<>(value);
            }
        }
 
        return res;
    }
    
    public void put(final K key, final V value) {
        synchronized (this.map) {
            // Add the value to the map.
            int hash = Objects.hashCode(value);
            this.map.put(key, new ExpirationValue(hash, currentTimeInMillis + this.timeToLiveInMillis));
 
            if (value == null) {
                this.valueMap.put(hash, (V) NULL);
            } else {
                this.valueMap.put(hash, value);
            }
        }
    }
}

Invalidation

Since the same values are pointed to by different keys it is possible to end up in a situation where a certain key is expired, and hence the value is removed, and we end up in cache miss of other keys. For example:

k1 (expires at 10:00) -> hash(v1) -> V1

k2 (expires at 10:05) -> hash(v1) -> V1

If the expiration of k1 removes v1 from the value map, fetching k2 (or any other key with the same value) will result in a cache miss.

This can be avoided if we keep a reference count for each of the values. That makes the implementation slightly complicated. In the above code, we simply do a cache miss and rely on the fact that it will get pre-filled.

Immutability Caution

While it all sounds great, one thing to consider here is immutability. If the values cached are being mutated outside of the cache, it will effect all the keys that are holding references to the value. And that will create hard to debug issues and possibly corrupt data on production.

To avoid that, we can do one or more of the following:

1. Ensure data immutability - do not modify the cached items

2. Clone the object and then return - will consume more space - also, why de-dup?

3. Ensure that the hash function computes a different hash for any modification to the object, and compare the hash to stored hash value before returning.

If the requirements are something of the sort of ‘every key can alter the value as it requires’, we can go ahead with the clone approach, although, it probably doesn’t make much sense on wht to do the deduplication in the first place?

To ensure the immutability of the cached values, with the 3rd approach, the get method would look like:

public Value<V> get(final K key) {
	ExpirationValue<Integer> hash;
        Value<V> res = null;
 
        synchronized (this.map) {
            hash = this.map.get(key);
        }
 
        if (hash != null) {
               V value = valueMap.get(hash.getValue());
               ...
	       if (hash != hashFunction(value)) {
		      // IMMUTABILITY VIOLATED
		      // either force a cache-miss or throw exception
                     throw new ConcurrentModificationException();
		}
               res = new DeDupedValue<>(value);
               ...
        }
        return res;
    }

Here, even if we find the value in the cache, we double check the authenticity of the value by recomputing the cache and comparing against what’s there in the keys. If there’s a mismatch, it means it got modified through some other key.

So, either we can force a cache-miss here, or we can throw an exception and let the user know what they are doing.

Improvements after deduped cache

Post dedup cache, we see significant improvements in the heap space usage. Also as a side effect, no more OOM errors, and we could get the instance types down to one level cheaper, saving quite a lot of $$s.

Conclusion

JVM eco-system has some wonderful tools to help with deep dives into the internals of runtime behavior. Eclipse MAT is one such tool. Before jumping into ‘increase the heap space’ or ‘get a larger machine’, it is worthwhile to investigate which components are actually causing the issue.

As seen, some clever data structure optimizations can actually save lots in terms of performance as well as money spent on the infrastructure.

OOM errors and Heap dumps

The situation here is with a standard Java TomCat service processing sync and async API requests and doing certain processing and responding back with the response. With occasional heap-dumps due to OOM errors. (You can enable heap dumps on out of memory with -XX:+HeapDumpOnOutOfMemoryError as JVM arg.)

OutOfMemory errors come when the JVM is not able to allocate required space, as there is none left in the heap space.

The obvious way to fix it, is to assign larger heaps. But the problem with that is it might cost a lot depending on how many servers you have, and what if the heap is still not enough?

The other way to fix it, is by ignoring it. Please don’t do it. And if you do, make sure to remove the JVM arg XX:+HeapDumpOnOutOfMemoryError as that will clutter the server host with so many .hprof files and eventually the server will run out of Disk space. (This reminds us to have alarms on disk usage.)

So, if we aren’t doing any of the above : using larger heap or ignoring it, then we are left with ‘fixing it’ as real engineers. And in order to do that, we need to analyze the heap dump files.

Analyzing heap dump files

We can use the wonderful Eclipse MAT to analyze our .hprof files. Open them up, and lets see what things are eating up the most memory.

Interesting, about 3 GB of the memory is taken up by a single ProxyClass instance ! And, what exactly is that proxy class?

(If you don’t know what is a proxy class, read it here, and here.)

So, essentially the proxy class is generated through AOP which implements a Cache over a method. And the cache is a LinkedHashMap used as an LRU.

Looking at the contents of the cache:

This is even more interesting. Most of the values in the Map are showing up as the same size. This maybe a normal case if data are kind of well-defined with fixed size packets. But the service doesn’t work with any such thing and its purely business data which are almost never standardized as fixed size.

Looking deeper at a few values (not shown here), we can conclude that they all have the same data. Of course we can manually check all of those, but a good sample size test would also work here to get some confidence.

Essentially, we are looking at a HashMap with many different keys, but having the same value, which consumes 55% of memory and 3 GBs and results in frequent OOM errors.

In the next article, we will see where the duplicate data are coming from and what can we do about it.

I am noting down those items I have found very useful to organize myself (and my work), plan better and kill entropy.

Emails

I am one of those who created a gmail account when it was freshly brewed. I still use it everyday. But the Spams, promotions etc., are well, nothing less than crazy. And it almost runs at 99% capacity.

Also, every person that shares my name, probably assumes they share my email too. This has led me to go for a paid Google Workspace, with my own domain.

Yeah it costs. But much better! And I love gmail.

Other emails that I use: protonmail, tutamail among others. Absolutely love both of them for their privacy features. Also the email alias feature from Proton!

The key is to split up the external comms across various emails at random. And never share the critical ones for random subscription or RSS feeds.

Notes

I write a lot of notes. And I like notes with markdown features. I started with EverNote which I loved, but they got too greedy and restricted the users to just 50 notes. That too without notice.

I found Notion quite nice. Although there are limitations on how much free content it can have, but I guess when I hit that limit, will switch to something else like Coda. None of the apps have unlimited notes as far as I can tell.

Planning

While Notion is pretty good with planning, calendar and such, sometimes I require to share it with others. And for that, I found Microsoft To Do, and Google Keep pretty handy.

Other items that I use time to time: YouTrack and Trello.

Password Managers

With hundreds of sites and apps to login, its risky to use the same password at many places. I used LastPass for a while, till they suffered a security breach. These days I am using better privacy and encryption providers: ProtonPass and BitWarden. Super useful!

Storage

I need more data to store my random useless pictures and PDF files. I am paying a minimal for a 200GB plan from Apple iCloud. I use it only because it is well integrated with my phone.

I also use Google Drive extensively and the paid plan (Google Workspaces) come with 30G per license.

Of late, I have started using some private encrypted storage, and I use mega, koofr, Filen and NordLocker.

VPN

Not a regular user of VPN. But if I do, I use ProtonVPN. There are some good European providers such as NordVPN. But haven’t found the need of it yet.

File Shares / Pastes

There are some very nice tools around these. I found wormhole and paste.ec particularly interesting.

Messaging

I don’t like WhatsApp much. But unfortunately thats what most people use. So, I am kind of stuck with it.

Given a choice, I prefer Signal over WhatsApp, as it doesn’t require you to add contacts by phone numbers.

I recently discovered Simplex. That’s another level of anonymity, but I doubt I will use it anytime soon.

This doesn’t mean much. This is one of those 32-bit numbers, written in hex, frequently used in dev or tests (similar to foo-bar or Alice-Bob), particularly in low level programming.

This is a nice memorable word. Other such words include DeadBeef, CafeBabe etc.

Trivia: CafeBabe is used in official Java class file format.