Event-Driven by Oskar Dudycz

Interactive Rubber Ducking with GenAI

You may already know that I’m a GenAI sceptic. And a general sceptic.

Do you know that scepticism comes from the Greek σκέπτομαι (skeptomai), meaning ‘to search, to think about, or look for’? So my intention is not to say no to everything new, but more to think about it first, and understand before I say yes.

There’s a lot of stuff about GenAI that makes me smile, but I still understand that my way is my way, and I won’t stop the world. I won’t even try. Thus, I want to research and consider how those tools can help me. I already wrote that I don’t feel like 10x Dev, but I’m finding more ways to get help from it.

One of the ways that helped me is something I call “Interactive Rubber-Ducking”.

Initially, I called it just brainstorming, but that wouldn’t be precise, as I’m not using it to brainstorm ideas, more to challenge and clarify them.

Most of the code I write nowadays is done in my OSS projects. I’m grateful to have a great community with people actively contributing in different ways; still, the canonical design and code work is on my side. As I work in an event-driven niche, I’m often alone with my own thoughts. I try to use the RFC process and discuss it with other fellow humans, but they are not always available. Even if they do, to avoid wasting their time, I need to know what to tell or ask them. I need to give some proposals (with alternatives) to have an effective discussion. I may seem organised, but that’s not always the thing. Sitting in your own head is not a great place to be in general. If you’re a technical leader or an architect, I’m sure that you know that solitude too well.

GenAI tools are not great sparing partners. They’re Yes men. If they read this article, they’d for sure confirm it. They’d probably do it even without reading it. Of course, you can ask them not to be sycophant. You can ask numerous MUSTS with capital letters and bolded NEVER here and there, and it can help, but it won’t fully beat the way they were trained.

And talking to yourself is a similar experience: you’ll find numerous ways to justify your own decisions, and looking at the same place for too long will make you miss obvious blind spots.

Ok, so why would we take those two blind “people” and try to make them help each other?

That’s kinda what “Interactive Rubber-Ducking” is. It takes a blind human with an idea, and another blind not-so-human asking questions. It starts with such a prompt:

Ask me one question at a time so we can develop a thorough, step-by-step spec for this idea. Each question should build on my previous answers, and our end goal is to have a detailed specification I can hand off to a developer. Let’s do this iteratively and dig into every relevant detail. Remember, only one question at a time.

Once we are done, save the spec as spec.md

Before asking another question, store the previous one with the answer in qa.md. Write literally the question and answer, not just a summary.

Here’s the idea:

Don’t treat it as “one magical prompt that will change your life”. Most important is why we’re doing it, what happens next, and who’s actually doing the work. Spoiler alert: it’s not LLM.

I’m using it as a command in Claude Code, and, most importantly, with beefier models like Opus, which can better reason and ask better questions. Doing it with lower-level models always gave me much worse results.

I’m using it with Claude Code, not Claude Chat, because I want the model to scan my codebase. I can ask it to look in certain areas or to reference my answers. I can even ask to search the web or MCPs like Context 7 to check documentation and APIs for popular libraries. Then it’s getting more into brainstorming sometimes, than rubber-ducking, but that’s fine.

As a result, we’ll get two artefacts:

qa.md - with the log of the back-and-forth discussion,
spec.md - in theory spec built by LLM, but imho it’s more of a concise summary.

It may look like Specification-Driven Design, but it’s not.

My goal for this exercise is not to get an actionable specification.

The goal is to get our LLM-based Rubber Duck to ask us hard questions and make us think, not to make the LLM think for us. Find blind spots, and challenge our thinking.

But we’re drivers, we need to know what we want to do, we need to know all the WHYs, and we also need to know HOW. LLM is here to help, but not to do creative work for us. It just pulls it out from our heads.

It also helps to see how our design may be seen by others, especially such mediocre thinkers as LLMs.

I don’t expect the Agent to be able to start implementing the spec. I expect it to reflect all considerations and summarise findings. I’m always double-checking to make sure it includes all the important points. If not, I’ll keep doing Q&A until I’m satisfied.

Having both of those files will allow us to keep a full discussion without losing important details, and a shorter version. We can feed that to another model for review, or try to work on tasks and develop a more detailed plan. Sometimes I plan on my own; for simpler tasks, I may ask the LLM to do it fully. Usually, I’m driving the LLM step by step, passing just specific asks.

The example? Why not.

I recently did such an exercise, trying to narrow down how to introduce the Second-level cache to Pongo and Emmett. What’s Second-level cache? A Second-level cache is a local store of data managed by the persistence provider to improve application performance.

Why do I want to introduce it? Because I got an issue from the user that rebuilding projections with a lot events can take too long. One of the reasons is that applying an event on a projection takes:

loading the current state,
updating it,
storing the result.

Those are two operations per event. If we have a batch of 100 events, that’d mean 200 operations; for 1000 events, this would be 2000, so the classical N+1 problem. We could do it differently, and within a batch:

group events by the target documents,
load all of them in one operation by finding all documents within an array of ids (taken from events),
caching them,
applying events in memory,
storing updated documents in one operation.

Then we’ll get at worst 102 operations for 100 events and 1002 for 1000 events, so a linear increase.

Still, I decided to add batching and introduce 2nd-level cache as a solution. I and my experience were a driving force behind that. I could, of course, rubber duck it, but I had that already planned, not just detailed. I also did some initial research before doing the session.

Let me show you the result below. Not too bad if you ask me. Of course, the task wasn’t very innovative; it’s a standard way for mature solutions. Still questions were surprisingly good and helped me.

I hope this log will show you (besides how bad I’m at typos) some insight into how I think, talk, and consider the design.

Tell me your thoughts, if that matches your way, if you haven’t done it yet, try it and tell me how it went!

Interactive Rubber Ducking example

My idea

I’d like to provide 2nd-level cache to Pongo. I want to either use keyv (https://www.npmjs.com/package/keyv) or lru-cache (https://www.npmjs.com/package/lru-cache) packages.

For now, I’d like to do it by id.

I probably need to either extend pongoCollection src/packages/pongo/src/core/collection/pongoCollection.ts with findeOneById and findManyByIds methods, or check if _id is passed in and then try to access the cache.

Cache should be set up either on the pongo collection, pongo db or pongoClient.

Optimistic Concurrency errors should invalidate cache (as that means someone else updated the record in db, e.g. through a different node).

InsertMany and insertOne should put it into the cache.

I’d also like to expand the handle method to allow passing either an id or an array of IDs to support batch loading of entities and improve performance, and to support batch update/insert/delete.

Thoughts?

Which defaults should I set? (Of course, allowing users to override it).

Check the related code I pasted. I want to keep it aligned and as non-invasive as possible.

Q1: Cache library choice — lru-cache, keyv, or custom interface?

Question: You mentioned both keyv and lru-cache. They pull in different directions. Which trade-off do you prefer?

A) Start with lru-cache, optionally let users swap adapters later.
B) Start with keyv so cross-node sharing works out of the box.
C) Define a thin cache interface (get/set/delete/clear) and ship lru-cache as the default adapter.

Answer: Custom cache interface (option C). No dependency on external APIs — keep it pluggable. Start with an in-memory provider only. Considered writing a custom LRU with me, but wants to ship fast. Key point: don’t stick to any third-party API surface, allow pluggability.

Q2: Where should the cache live in Pongo’s hierarchy?

Question: Three attachment points: A) PongoClient level (shared across all), B) PongoDb level (per database), C) PongoCollection level (per collection). Or configurable at multiple levels?

Answer: Allow defining it at different levels — cascading configuration. Set a default higher up, override per collection if needed.

Q3: Inheritance/override behavior for cascading cache config?

Question: For the cascading config — what should the inheritance/override behavior be?

Two sub-questions:

Should a lower level be able to disable caching even if a parent enables it?
Should cache settings (like max size, TTL) also cascade, or only the cache provider instance itself?

Answer:

Yes. Cache should be an object with type, general params and specific for this type, or 'disabled'. That’s a proper devex.
Yes, as above. But if one selects a different cache type then general params should be inherited but type-specific should not be inherited and should fallback to defaults.

Claude’s thoughts: The CacheConfig | 'disabled' pattern is clean — discriminated unions play nicely with TypeScript. The inheritance rule makes sense: general params (TTL, max size) cascade, but type-specific params reset to defaults when you switch providers, avoiding leaking nonsensical config (e.g., a Redis connection string bleeding into an in-memory cache). Also worth distinguishing between “I didn’t specify anything” (inherit from parent via undefined) and “I explicitly want defaults” (reset) — undefined vs 'disabled' vs CacheConfig gives a clean three-state: inherit / off / explicit.

Q4: Cache interface shape — async, batch ops, scoping?

Question: Settling the cache interface details:

Async or sync? Making the interface Promise-based from the start keeps it pluggable. Sound right?
Batch methods (getMany, setMany, deleteMany) — should the cache interface support batch ops natively, or loop internally?
Should clear() be scoped per collection, or always clear the entire cache?

Answer:

Maybe we could return either promise or no promise, then if we await it, it would support both? Also ensure that settings are close to what lru-cache and keyv are having.
Having them (even as syntactic sugar) would be a good devex.
We could pass down not only settings, but also the cache provider instance, then if settings are the same we can reuse cache without recreating it. If set had ttl provided, then we could do it by overriding those TTLs on item level. ClearAll probably would be something we should expose only for advanced, typically user should not be aware of cache.

Claude’s thoughts: MaybePromise<T> (i.e. T | PromiseLike<T>) works — await on a non-Promise just resolves immediately. No overhead for sync providers, no breaking change for async ones. Passing a cache instance is smart: config becomes either settings (Pongo creates the cache) or a pre-built instance (user brings their own). Multiple collections can explicitly share one cache instance. clear() should be internal/advanced — scope by collection prefix internally so a collection-level clear doesn’t nuke another collection’s entries when sharing an instance. For aligning with lru-cache/keyv settings: both support max (max entries) and ttl (TTL in ms). Start with max and ttl as the general params.

Q5: Cache key strategy

Question: Since cache instances might be shared across collections, how to avoid key collisions?

A) Pongo manages key prefixing internally ({dbName}:{collectionName}:{id}). The cache interface works with plain string keys, namespacing is Pongo’s concern.
B) Key is just the document _id. Simpler but only safe with per-collection instances.

Answer: A, definitely.

Q6: Cache invalidation and consistency with optimistic concurrency

Question: How should cache interact with different operations?

Successful writes → update/evict cache. Clear.
Concurrency errors → evict stale entry. Clear.
Reads with _id filter → cache hit or miss-then-populate. Clear.
handle method — should the read inside handle use the cache (risking more concurrency errors on stale data) or always go to DB?

Answer: Definitely use cache. We could consider disabling cache through handle options.

Claude’s thoughts: Use cache by default — optimistic concurrency is the safety net. If cached version is stale, write fails, we evict, caller can retry. Adding skipCache?: boolean to HandleOptions is a clean escape hatch for high-contention cases.

Q7: Should `skipCache` be available on read methods too?

Question: Should skipCache also be available on findOne and other read methods for symmetry, or keep it handle-only?

Answer: Good point, having skipCache on find methods (also findMany etc.) would be good.

Claude’s thoughts: Consistent API surface. Add skipCache?: boolean on the shared options type (e.g. CollectionOperationOptions) so it’s available everywhere without repeating yourself.

Q8: Batch operations on `handle` — signature and handler shape

Question: You want handle to accept an id or array of ids. Two design choices:

Overloaded signature (handle(id: string | string[], ...)) vs separate method (handleMany)?
Handler shape for batch: A) still receives/returns one document at a time (Pongo batches DB ops internally) or B) receives/returns full array?

Also noted: since PongoDocuments always have _version, if someone provides expectedVersion we could compare it against the cached version and decide if it’s stale without hitting the DB.

Answer: Option A for handler shape. handleMany would be consistent with other APIs, but single id or array (string | string[]) is clean enough.

Claude’s thoughts: Version comparison in cache is a cheap short-circuit — if cached _version doesn’t match expectedVersion, we know it’s stale without a round trip. On signature: TypeScript overloads can narrow the return type (string → single result, string[] → array). handle is a higher-level orchestration method, not direct CRUD, so a union input feels natural. Go with handle(id: string | string[], handler, options?).

Q9: Defaults — max entries, TTL, enabled by default?

Question: What defaults to set? Proposed: max: 1000, ttl: 60_000 (1 min), and should cache be enabled by default or opt-in?

Answer: Stick to recommended lru-cache defaults for now. Enable cache by default.

Claude’s thoughts: Enabling by default works because: (1) optimistic concurrency is always the safety net, (2) it’s document-by-id only so invalidation is straightforward, (3) users can set 'disabled'. lru-cache requires max to be set (no default) and has no default TTL. Suggested: max: 1000, no TTL (rely on LRU eviction + invalidation on writes/concurrency errors) — most lru-cache-aligned.

Q10: TTL or no TTL as staleness backstop?

Question: No TTL means cached entries live until LRU-evicted or invalidated by a Pongo write. If something updates the DB outside this Pongo instance (another service, direct SQL), the cache won’t know. Options:

A) No TTL, accept the trade-off — OC catches it on next write.
B) Set a conservative default TTL (e.g., 5 minutes) as a staleness bound.

Answer: Fine with using TTL and sometimes hitting OC issue. OC assumes rare conflicts.

Q11: Where does cache integration hook into existing code?

Question: Two approaches for minimal invasiveness:

Wrap at the pongoCollection level — intercept operations inside the factory function, driver/SQL layers untouched.
Decorator/wrapper pattern — cachedPongoCollection wraps PongoCollection, adds cache logic around delegation.

Answer: Leaning more to option 1. Thinks it’ll be easier to make observability precise eventually.

Claude’s assessment: Agreed with option 1. Cache decisions depend on internal state (whether filter targets _id, document _version, write success vs concurrency error). A wrapper would need to re-inspect all of that from the outside. For observability, cache hit/miss logic next to the actual DB call enables precise metrics in context. Pongo’s factory functions already compose behavior internally (transactions, schema migration) — cache is another concern at the same level. A decorator would duplicate filter-inspection logic and add indirection that doesn’t carry its weight.

Q12: Cache interaction with transactions

Question: How should cache interact with transactions?

Should writes within a transaction update the cache immediately, or only after commit?
Should reads within a transaction check the cache?

Answer:

After commit.
We should use cache, but allow skipping it on transaction or session options. We can also allow setting cache settings per session.

Claude’s thoughts: Reads within a transaction benefiting from cache is more aggressive but logical — most transactions are short-lived. The narrow case: insertOne then findOne same _id within a transaction — cache won’t have the uncommitted insert, falls through to DB, gets the right answer anyway. No correctness issue, just no perf benefit for that case. Per-session cache settings fit the cascading pattern (e.g., startSession({ cache: 'disabled' }) for batch migrations). Proposed cascade priority: client → db → collection → session → per-operation skipCache.

Q13: Cascade priority — does session override collection?

Question: Does the cascade priority client → db → collection → session → per-operation skipCache feel right? Session overriding collection is the non-obvious part — the argument being that session is the most immediate runtime context, while collection is structural config. Or would you prefer collection always wins over session?

Answer: That’s fine. Session should precede collection, as normally you create session to e.g. share connection, and join the number of operations logically, so it’d make sense actually to also do it to e.g. override settings like caching.

Claude’s thoughts: Session as logical grouping of operations with shared context (connection, transaction) — overriding cache settings at that level is extending the same concept. Bulk import in a session? Set cache disabled once on the session instead of skipCache on every call. Final cascade: client → db → collection → session → per-operation.

Q14: Cache provider error handling

Question: If the cache provider throws (e.g., Redis connection drops), should Pongo:

A) Treat it as a cache miss — fall through to DB silently, maybe log a warning. Cache failures never break the application.
B) Propagate the error — let the caller deal with it.

Answer: A — yes, when we add observability to Pongo, we need to make it also a metric and trace event/attribute/log entry.

Claude’s thoughts: Cache is an optimization, not a correctness requirement. Silent degradation now, observable degradation when the observability layer lands.

Q15: Cache warming, events/hooks, and delete operations

Question: Three remaining areas:

Cache warming — should Pongo support pre-populating the cache on startup?
Cache events/hooks — should users be able to listen to cache hits/misses/evictions?
Delete operations — deleteOne/deleteMany should evict from cache. Anything non-obvious there?

Answer:

For now keep it as a future concept, or define pluggable strategy for users.
You can add the basic hook to the Pongo Cache wrapper. Shouldn’t be that hard to just call some function.
Yes, they should evict if they can, we can delete (for now) when deleteMany deletes by ids.

Claude’s thoughts: Cache warming deferred — can add a strategy hook later. Basic event callbacks (onHit, onMiss, onEvict, onError) on the cache wrapper are low-cost and useful for debugging even before full observability. Delete eviction scoped to id-based deletes for now — filter-based deleteMany doesn’t evict (no way to know which ids were affected without querying first).

Interactive Rubber Ducking result

Pongo 2nd Level Cache — Specification

Overview

Add a document-level (by _id) 2nd level cache to Pongo with a pluggable provider interface. Ships with an in-memory provider backed by lru-cache. Enabled by default. Designed to be minimally invasive to existing code.

Cache Interface

`PongoCacheProvider<T>`

A thin, Pongo-owned interface. No coupling to any third-party API.

type MaybePromise<T> = T | PromiseLike<T>;

interface PongoCacheProvider {
  get(key: string): MaybePromise<PongoDocument | null | undefined>;
  set(key: string, value: PongoDocument, options?: { ttl?: number }): MaybePromise<void>;
  delete(key: string): MaybePromise<void>;
  getMany(keys: string[]): MaybePromise<(PongoDocument | null | undefined)[]>;
  setMany(entries: { key: string; value: PongoDocument; ttl?: number }[]): MaybePromise<void>;
  deleteMany(keys: string[]): MaybePromise<void>;
  clear(): MaybePromise<void>;
}

MaybePromise return types: sync providers (in-memory) return values directly, async providers (Redis) return Promises. await handles both transparently.
Batch methods (getMany, setMany, deleteMany) are first-class. Default in-memory implementation may loop internally, but the interface allows optimized batch ops for external providers.
clear() is internal/advanced — not exposed to typical users. When sharing a cache instance across collections, scoping is handled via key prefixing by Pongo, not by the provider.

Cache key strategy

Pongo manages key prefixing internally: {dbName}:{collectionName}:{documentId}.

The cache provider works with plain string keys — namespacing is Pongo’s concern, not the provider’s.

Event hooks

The Pongo cache wrapper supports basic callbacks:

onHit?(key: string): void
onMiss?(key: string): void
onEvict?(key: string): void
onError?(error: unknown, operation: string): void

These are optional and intended for debugging and future observability integration.

Configuration

`CacheConfig`

type CacheConfig = {
 type: string;               // e.g., 'in-memory', 'redis', etc.
 max?: number;               // max entries (general param, cascades)
 ttl?: number;               // TTL in ms (general param, cascades)
  // type-specific options live here too, keyed by type
 [key: string]: unknown;
} | 'disabled';

Three states:

undefined — inherit from parent level
'disabled' — explicitly turn off caching at this level
CacheConfig object — explicit configuration

Cascading configuration

Cache config can be set at multiple levels. Each level inherits from its parent unless explicitly overridden:

client → db → collection → session → per-operation

Inheritance rules:

General params (max, ttl) cascade down.
If a lower level switches type, type-specific params reset to defaults (not inherited from parent).
Session overrides collection — session is a logical grouping of operations, natural place to override runtime behavior (e.g., disable cache for a bulk import).
Per-operation skipCache?: boolean is the most granular escape hatch.

Passing a cache instance

Users can provide either:

Settings — Pongo creates and manages the cache provider.
A pre-built cache provider instance — Pongo uses it directly.

If settings are the same across multiple collections, Pongo can reuse the same provider instance internally. When a user passes an instance, multiple collections can explicitly share one cache.

Defaults

Enabled by default
max: follow lru-cache recommended defaults (1000)
ttl: follow lru-cache recommended defaults
Default provider: in-memory (lru-cache)

Integration points

Where: `pongoCollection` factory function

Cache logic is added directly inside pongoCollection, not as an external decorator/wrapper. This gives cache operations access to internal state (filter inspection, _version, write outcomes) and keeps observability precise.

Read operations

findOne:

If the filter targets _id, check cache first.
Cache hit → return cached document.
Cache miss → query DB, populate cache, return.
skipCache?: boolean option available.

findMany / other query methods:

If the filter is a list of _id values, check cache for each.
Return cached hits, query DB for misses, populate cache with DB results.
Non-_id filters bypass cache entirely (cache is by-id only).

Write operations

insertOne:

After successful insert, put the document into cache.

insertMany:

After successful insert, put all documents into cache.

updateOne / updateMany / replaceOne:

After successful write, update the cache entry with the new document state.

deleteOne:

After successful delete, evict from cache.

deleteMany:

If deleting by ids, evict those ids from cache.
Filter-based deleteMany does not evict (no way to know affected ids without extra query). Future improvement possible.

Optimistic concurrency

On concurrency error (version mismatch), evict the stale entry from cache. This is critical — a concurrency error means someone else updated the record (e.g., from another node), so the cached version is stale.
If a caller provides expectedVersion and the cached document has _version, compare them in memory. If they don’t match, we know it’s stale without hitting the DB — cheap short-circuit.

`handle` method

Accepts id: string | string[] (overloaded: string returns single result, string[] returns array).
Read phase uses cache by default.
Write phase updates/evicts cache based on outcome.
skipCache?: boolean available in handle options.
For batch: loads all ids from cache, fetches misses from DB in one query, processes handler per document, batch writes to DB, updates cache after commit.

Transactions

Writes update cache only after commit. Not during the transaction — uncommitted data should not enter the cache.
Reads within a transaction check cache by default (most transactions are short-lived, cached data is probably valid). If the transaction modified a document and then reads it, cache miss falls through to DB which returns the correct uncommitted state.
Cache can be disabled per-session: startSession({ cache: 'disabled' }).
Per-session cache settings follow the cascade (session overrides collection).

Error handling

Cache provider errors are swallowed — treated as cache misses. The application never fails because of a cache failure.

Cache is an optimization, not a correctness requirement.
Errors trigger onError hook for debugging.
Future: errors will become metrics and trace events when Pongo’s observability layer lands.

Future considerations (out of scope for v1)

Cache warming: pluggable strategy for pre-populating cache on startup. Interface hook defined but no default implementation shipped.
Query-result caching: caching results of non-_id queries. Complex invalidation — deferred.
Cross-node cache sharing: Redis/Memcached providers. The interface supports it; no provider shipped in v1.
Filter-based deleteMany eviction: would require querying affected ids before delete.
Observability integration: metrics (hit ratio, latency saved, eviction counts), trace attributes, log entries for cache operations.

Implementation approach

Define PongoCacheProvider interface and CacheConfig types.
Implement in-memory provider using lru-cache.
Build Pongo cache wrapper (key prefixing, event hooks, error swallowing).
Wire cascading config resolution (client → db → collection → session → per-op).
Integrate into pongoCollection: intercept _id-based reads, update cache on writes, evict on deletes and concurrency errors.
Extend handle to accept string | string[] with batch support.
Add skipCache to operation options.
Tests: unit tests for cache provider, integration tests for cache + collection operations, e2e tests for concurrency scenarios.

Cheers!

Oskar

p.s. and yes Nano Banana also did pretty well generating the cover of this post!

p.s.2. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

The End of Coding? Wrong Question

Be careful what you wish for, because your wish may come true.

What LLMs revealed is how many people in our industry don’t like to code.

It’s intriguing that now they claim and showcase what they “built with Claude”, whereas usually that means they generated a PoC.

It’s funny, as people still focus on how they’re building, so it’s all about the code. And if that’s the message sent outside, together with the thought that LLMs are already better than “average coder Joe”, then the logical follow-up question is: why do we need those humans in the loop?

I think that most people look at the forest and see trees. The current way of working with LLMs is not scalable. It’s transition phase. I can’t imagine calling myself an engineer and doing ONLY stringly-typed development with chat or markdown.

What I can imagine is getting help from it, and using those tools as a help for research, for generating the OUTPUT, still keeping me responsible for the OUTCOME.

Why am I saying that we’re in the transition phase? As prompting in our natural language is not precise, it’s verbose, and adding a translation layer between our freeform prompts and programming languages is a waste of time (and tokens, which LLM vendors love).

I think that we’ll still be coding, but with some other layer, as LLMs are good with structured input, like programming languages. So we might need other programming languages than we have atm. Might we need different tools to evaluate LLMs’ output to make it deterministic? Might we need a different approach for engineering to make it scalable? Might we need more?

Still, I don’t see those discussions.

I mostly see: noise and celebrities who don’t code showing their beautiful PoCs. And doing a mic drop about the end of coding. Just like PoC represents the whole Software Development Life Cycle.

So, will we code or not?

If the answer is yes, let’s talk about what’s next, then let’s discuss how and when.

If the answer is not, then let’s talk about how our job will change, ask if it’s still engineering, etc.

Let’s try to make our discussions more precise, more focused on the essence, and avoid them from becoming just a bunch of anecdotal evidence.

Noone outside our industry cares how we code unless it changes the cost, quality, or delivery time.

Let’s discuss the impact that matters, rather than just the amount of code we produce (or not).

Now, guess who I am quoting:

Imagine you’re a software application developer. Your programming language of choice (or the language that’s been foisted on you) is Java or Typescript. You’ve been at this for quite a while and your job doesn’t seem to be getting any easier.

These past few years you’ve seen the growth of multiple incompatible architectures. Now you’re supposed to cope with all this and make your applications work in a distributed client-server environment. The growth of the Internet, the World-Wide Web, and “electronic commerce” have introduced new dimensions of complexity into the development process.

The tools you use to develop applications don’t seem to help you much. You’re still coping with the same old problems; the fashionable new object-oriented techniques seem to have added new problems without solving the old ones.

You say to yourself and your friends, “There has to be a better way”!

The Better Way is Here Now

Now there is a better way—it’s our new model. Imagine, if you will, this development world…

It’s still dead simple.

Your development cycle is much faster.

Your applications can be created across multiple platforms. Write your spec once, and you never need to port them—they will be recreated if you without your hand-rolled modification on multiple operating systems and hardware architectures.

Your applications are adaptable to changing environments.

Your end users can trust that your applications are secure, and you can use protection against viruses and tampering through security scans.

You don’t need to dream about these features. They’re here now.

And also this from another source:

When I started interviewing programmers in 2005, I would generally let them use any language or tool they wanted to solve the coding problems I gave them. 99% of the time, they chose Java.

Nowadays, they tend to choose LLMs.

Now, don’t get me wrong: there’s nothing wrong with LLN as an implementation tool.

Wait a minute, I want to modify that statement. I’m not claiming, in this particular article, that there’s anything wrong with LLM as an implementation tool. There are lots of things wrong with it but those will have to wait for a different article.

Instead what I’d like to claim is that LLM is not, generally, a hard enough programming tool that it can be used to discriminate between great programmers and mediocre programmers. It may be a fine tool to work in, but that’s not today’s topic. I would even go so far as to say that the fact that LLMs aere not hard enough is a feature, not a bug, but it does have this one problem.

Well, I cheated you, but only a bit. I changed “Java” to “LLM ” and cut some phrases.

The first one is from “The Java Language Environment” by Sun Microsystems, introducing Java in 1995.

The second one was from Joel Spolsky’s “The Perils of JavaSchools” article, written in 2001.

Let me be clear. I’m not trying to do grandpa talk on the old days and claim that it’s the same old thing.

What I’m trying to say is that we were continuously introducing new abstractions into our development cycle to scale it. By scale, I mean: getting more people to deliver more code. Even Java was invented for precisely this goal. Yes, the one that’s together with craftsmanship madness presented as “the enterprisy complex environment”. In the early days, it was just said that it’ll make us dumber.

The goal of abstraction is not to gatekeep but to allow us to reduce cognitive load. We invented new languages to help us, but then added more components like a distributed environment, multiregion, because of globalisation. For the same reasons, we use cloud-native tooling so we don’t have to deal with it.

Some of the architecture and security tools were commoditised by the cloud. We don’t need to think about much stuff we had to do before.

Will Claude-native do the same?

We don’t need to learn Assembler, C++, Lisp anymore; we have lost a lot of mechanical sympathy. We deal with higher abstractions, but we still engineer solution, we still code, is it a different coding? It is. Is it better or worse? Well, it is how it is. As Gerald Weinberg said:

Things are the way they are because they got that way

And now the question is whether we’re fine with the way we’re doing stuff, and where we’re getting to.

Personally, I don’t think that stringly-typed markdown or chat-based design will be the thing in the future.

Knowing how skilled we were always with:

breaking down tasks into smaller chunks,
writing precisely what we had in mind,
thinking before doing,
waging tradeoffs,

I’m optimistic about the Spec-Driven Design idea. It’s going to be great.

Not.

If we want to call ourselves engineers, we need to put more structure and determinism.

I agree that reviewing all code generated by the GenAI is not sustainable.

But I also don’t think that generating tons of code is, in general, sustainable.

With the current state of the art we have, sure, that’s some solution to just generate based on the tools we have.

But let’s start to think what’s next.

Simon Wardley brought an interesting point:

That said, it is fine for an entire culture to decide that producing outputs matters more than understanding mechanisms. You only have to compare the practical engineering of the Roman Empire and the loss of inquiry from science in the Hellenistic age to see this. When the Roman Empire collapsed, the practical knowledge embedded in those institutions (how to maintain aqueducts, how to produce certain grades of concrete) was lost remarkably quickly. Not because people decided to forget it, but because the knowledge was procedural, embedded in chains of practice rather than recorded as transferable understanding. When the chains of practice broke, that embedded knowledge went with them. We had to rediscover the art of inquiry (i.e. Science) to bring them back.

So if we don’t code, how do we hone our skills? How will newcomers from LLMSchools (using Joel’s terms) be able to decide whether something is wrong or right? I don’t think that you can be good on something without doing it.

The paradox with code is that it’s not an asset; it’s a liability. Some say that code doesn’t matter, only proper design. But how do you define “proper design”? Yes, code style doesn’t matter as long as it works as expected. But…

But code eventually matters, as that’s the source of truth for what’s on production. As Alberto Brandolini said:

It’s developers’ (mis)understanding, not domain experts’ knowledge, that gets released in production.

Now, it’s the developers’ and LLMs’ misunderstandings that are deployed to production, not the expert’s knowledge. Neither the markdown spec.

And coding is just one danger.

Outsourcing thinking is an even more dangerous path, as:

If LLMs are doing everything, then what are humans for? Aren’t we cutting the branch on which we’re sitting?
LLMs are statistical parrots. They repeat the most possible answer. Which means mediocre. This can still be fine enough for many cases, but for those we want to make a difference for? Definitely not.
Just like we’re losing our coding skills by not doing them, we’re losing design skills by not practising them.

Of course, whatever happens, LLMs will stay with us. How and where it’s hard to say. Unless you have a Magic 8 Ball of 100% correct predictions. I don’t.

That’s why I’d like our industry to finally start mature discussions on the real impact. I would like us to stop acting like children, bragging about generating code and then claiming that code doesn’t matter.

I’d like to think about how to reshape our SDLC process and make it sustainable.

I’d like us to think about what tools we need, and how to change what we have.

If we won’t finally start to do it, then things will be the way they are because they got that way.

And it might not be what we wished for.

Cheers

Oskar

p.s. to kinda prove that I’m more sceptic and pragmatic than hater, I recently started playing with building an Agent with Emmett to better understand those tools. If you’d like to read about the findings and honest thoughts I have while doing it, please comment!

Parse, Don't Guess

Last time, I shared with you how sneaky I was on transaction handling.. Today, the opposite: I’ll tell you how I fixed the issue when I tried to be too sneaky. I already told you that Sneaky Code Bites Back. The moral? Do as I tell, not how I do.

In some environments, we’re spoiled. We’re getting a lot from a Base Class Library or standard frameworks, so we stop thinking that those issues can exist. For instance, serialisation. Do you know how many data types JSON has? 6. Six. Sześć.

Exactly those:

string,
number,
boolean,
object,
array,
and (TADA!) null.

What about number precision and size? It is. That’s what I can tell you, but it’s not enough, e.g., to keep big int/long, etc. What about Dates? Also, there are none. I wrote about it longer here or how much fun that brings.

If you use statically typed languages and runtimes like C#, Java, etc., your serialiser can, in addition to parsing, perform mapping and, sometimes, validation. And it can also be tricky, as nicely Alexis King put in her “Parse, don’t validate”.

If you’re in a dynamic environment, like JavaScript, then you’re left with parsing and explicit mapping afterwards. What about TypeScript? Same case, types are only used during compilation, then erased and not visible at runtime. So, the place where we do parsing.

Because JSON was defined a long time ago, JavaScript moved on and now supports bigints (Big Integers) and Dates natively (what an achievement!), which created a gap I wanted to fill.

As you know from my previous articles (e.g. this one), big integers are quite important in distributed processing. You can represent the position in log with them. Since your log may be quite long, regular numbers aren’t enough. Or they’re long enough, until they overflow, then they’re not anymore.

I’m using those bigint types extensively in internals in Emmett and Pongo. And I store them in JSONs. I store them as alphanumeric strings, because strings don’t have a max length (or at least I don’t know such).

So, for instance, an event payload can look like:

{
  "type": "InvoiceIssued",
  "data": {
    "invoiceNumber": "123",
    "version": 1,
    "issuer": "John Doe",
    "issuedAt": "2026-02-23T14:07:20Z"
  },
  "metadata": {
    "streamPosition": "3",
    "globalPosition": "928391"
  }
}

As you can see in the metadata stream and global positions, the values are bigints (even if they’re smaller than the maximum value), and data can also use bigints if the user decides to (e.g., invoice number).

And encoding data is simple: you convert it to a string, call it a day. But how to get it back?

And here’s where my struggles started. How do you know that someone intentionally used bigint when they just wanted to store a number as a string?

There are several options. The first one is: encode value.

We could store it, for instance, as:

prefixed value: “_bigint:928391”. But then you need to find a prefix that will be unique enough not to cause conflicts,
nested object, e.g. { “_kind”: “bigint”, value: “928391” }.

Then, either based on the prefix or the object structure, we could automatically decode the value. Still, ther creates other issues, as the structure no longer matches the original value. If we’re just storing and retrieving, that shouldn’t be so bad, but… But remember that in Pongo I’m allowing the use of PostgreSQL and SQLite as document databases, supporting such queries:

const invoices = pongoDb.collection<Invoice>("invoices");

const invoiceNumber = 123n;
const invoice = await invoices.findOne({ invoiceNumber });

That gets translated into a fancy JSONB SQL query.

Of course, I could work around it by encoding the value, but… But I was lazy!

I decided to use a Get Out of Jail Free Card and just treat all strings with numbers as bigints. Sneaky. And it will get even sneakier.

In JavaScript, JSON.parse accepts a parameter that allows you to provide custom mapping logic. I decided to use it and check if the string is alphanumeric, and gulp, I’ve used a regular expression to parse it:

const bigIntReviver: JSONReviver = (_key, value) => {
  if (typeof value === 'string' && /^[+-]?\d+n?$/.test(value)) {
    return BigInt(value);
  }

  return value;
};

Yes, it’s either DNS or Regex. Or both.

I explained in another article that JavaScript runtime doesn’t like where you do CPU-heavy computations.

Small Regex isn’t CPU-heavy, but if you consider that ther will be done for each string in each document or event you try to deserialise, and multiply that by the number of concurrent requests? That can cause the JavaScript event loop to freeze.

What’s more, I plugged that automatically into node-postgres driver custom type handling, so each JSONB deserialization goes through it.

Again, not shit, Sherlock. I should have known it wasn’t the best choice, but I was busy trying to be sneaky at that moment.

Fortunately, a user, Dawid, benchmarked and noticed CPU freezes. It wasn’t catastrophic, but clearly needed a fix.

The Shift

And here I had several options. I could keep hacking on the same idea- maybe replace the Regex with a simpler string check, still globally. Or I could just ignore bigints during deserialisation entirely, let them stay strings, call it a day, move on. Or I could apply the encoding I mentioned earlier, prefixed values, and nested objects. All of those would fix the performance issue. And all of those would be the same kind of wrong choice I already made: trying to solve a schema problem without the schema.

Because that’s the actual mistake here, not the Regex. The pg driver has no idea what your schema looks like. It doesn’t know that “928391” is a bigint and “John Doe” is a name. It doesn’t know that “123” is an invoice number (bigint!) and “90210” is a zip code (string!). I asked it to guess, and it guessed wrong, because there is no right guess at that level.

Enough is enough. I had been planning to do ther properly for a while, and the performance issue gave me the push I needed.

Old rule says: “Make it work, make it right, make it pretty”. I had “make it work” covered for a long time. Now it was time for “make it right”.

And honestly? It wasn’t that hard. Maybe because “make it work” came first, I already understood the problem well enough to see the shape of the solution.

In Pongo, I dropped the automatic bigint parsing from the driver entirely. If you want bigint or date parsing, you say so at the client level:

const client = pongoClient({
  driver: databaseDriver,
  connectionString: postgresConnectionString,
  serialization: {
    options: {
      parseBigInts: true,
      parseDates: true,
    }
  },
});

By default, strings stay strings. You opt in. I didn’t want to break things for users who don’t care about bigint precision or don’t have performance-sensitive workloads. The serializer became an explicit parameter passed down to each query, each collection, each operation- instead of a global that silently changed everything.

That was the “make it right” part for Pongo. But disabling alone isn’t a solution, it’s a band-aid. Users who need bigints and dates still need a way to get them back after deserialisation. The question is: where does that conversion happen?

And that’s where upcasting comes in. Let me start with a simple example in Pongo, then build up.

Say you have a user document. In the database, dates are stored as ISO strings, and the version counter is a numeric string (because JSON). But in your application, you want proper Date objects and bigints:

type UserDocStored = {
  name: string;
  createdAt: string;
  lastLogin: string;
};

type UserDoc = {
  name: string;
  createdAt: Date;
  lastLogin: Date;
};

The upcast function does the conversion:

const upcast = (doc: UserDocStored): UserDoc => ({
  name: doc.name,
  createdAt: new Date(doc.createdAt),
  lastLogin: new Date(doc.lastLogin),
});

You wire it into the collection, and every read goes through it:

const collection = pongoDb.collection<UserDoc, UserDocStored>(
  'users',
  {
    schema: { versioning: { upcast } },
  },
);

What’s in the database:

{ "name": "Alice", "createdAt": "2024-01-15T10:30:00.000Z", ... }

What you get back:

{ name: 'Alice', createdAt: Date, ... }

That’s all. new Date(str) is cheap. Running a Regex against every string in the document is not. The CPU freeze Dawid spotted came from that check running millions of times at the driver level for every field on every concurrent request. With upcasting, the conversion runs only for the fields you declared, in a plain function, no Regex.

But ther is just type mapping - the simplest case. As I wrote about in my serialisation article, the explicit mapping pattern is useful for much more than just fixing types. It’s the same pattern you need for schema versioning. It defines the stored schema and the application schema separately together with function to transform one into the other.

Let’s say business requirements changed. You now need to group user data differently: a profile object for identity data, and a timestamps object for temporal data. The V1 documents are flat. The new V2 shape is nested:

type UserDocV1 = {
  name: string;
  createdAt: string;
  lastLogin: string;
};

type UserDocV2 = {
  profile: {
    name: string;
  };
  timestamps: {
    createdAt: Date;
    lastLogin: Date;
  };
};

Ther isn’t just a type change like string-to-Date anymore. The structure itself is different. Flat fields became nested objects, and field names moved into sub-objects. And you have thousands of V1 documents already stored. You can’t migrate them all at once (or don’t want to, because it’s risky, and some consumers might still expect V1). But your application now expects V2.

Compatibility FTW

Ther is where backward and forward compatibility come in.

Backward compatibility means: old data still works. V1 documents stored months ago need to be readable by the V2 code. The upcast handles ther. It reads the document in whatever shape it has and transforms it into V2.

Forward compatibility means: new data doesn’t break old consumers. If you have another service or an older deployment that still reads the V1 format, it needs to keep working. The downcast handles ther. When storing V2 documents, it writes the V1 fields alongside the V2 fields, so older readers can still find what they expect.

Together:

type StoredPayload = UserDocV1 & UserDocV2;

const upcast = (doc: StoredPayload): UserDocV2 => ({
  profile: doc.profile ?? { name: doc.name },
  timestamps: {
    createdAt: new Date(doc.timestamps?.createdAt ?? doc.createdAt),
    lastLogin: new Date(doc.timestamps?.lastLogin ?? doc.lastLogin),
  },
});

const downcast = (doc: UserDocV2): StoredPayload => ({
  name: doc.profile.name,
  createdAt: doc.timestamps.createdAt.toISOString(),
  lastLogin: doc.timestamps.lastLogin.toISOString(),
  profile: doc.profile,
  timestamps: doc.timestamps,
});

Look at the upcast: if the nested profile or timestamps fields exist (document was written by V2 code), it uses them. If they don’t exist (for the old V1 document), it falls back to the flat fields. One function handles both old and new documents: that’s backward compatibility.

And look at the downcast: it writes name, createdAt, lastLogin as flat string fields (V1 shape) alongside profile and timestamps (V2 shape). A service still reading V1 sees the flat fields and works fine. A service reading V2 sees the nested ones. That’s forward compatibility.

You wire both into the collection:

const collection = pongoDb.collection<UserDocV2, StoredPayload>(
  'users',
  {
    schema: { versioning: { upcast, downcast } },
  },
);

From here, your application code only deals with V2. The collection handles the translation in both directions:

const v2Doc: UserDocV2 = {
  profile: { name: 'Alice' },
  timestamps: {
    createdAt: new Date('2024-01-15T10:30:00.000Z'),
    lastLogin: new Date('2024-06-20T14:45:00.000Z'),
  },
};

await collection.insertOne(v2Doc);

What’s stored (downcasted, both shapes for compatibility):

{
  "name": "Alice", 
  "createdAt": "2024-01-15T10:30:00.000Z",
  "lastLogin": "2024-06-20T14:45:00.000Z",
  "profile": { 
    "name": "Alice" 
  },
  "timestamps": { 
    "createdAt": "2024-01-15T10:30:00.000Z",
    "lastLogin": "2024-06-20T14:45:00.000Z" 
  } 
}

Then you can read it back with:

const doc = await collection.findOne({ ... });

And get upcasted to V2 data in your application code:

{
  "profile": { 
    "name": "Alice" 
  },
  "timestamps": { 
    "createdAt": "2024-01-15T10:30:00.000Z",
    "lastLogin": "2024-06-20T14:45:00.000Z" 
  } 
}

Same collection, V1 and V2 documents coexisting. insertMany, replaceOne, findOne: all go through the upcast/downcast. No batch migration needed. You roll out the new code, and old documents are handled transparently.

There’s another thing the downcast gives you: querying remains backwards-compatible. Because the downcast writes the flat V1 fields alongside the nested V2 ones, a query like collection.findOne({ name: ‘Alice’ }) still works even though V2 code doesn’t use name directly anymore. The V1 field is there in the stored document. That matters if you have queries or indexes built against the old shape. They don’t break.

Now, for events, ther matters even more. In event sourcing, stored events are immutable- the log is append-only, and you don’t modify what was already written. I wrote about versioning patterns in more detail. The core idea is: your business evolves, your code evolves, your event schemas evolve, but the events in the store stay as they were. You can’t go back and rewrite them (well, you can, but you really shouldn’t). Upcasting is how you bridge the gap.

For Emmett, the same pattern works at the event store level. You define the stored shape (what JSON gives you from the database) and the application shape (what your code works with):

type ShoppingCartOpenedFromDB = Event<
  'ShoppingCartOpened',
  { openedAt: string; loyaltyPoints: string }
>;

type ShoppingCartOpened = Event<
  'ShoppingCartOpened',
  { openedAt: Date; loyaltyPoints: bigint }
>;

And an upcast that handles each event type:

const upcast = (event: Event): ShoppingCartEventWithDatesAndBigInt => {
  switch (event.type) {
    case 'ShoppingCartOpened': {
      const e = event as ShoppingCartOpenedFromDB;
      return {
        ...e,
        data: {
          openedAt: new Date(e.data.openedAt),
          loyaltyPoints: BigInt(e.data.loyaltyPoints),
        },
      };
    }
    case 'ShoppingCartConfirmed': {
      const e = event as ShoppingCartConfirmedFromDB;
      return {
        ...e,
        data: {
          confirmedAt: new Date(e.data.confirmedAt),
          totalCents: BigInt(e.data.totalCents),
        },
      };
    }
    default:
      return event as ShoppingCartEventWithDatesAndBigInt;
  }
};

You pass it when reading a stream:

const { state } = await eventStore.aggregateStream<
  ShoppingCartState,
  ShoppingCartEventWithDatesAndBigInt
>(shoppingCartId, {
  evolve: evolveState,
  initialState,
  read: { schema: { versioning: { upcast } } },
});

Or in a command handler:

const handle = CommandHandler<ShoppingCart, ShoppingCartEvent>({
  evolve,
  initialState: () => ({ ... }),
  schema: { versioning: { upcast: upcastDatesAndBigInt } },
});

The difference with events is that you can’t update them in place. For documents, you have both directions: upcast on read, downcast on write. For events, upcasting is the main tool because the event store is append-only. Old events stay as they were written. But downcasting has its place too.

Consider ther: you have a projection or a subscriber that was built months ago against the old event schema. Maybe it’s a read model that listens to ShoppingCartOpened and expects clientId as a flat string. But your current code evolved. Now ShoppingCartOpened carries a client object with id and name:

// What old subscribers expect
type ShoppingCartOpenedV1 = Event<
  'ShoppingCartOpened',
  { clientId: string; openedAt: string }
>;

// What current code produces
type ShoppingCartOpenedV2 = Event<
  'ShoppingCartOpened',
  { client: { id: string; name: string }; openedAt: Date }
>;

Upcasting enables the current code to read older events. The ones stored with just clientId. Downcasting helps old subscribers consume new events. It transforms the new client object back into the flat clientId they expect. Same principle as with documents, but especially important here because event subscribers often live in separate services or deployments that you can’t update all at once.

And the same upcast function that started as simple type mapping string → Date, string → bigint handles ther structural change too. You just add another case to the switch:

case 'ShoppingCartOpened': {
  const e = event as ShoppingCartOpenedV1;
  return {
    ...e,
    data: {
      client: { id: e.data.clientId, name: 'Unknown' },
      openedAt: new Date(e.data.openedAt),
    },
  };
}

Old events get the client object synthesised from the flat clientId. New events already have it. The evolve function only deals with the V2 shape.

And here’s where things started to click for me. I added upcasting to fix the bigint problem: explicit type mapping instead of a Regex. But the same mechanism, without any changes, also handles structural versioning. The simple string → Date mapping from the first example is the same code path as the clientId → client migration above. It’s one function, one place, one pattern for all of it: type coercion, field restructuring, schema migration.

Right decisions stack

Right decisions stack. The Regex hack was blocking the slot where upcasting should have been all along. Once I removed it, the performance got fixed, and I got schema versioning on top. One fix created room for the next one, which created room for the next. That doesn’t happen when you keep patching around the same bad decision.

Looking back, maybe the Regex wasn’t the wrong first move. The rule is “make it work, make it right, make it pretty”, in that order. The Regex made it work. It had performance problems, but it still let me ship and learn where the real problem was. If I had tried to design the upcast/downcast system from scratch, without having lived with the Regex for a while, I might have over-engineered it or missed the connection to schema versioning entirely. The understanding came from living with the shortcut.

Dawid raised a performance issue with Pongo projections, but the same Regex was running in Emmett as well. I could have fixed it in one place and called it a day. Instead, I used it as a push to do the thing I’d been planning anyway and applied it to both Pongo and Emmett to keep things consistent. Because I already understood the problem well enough, “make it right” turned out easier than I expected.

You can recover from shortcuts. You should. But you also shouldn’t be afraid to take them in the first place, as long as you come back and do it properly.

Full changes:

Cheers!

Oskar

p.s. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, putting pressure on your local government or companies. You can also support Ukraine by donating e.g. to Red Cross, Ukraine humanitarian organisation or donate Ambulances for Ukraine.

How I cheated on transactions. Or how to make tradeoffs based on my Cloudflare D1 support

We’re being told that software design is the art of making tradeoffs. But… Are we taught how to make them?

Not that it’s easy to teach tradeoffs, it’s a subtle art of explanation. You need to provide enough context and be precise so others don’t treat tradeoffs as a general best practice. Because they’re usually not such, they typically come from the muddy banks of the Wishkah River.

I think that a decent way is to tell the story. Not the fairy tale, but the specific tradeoff applied in practice. That’s what I’m going to do today: tell you how I cheated on database transactions.

Dumbo

Do you know Dumbo? Of course, flying elephant, such a sweetie. It’s also a codename of my Open Source project. I didn’t tell you about it so far, as it’s a shared dependency for Pongo and Emmett, responsible for:

connection pooling,
safe connection lifetime management,
handling transactions,
migrations,
SQL execution.

Not that small a scope for a Dumb tool, aye? Still, the intention is to make usage dumb, hiding the weirdness of a specific SQL database, so I solve it once and don’t need to be constantly distracted thinking about it, but focus on:

How to append and process events the best way in Emmett,
How to make the best JSON handling and translation into specific SQL dialects in Pongo.

Dumbo usage is quite simple:

import { dumbo } from '@event-driven-io/dumbo';
import { pgDumboDriver } from '@event-driven-io/dumbo/pg';

const pool = dumbo({ connectionString, driver: pgDatabaseDriver });

You need to set up a specific database driver (e.g. pg for PostgreSQL or sqlite3 for SQLite), as Dumbo now supports multiple relational databases.

Having that, you can do stuff like:

import { SQL } from '@event-driven-io/dumbo';

await pool.execute.batchCommand([
  SQL`CREATE TABLE test_users (id SERIAL PRIMARY KEY, name TEXT)`,
  SQL`INSERT INTO test_users (name) VALUES ('Alice'), ('Bob')`,
]);

And also do queries:

const count = pool.execute.query<{count: number}>(
  SQL`SELECT COUNT(*) as count 
      FROM test_users 
      WHERE ${SQL.in('id', userIds)}`,
);

It’ll handle query parameterisation, data escaping, etc.

It can also handle transactions:

const users = await pool.withTransaction(async (tx) => {
  await tx.execute.command(
    SQL`INSERT INTO test_users (name) 
        VALUES (${firstUserName}), (${secondUserName})`,
  );

  return execute.query<User>(
    SQL`SELECT *
        FROM test_users 
        WHERE ${SQL.in('id', userIds)}`,
   );
});

As simple as this looks, you should already have the question about the first tradeoff I made in your head.

Why on earth I think that’s a sane move to write my own multi-database driver?!

Well, I agree, that’s not the best move at the first glance, but let me explain to you why in MY context this actually makes more sense:

Yes, there are tools like Knex, Kysely, Drizzle, etc. in Node.js land that handle similar stuff. They’re nice, I really like them, I really do, but… But they are all big and are bringing a lot of their conventions, and when I’m building storage tools like Emmett and Pongo, I need to have more control. I don’t want those tools and their limitations drive my architecture decisions. I also don’t want to be surprised when the creator decides to drop working on it, or when I become a victim of a supply chain attack. Still, when I want a new kitchen table, I don’t start by going to the forest with a saw. I’m still using existing database drivers, what’s more I’m allowing people to choose the one they prefer. I just don’t want to be driven (and also my users be driven) by some other higher abstraction tool.
Even with that, this decision can seem like a bold “how hard can it be?!” statement. And it is, but… But I’ve built, or was co-authoring, such tools in the past. Some were proprietary, some were Open Sourced (see Weasel). And they served me well.
And last but not least, the last point. Well, I only had those two above. OK, I can add that I didn’t plan to make it a general-usage tool, just a small one for my needs.

So I did.

And here we’re at the moment when I had to cheat on transactions because of those decisions.

Cloudflare D1

All relational databases seem similar, but only until you start using them extensively, or until you need to write your own storage library. Then you learn stuff that not necessarily the stuff you’d like to spend time on. For instance:

What’s the difference between databases like PostgreSQL and single-threaded databases like SQLite or DuckDB, and how concurrent processing can be surprising
or that sqlite3 only calls the first query, but the next silently ignores,
etc.

Still, well, abstractions like Dumbo give the possibility to massage such cases behind the scenes.

And then Cloud Databases and SaaS Databases came into play, like Cloudlfare D1.

I was motivated by the generous sponsorship from Sam Hatoum to make supporting Emmett and Pongo a top priority. Thanks Sam, appreciate that! And I made it, but I had to cheat a bit.

Databases like Cloudflare D1 and Supabase expose databases as pay-as-you-go services. They optimise deployment, making it highly scalable, so you don’t need to care about it. It’s cost-effective for start-ups and scenarios when you’re not under a huge load. If you are, then you’ll need to pay more, but they also give you autoscaling at least.

They do it by exposing the database API through HTTP API, for instance PostgREST. This gives them easier management around throughput, security, etc., as each call is made as an HTTP request through the exposed API.

Yet, that kills some options like: ekhm, transactions. The challenge with transactions is that they don’t scale (that’s why MongoDB is WebScale). They don’t scale, as you’d need to open a transaction, do some freehand operations, and then commit or rollback. That means (typically) you need to keep a connection open during that time. If you’re building SaaS, it’s a no-go, because sneaky users would open it for a few hours, do crazy stuff, and kill your SaaS resources’ utilisation.

But, boy, aren’t transactions one of the selling points of relational databases? They do, so how to proceed?

Then you need to cheat, as I did.

Repeatable reads, batches, etc.

Cloudflare D1 doesn’t provide transaction data for the reasons I outlined, but it doesn’t leave us without other tools. Those two tools to let me do the smoke and mirrors trick are:

sessions,
SQL batches.

What are sessions? Per Cloudflare Docs:

A session encapsulates all the queries from one logical session for your application. For example, a session may correspond to all queries coming from a particular web browser session. All queries within a session read from a database instance which is as up-to-date as your query needs it to be. Sessions API ensures sequential consistency for all queries in a session.

Essentially, that means that we’re getting repeatable reads. So when we’re starting specific sessions, they will be handled sequentially.

And now, the second ingredient: Batches. Per Cloudflare docs

Sends multiple SQL statements inside a single call to the database. This can have a huge performance impact as it reduces latency from network round trips to D1. D1 operates in auto-commit. Our implementation guarantees that each statement in the list will execute and commit, sequentially, non-concurrently.

Batched statements are SQL transactions. If a statement in the sequence fails, then an error is returned for that specific statement, and it aborts or rolls back the entire sequence.

To send batch statements, provide D1Database::batch a list of prepared statements and get the results in the same order.

So Cloudflare didn’t allow us to do the full, freehand transaction, but they allowed us to send multiple statements that internally will be executed as a SQLite Transaction. When the request is handled, it’ll open a transaction, run the statements, and return the results.

Cool, let’s mix this soup together, as having that, I decided to:

Fail automatically if someone tries to create a transaction on Cloudflare D1 with an error:

D1 does not support SQL transactions (BEGIN/COMMIT/ROLLBACK/SAVEPOINT). Use { mode: “session_based” } to opt-in to session+batch semantics, or use ‘connection.execute.batchCommand() for atomic multi-statement execution.

Then they get clear information on the first try.

Allow users to explicitly open a session-based transaction by providing mode:

const users = await pool.withTransaction(
  async (tx) => {
    await tx.execute.command(
      SQL`INSERT INTO test_users (name) 
      VALUES (${firstUserName}), (${secondUserName})`,
    );

    return tx.execute.query<User>(
      SQL`SELECT *
              FROM test_users 
              WHERE ${SQL.in('id', userIds)}`,
    );
  },  
  { mode: 'session_based' }
);

When they do it, they will need to be aware of the limitations of the tool they have. So that this will internally create a D1 session, and only handle a single batch of operations properly. We’re mimicking the sequential processing by the session-based repeatable reads capability. Still, we need to remember that we won’t be able to roll back changes across multiple statements. Only a single command or batch command is an atomic operation.

We can’t, for instance, run a batch of updates, and fail the whole batch if one update didn’t change any record. The batch will only fail if the database throws an exception. An exception can be in SQLite only called by a table constraint or trigger.

By making this choice to require explicit mode and naming it explicitly, I didn’t manage to cover all cases, but at least made it safe, so people need to learn about this non-default behaviour and be more careful about it. When designing an API, it’s usually better to start with a more strict option and do a bit of “scarification” sometimes.

Still, when I implemented that in Emmett and Pongo, I intentionally used it to enable event appends, but document operations, etc.

If you’d like to try it, you can check Emmett’s or Pongo’s beta versions.

For Pongo, you can install it with:

npm install @event-driven-io/[email protected]

And use it as:

import { d1PongoDriver } from '@event-driven-io/pongo/cloudflare';

const client = pongoClient({
  driver: d1PongoDriver,
  database,
  transactionOptions: { mode: 'session_based' },
});

Or in Emmett by installing:

npm install @event-driven-io/[email protected]

And use it as:

import { getSQLiteEventStore } from '@event-driven-io/emmett-sqlite';
import { d1EventStoreDriver } from '@event-driven-io/emmett-sqlite/cloudflare';

const eventStore = getSQLiteEventStore({
  driver: d1EventStoreDriver,
  database,
});

Still, even if you don’t care about Emmett, Pongo, and my Open Source project, I hope that this will give you decent inspiration for your own tradeoffs analysis.

I hope that you learned a bit about how design APIs work, how to check the guarantees of your tools and learn to walkaround them when you have to.

Cheers!

Oskar

On rebuilding read models, Dead-Letter Queues and Why Letting Go is Sometimes the Answer

In the last article, I explained how to rebuild Event-Driven Read Models in a safe and resilient way. I asked readers to let me know if they find any blind spots in my design.

Well, I found one myself.

This article is about that edge case, but more importantly, it’s about the rabbit hole I went down thinking about how to “fix” it. At the end of this hole, there is a nice learning about dealing with distributed systems. Sometimes the best engineering isn’t about preventing all failures. It’s about recognising your blind spots and making sure they don’t catch you off guard.

Everyone has a plan until they get punched in the mouth

Let me recap the situation described in the previous article. We’re using PostgreSQL to store events and read models. We want to rebuild an inline projection, one that’s applied in the same transaction as the appended event.

In the design from my previous article, during a rebuild, we:

Mark the projection as “rebuilding”.
Skip inline projections (the rebuild process will catch up anyway).
Process all historical events.
Mark the projection as “active” again.

The hybrid locking strategy with advisory locks and status checks ensures that inline projections know when to skip and that only one rebuild runs at a time.

Sounds solid. Here’s where it breaks:

The rebuild process checks for new events, sees none (because event 1000 is still in an uncommitted transaction), declares victory, and sets the projection to active. Meanwhile, the append transaction has already decided to skip the inline projection. When it finally commits, the event exists but was never projected. The rebuild process is done, and only new events will update the projection during the next update to the new event append.

We could handwave it with it won’t ever happen!. But, well, under load, this will happen eventually. You might not even notice it, but it will. The timing window is small, but with enough throughput, it becomes a certainty.

The Rabbit Hole of “Fixes”

My first instinct was to engineer my way out of this. Surely with enough clever coordination, we can close this gap?

Let me walk you through the rabbit hole.

Attempt 1: Wait for In-Flight Transactions

PostgreSQL provides pg_snapshot_xmin(pg_current_snapshot()) which tells us the oldest transaction that’s still running. The idea: after processing events, wait until all potentially-skipping transactions have completed before marking the projection as active.

Why it fails: We don’t know what we’re waiting for. While we wait for existing transactions to complete, new transactions start and make their own skip decisions. The target keeps moving. We can never “catch up” because new skips happen while we’re waiting for old ones to become visible.

Attempt 2: Use Transaction IDs as Boundaries

PostgreSQL assigns monotonic transaction IDs to each transaction when it starts. This seems useful—what if we record a “sealing” transaction ID when we’re ready to complete the rebuild, and use it to make decisions?

The logic would be:

When rebuild is ready to complete, record sealing_txid = current_transaction_id

Any event from a transaction with ID lower than the sealing point was “in flight” during rebuild, so it should be handled by async Events from transactions with higher IDs started after sealing, so they can safely use inline projections (the read model will be ready)

Sounds reasonable?

Here’s why it doesn’t work.

The core problem: transaction ID order ≠ commit order.

When a transaction starts, PostgreSQL assigns it the next available transaction ID. But transactions don’t commit in the order they started. A transaction that started earlier (lower ID) might take longer to complete and commit after a transaction that started later (higher ID).

Let’s trace through a concrete scenario:

Here’s what happened:

Tx 99 started first and got the lowest transaction ID. It inserted an event and decided to skip the inline projection (status was ‘rebuilding’). But then it got slow—maybe network latency, maybe the application did other work.
Tx 100 (the rebuild) started second, recorded sealing_txid = 100, and prepared to complete.
Tx 101 started third. It checked: “Is my transaction ID (101) >= sealing_txid (100)?” Yes, so it assumed the read model was ready and processed its inline projection. It committed successfully.
Tx 100 marked the projection as active and committed.
Tx 99 finally committed. But it had already decided to skip the projection back when status was ‘rebuilding’. That decision was made, the skip happened, and the event is now missing from the read model.

The fundamental issue: we can’t see uncommitted transactions. When Tx 100 set the sealing point, it had no way to know that Tx 99 was still out there, holding an event that would skip projection. Transaction 99 is invisible until it commits, but by then it’s too late.

You might think: “Just wait until all transactions before the sealing point have committed!” PostgreSQL even provides pg_snapshot_xmin(pg_current_snapshot()) which tells you the oldest active transaction. But this leads us back to Attempt 1—while we wait, new transactions start and make their own skip decisions. The target keeps moving.

I wrote about this exact problem in How Postgres sequences issues can impact your messaging guarantees. The same visibility challenges that affect outbox ordering apply here. Transaction IDs are useful for ordering within committed data, but they can’t help us coordinate with transactions that haven’t committed yet.

Attempt 3: Lock Appends During Transition

What if we use advisory locks more aggressively? The idea: when rebuild is ready to complete, acquire an exclusive lock that blocks the entire append path. While holding this lock, flip the status to ‘active’. No appends can be in progress during the flip, so no race condition, right?

Here’s the proposed flow:

Rebuild finishes processing historical events
Acquire exclusive advisory lock (all new appends must wait)
Set status = ‘active’
Release lock
Waiting appends resume and sees ‘active’, processes inline normally

It should work. In theory. We’re creating a synchronisation point where nothing can slip through. But there’s a subtle problem: the skip decision was already made inside the append transaction.

An append transaction doesn’t just check the lock and status at one instant. It does several things:

BEGIN transaction
INSERT the event
Try to acquire a shared advisory lock
Check projection status
Decide: process inline or skip?
Execute that decision
COMMIT

The decision in step 5 happens inside the transaction. If the transaction saw ‘rebuilding’ status, it was skipped. That decision is now part of the transaction’s pending work. The transaction might be waiting to commit or doing other work, but the skip decision has already been made.

Our exclusive lock in the completion flow blocks step 3 - new transactions can’t acquire the shared lock. But what about transactions that already passed step 5 and are just waiting to commit?

The timing here is tricky. Advisory locks in PostgreSQL can be either:

Transaction-scoped (pg_advisory_xact_lock): released automatically when transaction commits
Session-scoped (pg_advisory_lock): held until explicitly released or session ends

If we use transaction-scoped locks for inline projections (which makes sense—you want the lock tied to the transaction lifetime), then the append transaction might have already released its shared lock by the time we try to acquire an exclusive lock. The lock protected the status check, but the transaction is still running with its skip decision already made.

Even if we could perfectly synchronise the lock acquisition, there’s another problem: advisory locks are session-scoped, not durable.

If the connection dies while holding the exclusive lock:

The lock releases immediately (that’s how advisory locks work)
We might have partially updated the status
The system is now in an unknown state
Other transactions resume with potentially inconsistent data

We can’t build reliable coordination on something that vanishes when connections fail. This is exactly why the original article combined advisory locks with persistent status checks—but that combination doesn’t solve this particular race condition.

TLDR on Attempts

Every “fix” follows the same pattern:

Identify a coordination point.
Discover there’s a window before that point we can’t see.
Try to close that window.
Create a new window somewhere else.
Repeat.

We’re not solving the problem. We’re relocating it.

So is it a bug in the initial design? Depending how you look on that. I see it as an example of fundamental property of concurrent systems. PostgreSQL’s isolation guarantees mean uncommitted transactions are invisible to other transactions. That’s a feature, not a bug - but it means there’s always a window we can’t see into.

Stop Fighting, Start Tracking

After chasing my tail through various “solutions,” I stepped back and asked a different question.

Instead of: “How do we prevent events from being skipped?” (which requires blocking appends or seeing into uncommitted transactions—both unacceptable because of performance, guarantees etc.)

I started to think on: “How do we know when an event was skipped?” and “How do we ensure skipped events get processed eventually?”

Both of these are solvable.

If we can’t prevent skips from happening, let’s make them visible. When an inline projection skips, it could record that it skipped in the same transaction as the event append.

If the append transaction rolls back, the skip record also rolls back (there’s no event to worry about). If it commits, we have a durable record of exactly what was skipped.

In Emmett, I’m using a dedicated emt_system_messages table rather than reusing the regular emt_messages table or creating a simple “skipped events” table. This might seem like over-engineering—why not just create a simple table with projection ID and event position? Or why not just reuse the existing messages table?

Why a dedicated system messages table?

The regular emt_messages table is for business events—the actual domain events that drive your application. Mixing system-level concerns (like “this projection skipped this event during rebuild”) with business events pollutes the event log and makes it harder to reason about. A simple “skipped_events” table with just (projection_id, event_position) could be enough, but in the long term, it may be hard to maintain. As systems evolve, we’ll have more cases where we might want to skip events. Plus, having system events recorded with all metadata gives us full observability of the internals of our system. We could even make tables for projections and processors’ status, built as read models from system events! Still, let’s hold our horses and get back to our use case.

The system messages table could mirror the structure of regular messages with:

Global position sequencing (with all the transaction visibility handling from the outbox ordering article)
Transaction ID tracking for proper visibility checks
Archiving support via is_archived flag
Partitioning for performance

If we created a throwaway “skipped events” table, we’d need to solve all these problems again. We’d essentially be building a second event log with the same guarantees.

The skip could be stored as a system message where:

Stream ID = the projection identifier (name + version), so we can query all skips for a specific projection
Message data = reference to the original event (sequence ID from the event log)
Message metadata = processor ID, reason for skip, timestamp
Message type = indicates this is a “skip during rebuild” system event

Won’t this table grow too much? It may, but we can archive skip events when they’re no longer needed.

When the rebuild processor handles a skipped event, we could archive it. In Emmett, this means setting is_archived = true on the record. The table is partitioned by this flag. Archived records automatically move to a separate partition, which can even be on a different disk drive.

Why archive instead of delete?

Audit trail: You can see what was skipped and when it was processed
Debugging: If something goes wrong, you have history to investigate
Idempotency: If a processor crashes and restarts, it won’t reprocess already - archived skips

Later, we could define retention policies and clean up old archived records. We could do it more aggressively than for business events, since these are operational records, not business history. You might keep business events forever, but archived skip records only need to stick around long enough for debugging (days or weeks, not years).

The same processor that handles regular events for the projection could also process its skipped records. When it reads an event from the main event log and applies the projection, it also checks for and archives any corresponding skip record for that event position. This keeps everything in sync.

As mentioned, this also opens the door for other system events in the future - not just rebuild skips, but potentially failed projections, poison messages, or audit records. The infrastructure would already be in place, separated from the business event log.

The Final Flow

Let me walk through how this works in practice, bringing together all the pieces.

During rebuild

The projection status is ‘rebuilding’. The async rebuild processor works through historical events from the beginning. Meanwhile, normal operations continue. Events are appended, and inline projections check the status. When they see ‘rebuilding’, they skip the projection but record a skip message in emt_system_messages within the same transaction.

When rebuild catches up

The rebuild processor eventually reaches the current position where no new events are visible. At this point, it sets the status to ‘active’. From now on, new appends will process their inline projections normally.

But what about those skip records? The events they reference exist, but their projections were never applied.

Draining the skipped events

The rebuild processor (or a dedicated processor, or a manual trigger—your choice) now queries emt_system_messages for skip records belonging to this projection. Using the same transaction visibility rules from the outbox pattern (transaction_id < pg_snapshot_xmin), it only sees skip records from committed transactions.

For each skip record:

Find the referenced event in the event log (or use its data if stored in skipped event)
Apply the projection for that event
Archive the skip record (set is_archived = true)

The skip record and the event are committed in the same transaction. This gives us a simple invariant: if an event exists, either its projection was applied inline (status was ‘active’) or a skip record exists (status was ‘rebuilding’). There’s no third option where an event exists without a trace.

The drain process might find new skip records appearing. Transactions that were in flight during the status change, committed after we started draining. That’s fine. We keep querying until no more visible skip records exist. They will stop appearing as we alread stopped rebuilding processor, so no more inline projections should be skipped.

The “drain skipped events” phase can be automatically triggered by finishing projection rebuild. It could be handled as the 2nd phase of rebuild processor, or triggering a dedicated one. It could be also just handled by a human initiating the drain.

This flexibility lets user choose the approach based on the specific operational requirements.

The Dead Letter Queue Pattern

What we’ve built is essentially a Dead Letter Queue (DLQ)—a place where messages that couldn’t be processed normally are stored for later handling.

This pattern exists in every serious messaging system:

Apache Kafka: Dead Letter Topics for messages that fail consumer processing
RabbitMQ: Dead letter exchanges for rejected or expired messages
AWS SQS: Redrive policies that move messages to a DLQ after N failures
Azure Service Bus: Built-in dead-letter sub-queues for each queue

The pattern is universal because all messaging systems face the same fundamental problem: sometimes messages can’t be processed immediately, and you need a place to store them without blocking the rest of the system.

It’s a topic in its own right, as it’s not perfect and should be used cautiously. Many teams fail to apply them correctly. The DLQ becomes like a car alarm in a parking lot, technically signalling a problem, practically ignored by everyone. And that’s what I see in many systems: teams set up DLQs and do nothing about them.

It starts innocently. You configure the DLQ “just in case.” A few messages end up there during a deployment. Someone says, “We’ll look at it later.”

More messages accumulate. The DLQ becomes background noise—a number on a dashboard that nobody checks. Eventually, something critical lands there, and nobody notices until a customer complains.

That’s why in the discussed design, skip records aren’t meant to accumulate indefinitely. The rebuild processor drains them during completion.

Retention policies clean up archived records after a reasonable period. If skip records exist for too long, that’s a signal that something is wrong with the rebuild process - and users should know about it.

A DLQ is only helpful if it’s monitored, processed, and understood why messages end up there. Otherwise, it’s just a fancy way to lose data slowly rather than immediately.

The Broader Lesson

This article isn’t really about PostgreSQL advisory locks or projection rebuilding. It’s about how we approach problems in distributed systems.

When we find a race condition, the instinct is to fix it. Add a lock. Add a check. Add a coordination phase. But each “fix” often just moves the race condition somewhere else. We went through three attempts—waiting for transactions, using transaction ID boundaries, locking appends—and each one failed for a different reason. We weren’t solving the problem; we were relocating it.

At some point, I had to ask myself: am I making this system more reliable, or just more complicated?

The answer came when I changed the question. Instead of asking “how do I prevent events from being skipped?” I asked, “How do I know when an event was skipped, and how do I make sure it gets processed eventually?”

The first question has no good answer, not without blocking appends, which defeats the purpose. The second question is straightforward: record the skip in the same transaction as the event, and process it later.

A system isn’t trustworthy because it never fails. That’s impossible for anything sufficiently complex. A system is trustworthy because you know when it can fail, how it will fail, and how to recover. The skip tracking approach doesn’t prevent failures during the transition period. It makes them visible and recoverable. That’s a stronger guarantee than complex coordination machinery with hidden edge cases.

Sometimes the best engineering decision is to accept what you can’t control and focus on what you can. We can’t control PostgreSQL’s transaction visibility rules. We can’t see into uncommitted transactions. We can’t make the append decision and the rebuild completion atomic without stopping the world.

What we can control is recording skipped items, ensuring those skips are processed, and making the whole thing observable. That’s enough.

The blind spot I found wasn’t really about the specific race condition. It was a reminder that distributed systems have fundamental constraints we can’t engineer around, at least not without trade-offs worse than the original problem.

Transaction isolation and concurrent systems mean we can’t have zero-downtime rebuilds with perfect inline projection consistency and no coordination overhead, all at the same time.

Something has to give.

What we get instead is explicit tracking of what was skipped, guaranteed eventual processing via the system messages table, observability into the transition period, and crash recovery that doesn’t lose data. The read model might be briefly inconsistent during the transition, but we know exactly what’s missing, and we have a clear path to fix it.

That’s the kind of system I can reason about and trust.

If you’re dealing with similar challenges, I’m happy to help through [email protected]">consulting or mentoring. You can also join the discussion in our Emmett Discord—we have a nice community working through these exact problems.

Or check also other related resources:

Cheers!

Oskar

Rebuilding Event-Driven Read Models in a safe and resilient way

Let’s make a soup today: a blog soup. We’ll mix multiple ingredients like:

events (obviously),
read models,
inline and async projections,
rebuilding read models,
backfilling new ones with data from existing events,
scaling async processing horizontally,
distributed locking,
PostgreSQL and its Advisory Locks.

Sounds a lot? Well, the soup should be nutritious.

In an event-driven way, after handling business logic, we record new facts and call them events. They gather information about what has happened. That brings many benefits, such as business observability by keeping a log of them. Especially if we’re doing Event Sourcing, we can make the next decision based on them.

Typically, we’re using events in two ways:

reacting to them, triggering and integrating the steps of our business workflow,
projecting them, and getting the flattened interpretation of our system state inside read models.

Such processing can happen asynchronously, but doesn’t have to. If we’re using a transactional database (like PostgreSQL, SQLite, or even MongoDB), we can update our read models in the same atomic transaction that stores new events. Such a process is typically called inline projection. Event stores like Emmett and Marten allows that. Still, you can do the same if you’re using outbox pattern, then you can achieve the same without Event Sourcing.

Inline processing is tempting because we’re getting immediate consistency. Yet, there’s no free lunch here. We’re slowing down our event append as we need to process more, and our transactions will be open longer, which can cause deadlocks, etc. We also can’t take advantage of batching event processing.

Also, as Bastian Waidelich rightfully pointed out to me, using inline projections also increases the coupling and fragility of our business logic. If the inline projection fails (due to a bug in its projection logic, a database constraint or other random issue), then we won’t be able to append our event, which is counterintuitive, as why would read model block our business logic (e.g. confirming shopping cart).

My thumb rule is that for single stream, simple projections, I prefer inline projections, but for more complex or workflow processing, I’d go with async.

The big benefit of a durable event log is that we can correct past mistakes and gain more insights from existing data.

How does it look in practice? Let’s say we initially had a basic read model that showed a summary of the specific shopping cart. In TypeScript this could look as follows:

type ShoppingCartSummary = {
  _id?: string;
  productItemsCount: number;
  totalAmount: number;
};

Besides the data, we also need a method that represents how we apply events on top of the existing state to get the next, evolved state.

const evolve = (
  document: ShoppingCartSummary | null,
  event: ProductItemAdded | ProductItemRemoved,
): ShoppingCartSummary => {
  document = document ?? { totalAmount: 0, productItemsCount: 0 };

  switch (type) {
    case 'ProductItemAdded':
      return withAdjustedTotals({
        document,
        productItem: event.data.productItem,
        by: 'adding',
      });
    case 'ProductItemRemoved':
      return withAdjustedTotals({
        document,
        productItem: event.data.productItem,
        by: 'removing',
      });
  }
};

const withAdjustedTotals = (options: {
  document: ShoppingCartSummary;
  productItem: PricedProductItem;
  by: 'adding' | 'removing';
}) => {
  const { document, productItem, by } = options;
  const plusOrMinus = by === 'adding' ? 1 : -1;

  return {
    ...document,
    totalAmount:
      document.totalAmount +
      productItem.unitPrice * productItem.quantity * plusOrMinus,
    productItemsCount:
      document.productItemsCount + productItem.quantity * plusOrMinus,
  };
};

Sounds fine, but well, it may appear that either we, implementing it, or our business, through requirements, forgot about some requirements. Or we didn’t forget anything, but just requirements evolved as they tend to always do.

What if we need to handle now also:

Show the shopping cart status and show whether the shopping cart is open, confirmed or cancelled.
not only show totals, but also a list of product items with their details.

Our new read model would look like:

type ShoppingCartSummary = {
  _id?: string;
  productItemsCount: number;
  totalAmount: number;
  status: 'Opened' | 'Confirmed' | 'Cancelled';
  productItems: ProductItem[];
};

type ProductItem = {
  productId: string;
  name: string;
  quantity: number;
  unitPrice: number;
}

And here we should ask ourselves the following questions:

Is it really the same read model or another one that’ll be used in another place in our UI? Maybe the initial just shows basic data in the menu bar, and this will be used as the summary before confirmation?
If the read model is the same, then are you fine with downtime where you clean the old data, and reprocess events?
If it’s the new read model, do we need to backfill it with the old data?

There’s no best practice here; we need to do the drill and be prepared for multiple options.

Add new vs update existing.

In our case, my initial guess would be that this should be a new read model. We need to add significantly more data and new event handling for confirmation and cancellation.

You could say that:

Why add a new read model with similar data? Can’t we just do a subquery?

Of course, you can. I wrote about that more in How to create projections of events for nested object structures?. This can make sense if those read models will always evolve together.

If you need to run multiple views on the same read models, you’re increasing coupling. As such, when you’re rebuilding the read model, then potential downtime will impact both. Also, each time you adjust one view, ensure you haven’t broken the others.

You’re getting a smaller storage size, and potentially don’t need to remember multiple read models, but are doing it just one.

My experience shows that optimising for the end storage until we check that it’s too big isn’t a good driver. Nowadays, storage is cheap, so my default is to keep read models separated, and also not reuse the same evolve logic. A bit of code duplication won’t harm us, but we’ll see benefits as our models evolve. We’ll decrease the cognitive load.

Still, if those models are indeed the same, or we bet they’ll constantly evolve together, then it could be fine to reuse them.

In place update, blue/green rebuild and backfilling data

Ok, I was already using rebuilding word multiple times. But how do we actually do it?

If you’ve read articles on checkpointing or positions in event stores, you already know that each event in the event store/outbox can have its unique, monotonic position. We can subscribe to notifications about new events and process them one by one. That’s how consumers and processors work in Emmett. Once we process a specific event, we can store the checkpoint in the end storage. This enables resilient failover when our processor dies for some reason. When we restart, we’ll read the last processed checkpoint first and start listening for events from that point.

We could reuse this not only for failover but also for rebuilds. Instead of starting processing from the last known checkpoint, we could reset the checkpoint in our database to a specific point (e.g., the beginning of the log). Then we’ll get recorded events again.

And this is the moment we need to decide whether to do an in-place update or a blue/green rebuild.

In-place means that we’re the same storage target. We need to truncate it and then apply data from scratch, typically. The downside is that when we clear it, our read models won’t be up to date with the current state of our event store. We already have newer events recorded, but our read models are empty. We need to fill them in.

The same happens when we create a new read model based on the existing events. When we create a new read model, it won’t magically contain data from existing events.

Typically, we need to spin up a background worker (e.g., Emmett consumer with a projector plugged in) that pulls all events since the beginning and runs projection logic. Obviously, that means we need to run it asynchronously, and depending on the data size, we’ll need to wait until it catches up. During that time, querying for the data will return outdated information - eventual consistency in practice.

That’s why there’s another option. Instead of truncating existing data, we can keep it as it is, or even keep updating it in the old form. Having that, we can build a new read model in parallel. This read model could have a different name, or just a suffix, e.g. V2, V2.0.1, whatever we find helpful.

Then, once this new read model catches up (so it has processed all events, or is close enough to the latest event with a defined threshold), we can switch queries from the old to the new read model storage.

This is actually the preferred way, but it’s a bit more challenging when it comes to dynamically switching the query target. If you’re using Pongo, that’s not that hard, since you just switch the text-based collection name, which is just another table. But if you’re using an ORM, adding a new table dynamically and mapping it can be much more challenging.

Concurrency issues while (re)building read models

To make it harder, eventual consistency is not the only challenge we’ll face. We also need to deal with concurrency and parallel processing.

What should we do when a new event is appended while we’re rebuilding/backfilling the inline projection? If we try to update the end storage in the middle of the processing, we can end up with an inconsistent or erroneous state.

Another issue is what to do if we’re in a Kubernetes-like setup and don’t have full control over the number of instances of the same service? Or what happens if we accidentally spin up multiple rebuild workers?

Then we’re doomed, or at least the consistency of our read model data.

How do we solve it? Let me explain my plan for Emmett.

Distributed Locking and PostgreSQL advisory locks

I encourage you to check my other article: Distributed Locking: A Practical Guide. Yet, don’t worry, I won’t leave you with Read-The-Fucking-Manual type of answer.

Distributed locks are a fundamental tool for coordinating concurrency across systems. We can use a central place, typically scalable on its own, that’ll be used in multiple instances of our service, to ensure that exactly one can request a lock and run specific code.

In the mentioned article, I described multiple options and popular tools for handling that (e.g., Redis, Zookeeper, Kubernetes Replica Sets), as well as PostgreSQL and its Advisory Locks. Let me focus today on the last one and describe how I want to use it in Emmett’s PostgreSQL projection handling.

PostgreSQL gives us two options for coordination.

Row-level locks lock individual table rows. You do SELECT … FOR UPDATE, and anyone else trying to modify that row waits. The lock is tied to the specific row in a table.

BEGIN;
SELECT * FROM some_table WHERE id = @key FOR UPDATE;
-- make changes
COMMIT;

Such locks are straightforward, as we explicitly state what we want to lock. We could define the following table for projections:

CREATE TABLE IF NOT EXISTS emt_projections(
      version                       INT                    NOT NULL DEFAULT 1,  
      type                          VARCHAR(1)             NOT NULL,
      name                          TEXT                   NOT NULL,
      status                        TEXT                   NOT NULL,  
      PRIMARY KEY (name, version)
);

The background worker could lock the projection by a specific name and version and set the status to “rebuilding”. Then, during handling the inline projection, we could use the same lock and check whether the status is “active”; if not, skip processing. Using a lock-in inline projection would also prevent the rebuilding process from starting, as they wouldn’t acquire the lock.

And that’s a nuke option, as it would work but could create a performance problem. If every inline projection needs to grab a row lock on a coordination row, they will all be processed sequentially, one at a time. That’s a throughput killer when you’re appending thousands of events per second. Each of these append would require access to the lock for the specific projection type (e.g. our shopping cart summary), making not only updates but also event appends sequential. That’s not acceptable for most cases.

Advisory locks are different. They’re locks on arbitrary integers. PostgreSQL doesn’t care what the integer means—it just manages who holds the lock. No table rows involved. It also allows two modes: exclusive or shared.

-- Shared lock: many can hold simultaneously
SELECT pg_try_advisory_xact_lock_shared(12345);

-- Exclusive lock: only one holder, blocks shared locks
SELECT pg_advisory_lock(12345);

Those locks are either scoped to an open connection session or an opened transaction (with xact with name) and released automatically once they end.

Shared locks allow multiple sessions to access the lock with the same value. Exclusive lock blocks both other exclusive locks and new shared locks.

This allows a design where shared locks are used for readers, and exclusive locks for writers and maps directly to our problem:

Inline projections take shared locks as they run concurrently, and just need to check if there’s no async job updating these projections
Rebuilds take exclusive locks - they block inlines and other instances of async processing.

We just need to do one more thing: since advisory locks take integers, we need to map our projection name and version to them. We can do it by a consistent hash algorithm, either in the application code or in PostgreSQL.

PostgreSQL provides a built-in MD5 hash function. It’s not perfect, as it’s not a sophisticated hash, but it’s fast enough and predictable. In our case, we won’t have thousands of projections in our application, so the risk of a hash collision is negligible. If you’re still worried it’s too high, we could store id in our projections table and use it instead of hash-mapping. Still, if we used md5 function, it could look as follows:

// shared for inline projections
SELECT pg_try_advisory_xact_lock_shared(
        ('x' || substr(md5(?), 1, 16))::bit(64)::bigint
) AS acquired;

// exclusive for inline projections
SELECT pg_try_advisory_xact_lock(
        ('x' || substr(md5(?), 1, 16))::bit(64)::bigint
) AS acquired;

Where as query param, we’d pass the joined projection name and its version.

Thanks to that, multiple inline projections can access the lock if it’s not held exclusively by the async (re)building worker. Thanks to that, we’re not blocking event appends because of the lock on the inline projections.

An exclusive lock can be held only when there’s no single inline projection being applied at the moment.

What’s also cool is that advisory locks can be used with two strategies: fail fast when the lock is held, or wait for the lock to be released. The first option would be useful for inline projections, and the second for rebuilds.

Is that all? Well, not quite.

Why advisory locks alone aren’t enough

Advisory locks have a gap: they’re session-scoped. If the connection holding the lock dies, the lock releases automatically. For most cases, that’s fine and expected, but think through this scenario:

We have 1000 events in our event store.
Rebuild starts, acquires exclusive lock.
It truncates the projection and processes events 1-500.
Connection dies (network blip, out of memory kill, whatever).
Lock releases automatically.
Projection contains data only for events 1-500.
Inlines see no lock, start processing.
Inline applies event 1001, ending up with a potentially corrupted state (as there may be some events between 500 and 1001 that would impact the state).

Advisory locks can’t persist across connection failures. We need to add something that does.

The hybrid-locking approach

Emmett already maintains tables for tracking processors and projections. The relevant ones:

CREATE TABLE IF NOT EXISTS emt_projections(
    version         INT         NOT NULL DEFAULT 1,  
    type            VARCHAR(1)  NOT NULL,
    name            TEXT        NOT NULL,
    partition       TEXT        NOT NULL DEFAULT 'emt:default',
    kind            TEXT        NOT NULL, 
    status          TEXT        NOT NULL, 
    definition      JSONB       NOT NULL DEFAULT '{}'::jsonb, 
    PRIMARY KEY (name, partition, version)
) PARTITION BY LIST (partition);

CREATE TABLE IF NOT EXISTS emt_processors(
    last_processed_transaction_id XID8    NOT NULL,
    version                       INT     NOT NULL DEFAULT 1,
    processor_id                  TEXT    NOT NULL,
    partition                     TEXT    NOT NULL DEFAULT 'emt:default',
    status                        TEXT    NOT NULL DEFAULT 'stopped', 
    last_processed_checkpoint     TEXT    NOT NULL,    
    processor_instance_id         TEXT    DEFAULT 'emt:unknown',
    PRIMARY KEY (processor_id, partition, version)
) PARTITION BY LIST (partition);

The status column in the projections table can do what advisory locks can’t: persist status between connection crashes. Even if the connection dies, status = ‘rebuilding’ stays in the table. Inlines could check this and skip processing.

The processor’s table tracks checkpoint progress. When a rebuild runs, it updates last_processed_checkpoint as it goes. If it crashes and restarts, it can resume from where it left off rather than starting over.

Potentially, we could reuse the processor’s table, but having a dedicated projection table could also be useful for diagnostics (e.g., storing the projection definition) and for switching queries when a new projection version catches up. We could select the highest active projection version.

Having that, the query checking if the inline projection should be processed or skipped could look as follows:

WITH lock_check AS (
    SELECT pg_try_advisory_xact_lock_shared(
        ('x' || substr(md5($1), 1, 16))::bit(64)::bigint
    ) AS acquired
),
status_check AS (
    SELECT status = 'active' AS is_active
    FROM emt_projections
    WHERE partition = $2 AND name = $3 AND version = $4
)
SELECT
    COALESCE((SELECT acquired FROM lock_check), false) AS acquired,
    COALESCE((SELECT is_active FROM status_check), true) AS is_active;

Two checks, both must pass:

Can we get a shared advisory lock? Fails if a rebuild holds exclusive.
Is the projection status ‘active’? Fails if it’s ‘rebuilding’.

Then for async (re)building worker:

WITH lock_check AS (
    SELECT pg_try_advisory_lock(
        ('x' || substr(md5($1), 1, 16))::bit(64)::bigint
    ) AS acquired
),
ownership_check AS (
    INSERT INTO emt_processors (
        processor_id,
        partition,
        version,
        processor_instance_id,
        status,
        last_processed_checkpoint,
        last_processed_transaction_id
    )
    VALUES ($2, $3, $4, $5, 'running', '0', '0'::xid8)
    ON CONFLICT (processor_id, partition, version) DO UPDATE
    SET processor_instance_id = $5,
        status = 'running'
    WHERE emt_processors.processor_instance_id = $5 -- We already own it
       OR emt_processors.processor_instance_id = 'emt:unknown' -- Unclaimed
       OR emt_processors.status = 'stopped'  -- Previous instance finished or crashed
    RETURNING last_processed_checkpoint
)
SELECT
    COALESCE((SELECT acquired FROM lock_check), false) AS acquired,
    (SELECT last_processed_checkpoint FROM ownership_check) AS checkpoint;

Together Advisory locks prevent the race at the transition point where the connection was closed, and the rebuilding job is restarting. The status check handles crash recovery

The rebuild acquires the exclusive lock before updating the status. It waits for in-flight inlines (which hold shared locks) to finish.

If the checkpoint is null, the UPDATE matched no rows, then another instance owns the processor and is actively running. Back off.

If both succeed, you own the lock and the processor. The checkpoint tells you where to resume from (with a fallback to the beginning). Only then does it flip the status and start async processing.

If the rebuild crashes, the lock is released, but the status remains ‘rebuilding’. Inlines check status and skip.

The scenarios walkthrough

As a single good image speaks more than thousands of words, let me give you some diagrams to summarise how it works.

1. During inline projection while event appends:

Each inline projection grabs a shared lock before applying,
Multiple inlines can run concurrently (shared locks are compatible),
Projection updates happen normally.

2. When a rebuild starts:

Rebuild grabs an exclusive lock (waits for any in-flight inlines to finish),
Marks projection as “rebuilding”,
New inlines see the lock or status and skip,
Rebuild processes all historical events,
Marks the projection as “active” and releases the lock,
Inlines resume.

3. If rebuild crashes:

Lock releases automatically
Status stays “rebuilding”
Inlines keep skipping until another rebuild completes

4. Multiple async processors:

Each processor tries to acquire the exclusive lock,
First one wins, others wait or skip,
Guarantees that only one processor handles a projection at a time.

5. When you add a new projection:

No status row exists yet,
Inlines skip automatically,
First rebuild creates the row and backfills data.

TLDR

Just like making delicious soup, designing robust, fault-tolerant and performant distributed systems is not that easy. Building an event store is not that hard. But only if we exclude the async processing part.

I hope this walkthrough covers both the conceptual and practical aspects of handling projection rebuilds.

We used PostgreSQL and Advisory Locks, as PostgreSQL is cool and is a driving force in Emmett. But all the same principles apply to other tools and storage (with their specifics).

I explained why advisory locks and status columns complement each other:

Advisory locks handle the fast path (in-memory, no disk I/O for normal operations) and prevent races at transition points (rebuild can’t start until in-flight inlines finish)
Status column handles crash recovery (persists across connection failures) and new projection bootstrapping (inlines skip until first rebuild completes)

Neither alone is sufficient. Together, they provide the guarantees we need without external infrastructure. Just PostgreSQL doing what PostgreSQL does.

The cost on the hot path is microseconds: one in-memory lock check, one indexed read on a tiny cached table. For most systems, that’s an acceptable tradeoff.

I hope this article also shows you how to use distributed locking in practice.

Please tell me your thoughts and concerns, especially if you see any blind spots in this design! You can do that in our Emmett Discord, come on in, we have a nice community!

If you’re dealing with such issues, I’m happy to help you through consulting or mentoring. [email protected]">Contact me and we’ll find a way to unblock you!

Or check also other related resources:

Cheers!

Oskar

Multi-tenancy and dynamic messaging workload distribution

There are several reasons why I’m blogging.

The first one is that I forget, and writing helps me to remember and organise my findings.

The other is that I like to share my journey with the intention of sparing you, my dear reader, some of my struggles.

Last but not least, it gives me the chance to learn from discussions inspired by them. It’s always a chance to meet new perspectives, correct what I did or just trigger some recollection.

After the article about consumer, processors and all that messaging jazz, there was a great discussion on Emmett Discord, there were a lot of interesting threads there, but the one that inspired what you read today came from the Ismael Celis question. Ismael was curious about:

Distributing the workload dynamically between processors

And as that’s something I haven’t planned to deliver initially, but it’s a great question, let me put down some notes on my plan around it.

Why would we want to distribute the workload dynamically? What does that even mean?

Let’s start from the classical approach. We can define projections that will build read models based on the upcoming events. In Emmett, it can look like that when using Pongo as the storage tool:

const cartsSummaryProjection = pongoSingleStreamProjection({
  collectionName: shoppingCartsSummaryCollectionName,
  getDocumentId: (event) => event.data.shoppingCartId,
  evolve,
  canHandle: ['ProductItemAdded', 'ShoppingCartConfirmed'],
  initialState: () => ({
    status: 'pending',
    productItemsCount: 0,
 }),
})

This means that when the projection handles only ProductItemAdded and ShoppingCartConfirmed event types. It’ll insert or update rows in the table based on the shopping cart id from the event data.

We can plug this projection into a projector (the specific type of processor responsible for running projections).

const cartsSummaryProjector = postgreSQLProjector({ 
  processorId: 'shoppingCartSummary',
  projection: cartsSummaryProjection,
 });

And plug it into the consumer:

const consumer = postgreSQLEventStoreConsumer({
  connectionString,
  processors: [
    cartsSummaryProjector
  ],
});

Now, consumers will poll the PostgreSQL event store and pipe filtered events by type into our projector.

That’s simple, we know the filtering criteria, as consumers will know which event types their processor(s) can handle.

Things get harder when we don’t know those criteria upfront. When can that happen? What if we were building an e-commerce SaaS product that lets shop owners buy a subscription to run their shops? Then we won’t know all of our tenants upfront. They will register as they go.

What options do we have for such a multi-tenant setup?

The simplest option would be to add a custom filter that allows filtering events on the consumer by the tenant.

Emmett doesn’t support such a feature yet, but it could (and will at some point). We could extend our event metadata to contain the tenant and spin up a consumer for a dedicated tenant.

We could even wrap it into a dedicated function.

function tenantedConsumer(
  connectionString: string,
  tenant: string,
  processors: MessageProcessor[]) {
  return postgreSQLEventStoreConsumer({
    connectionString,
    processors,
    filterBy: (event) => event.metadata.tenant === tenant,
    projection,
  });
}

Of course, this wouldn’t be fully dynamic, but we could make it dynamic at the infrastructure level, e.g., by passing the tenant ID from an environment variable and spinning up a new Container (e.g., a Kubernetes Pod).

const consumer = tenantedConsumer(
  process.env.PG_CONNECTION_STRING,
  process.env.TENANT_ID,
  [ cartsSummaryProjector ],
);

We could also start a new job from the API endpoint, running it in the existing deployment; there are plenty of options. You can go wild and think about other ways.

Still, logically, they would look more or less like that. Either you’re scaling horizontally and sharding physically, tenants

Or you’re scaling vertically and running multiple tenants inside your box. Then, instead of spinning up new containers, you’re spinning up new jobs (processes, threads, virtual threads, etc.).

That’s nothing special for Emmett; other tools also do this, e.g., Spring Boot Kafka concurrent listeners.

And hey, btw. there’s one more reason why I’m writing about my Emmett design: to let you benchmark your design against it. Even if you’re not planning to use it, then those are considerations for you either as:

internal tooling creator,
user of other OSS tooling, to check how they solved it.

Separating tenants through sharding give you:

better option to scale and align the needs to the specific tenant workload,
makes possible full separation in terms of networking, storage, etc.
can be more costly.

Separating tenants through partitioning the load inside one box:

is usually cheaper,
easier to manage,
doesn’t give you a full separation and can fall into Noisy Neighbour issue.

So there’s no golden rule to choose, it depends on your tooling, needs, etc.

You can, of course, have a mixed solution, so running most of the small tenants in the same box, and the one with bigger security needs gets a special setup. Especially that dynamic split doesn’t have to be only per tenant. You can, e.g., have per region, per product range, per chain, etc.

Also, such a dedicated setup can work for a set number of dynamic options. If you have 1000 tenants, would you spin up 1000 containers or jobs? You can, but just because you can doesn’t mean that you should. The more containers or threads you have, the more you pay for the coordination costs. At some point, it’s better to group processing into a manageable range of containers or threads.

Ok, but how do we know which messages to process where? We could use a consistent hash. I wrote about it in detail in Understanding Kafka’s Consumer Protocol: A Deep Dive into How Consumers Talk to Brokers.

Kafka, by default, partitions its data on the producer side. The topic represents a logical split (e.g. all messages from the E-Commerce module), and the partition represents physical layout. Consumer groups receive messages. Kafka guarantees that precisely one consumer from the consumer group will receive messages from the specific partition. If we have fewer consumers than partitions, then of course, we can get more partitions to handle.

The pseudo code for distributing load to consumers could look as follows:

const partitionId = message.headers.partitionId;
const hash = consistentHashFunction(partitionId);
const consumerId = hash % totalNumberOfConsumersWithinGroup;

In our case, we could use the tenant id as the partition id.

Kafka’s strategy is also simpler than we might need. The partitioning is done on the producer side. The producer sends a message to the topic that already has a certain number of partitions:

const hash =  consistentHashFunction(message.header.recordId);
const partition = hash % totalNumberOfPartitionsWithinTopic;

In our case, if we’d like to allow distribution by any property, we’d need to load the message, read the field we want to partition/shard on, and send it to the appropriate container or thread.

The mechanism can get pretty hefty. You’d need a more sophisticated mechanism to know where to spin up what, how to distribute the load between containers or threads, and to make it resilient. We’re getting into the area of distributed consensus algorithms such as Raft and Paxos.

Do I want to go into this area with Emmett? Definitely not for free! I don’t think that’d be even worth it, as we have mature solutions like Kafka, RabbitMQ, and other messaging systems that implement such algorithms and specialise in that. I’d prefer to make it easier to forward messages to them and let them do the work.

What is definitely on the plate is the second option, which allows partitioned producers. You could define it as such:

const cartsSummaryProjector = postgreSQLProjector({ 
  processorId: 'shoppingCartSummary',
  projection: cartsSummaryProjection,
  partitionBy: (event) => event.metadata.tenant,
 });

Then the consumer would poll messages from all tenants, forward them to projectors, and the projectors would internally spin up worker threads per tenant or group them by consistent hashing.

If we add to that distributed locking or detecting conflicting checkpointing to ensure that there’s only one worker instance handling messages for the processor and partition id, then this should be good enough for the majority of cases.

Keeping in mind the flexibility in which you can group (or not) projectors within consumers, you can define your own topologies.

What are your thoughts? How do you deal with such cases? Would you like me to expand more on some cases?

Or maybe you’d like to help me and sponsor my work in this area in Emmett? Then your project could also benefit faster from it!

Cheers!

Oskar

Checkpointing the message processing

Let’s start by asking you two questions.

What Super Frog has to do with messaging?
When was the last time you wrote if statements in SQL? If it’s been a long time, have you at least seen them? If not, (don’t) worry, you’ll see them today.

Will it be a post about weird SQL usage? Not necessarily.

We’ll talk today about checkpointing our processing.

I’ve started my relationship with computers with games. I still have my Amiga 500. In those days, computers didn’t always have a hard disk. You’ve got a bunch of diskettes with different chapters of the game. Not all of them were simple games; many were quite sophisticated, and it took some time to finish them.

Yet they were dealing with limited diskette space, so it was best if they didn’t have to use any of it. How can you then allow you to stop playing and return to the previous state? Or how to not force you to start from the beginning of the game when you fell from the platform, and well, you died? You died in the game, ofc, that at least you should be able to recover, right?

As mentioned earlier, the limited space on diskettes and the additional complexity that came with it led many game makers to adopt a simple solution: checkpoints with codes.

After you passed a level, you got a code you could type when you started the game, and instead of starting from the beginning, you could go directly to the place where you left off. That worked pretty well for the platform and race car games, since your game’s storyline was always the same, immutable. If you had to go to level 27, the starting point and your character would always look the same. Of course, for RPG and strategy games, that’s a different story.

Surprisingly, this parallel also matches the recovery from a business process failure.

Let’s say we’re using message-based communication to streamline and make it more resilient. We don’t want to make it vulnerable to scenarios where we store information in one system, our process dies, and we don’t manage to notify the other parts.

We’re using Outbox pattern to enable that technically. We’re storing messages in the relational table within the same transaction, updating the state after running business logic. Thanks to that, either both states are updated, and the message is scheduled, or none of it is. We’re getting (eventual) consistency thanks to that.

Now we’re on the receiving end, so where we were in the previous article with the explanation of Consumers, Processors and all that jazz.

Let’s say that we’re using PostgreSQL and our Outbox structure looks as explained in the other article:

CREATE TABLE outbox(
   -- the autoincremented position of the message to respect the order
   position        BIGINT                    GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
   -- used to detect gaps in numbering
   transaction_id  xid8                      NOT NULL,
   -- unique message id, which can be used for deduplication or idempotency
   message_id      TEXT                      NOT NULL,
   -- the message type, e.g. `TransactionRecorded`
   message_type    TEXT                      NOT NULL,
   -- serialised message data, e.g. to JSON
   data            JSONB                     NOT NULL,
   -- diagnostic information on when the message was scheduled
   scheduled       TIMESTAMP WITH TIME ZONE  NOT NULL    default (now())
);

As you can see, besides the message ID, type, and data, we’re also storing the (global) position number and transaction ID (to ensure we don’t skip in-flight transactions that have requested the global position number, read more here for reasoning).

Now we can be polling it with the query like:

SELECT 
     position, message_id, message_type, data
FROM
     outbox
WHERE
     (
          (transaction_id = last_processed_transaction_id
               AND position > last_processed_position)
          OR
          (transaction_id > last_processed_transaction_id)
     )
     AND transaction_id < pg_snapshot_xmin(pg_current_snapshot())
ORDER BY
    transaction_id ASC,
    position ASC
LIMIT 100;

Now, thanks to that, we can have the global ordering guarantee on the receiving end. We’re trading a bit of performance for greater correctness. Not always acceptable, but for internal module communication or forwarding to the messaging system, that’s usually more than enough.

That’s also the place where we’re back to our checkpointing. How do we know the last processed position?

By default, that’s simple, we could either say:

start from the beginning and use -1 or some other hardcoded position,
start from the end and use the position of the last message in the table plus one.

Those cases can be fine when we need to handle all messages (e.g., adding a new process or reading a model), or when we don’t care about the past and need to process the newest notifications.

Still, the reality is just like in the old games, we’d like to start somewhere in the middle, precisely where we left off.

We need one more table for storing our checkpoints. It can look as follows:

CREATE TABLE processor_checkpoints
(
   -- subscription name
   processor_id                     TEXT      PRIMARY KEY,
   -- information about the position of the last processed message
   last_processed_position          INTEGER   NULL,
   -- used to detect gaps in numbering
   last_processed_transaction_id    xid8      NOT NULL
);

Simple stuff: the processor ID should be unique and reflect the processor’s logical name and the last processed position(s).

If that looks simple, then let me follow up with the next set of potentially simple questions:

How to store it?
When to store it?

If your answer is: “just do upsert statement”, then you’re kinda right, but that wouldn’t be simple, it’d be over-simplification. At least if you’d like to run it in production.

Let’s start with how. And, for that, let me bring you now the promised stored procedure with if statements, it’s a bit simplified version from Emmett:

CREATE OR REPLACE FUNCTION store_processor_checkpoint(
  p_processor_id      TEXT,
  p_position          BIGINT,
  p_expected_position BIGINT,
  p_transaction_id    xid8
) RETURNS INT AS $$
DECLARE
  current_position BIGINT;
BEGIN
  -- Handle the case when p_check_position is provided
  IF p_expected_position IS NOT NULL THEN
      -- Try to update if the position matches p_check_position
      UPDATE processor_checkpoints
      SET 
        "last_processed_position" = p_position, 
        "last_processed_transaction_id" = p_transaction_id
      WHERE "processor_id" = p_processor_id AND "last_processed_position" = p_check_position;

      IF FOUND THEN
          RETURN 1;  -- Successfully updated
      END IF;

      -- Retrieve the current position
      SELECT "last_processed_position" INTO current_position
      FROM processor_checkpoints
      WHERE "processor_id" = p_processor_id;

      -- Return appropriate codes based on current position
      IF current_position = p_position THEN
          RETURN 0;  -- Idempotent check: position already set
      ELSIF current_position > p_expected_position THEN
          RETURN 2;  -- Failure: current position is greater
      ELSE
          RETURN 3;  -- Default failure case for mismatched positions
      END IF;
  END IF;

  -- Handle the case when p_check_position is NULL: Insert if not exists
  BEGIN
      INSERT INTO processor_checkpoints ("processor_id", "last_processed_position", "last_processed_transaction_id")
      VALUES (p_processor_id, p_position, p_transaction_id);
      RETURN 1;  -- Successfully inserted
  EXCEPTION WHEN unique_violation THEN
      -- If insertion failed, it means the row already exists
      SELECT "last_processed_position" INTO current_position
      FROM processor_checkpoints
      WHERE "processor_id" = p_processor_id;

      IF current_position = p_position THEN
          RETURN 0;  -- Idempotent check: position already set
      ELSIF current_position > p_expected_position THEN
          RETURN 2;  -- Insertion failed, row already exists with a greater position
      ELSE
          RETURN 3;  -- Default failure case for mismatched positions
      END IF;
  END;
END;
$$ LANGUAGE plpgsql;

Oooh, even by copy and pasting, I’m already tired; there’s a fair reason why we’re not doing that too often nowadays.

Let me untangle that for you:

We’re trying to update the existing position or insert it if we’re storing it for the first time.
If all went fine, we’re returning 1 as a result to denote complete success.
If we saw that the checkpoint in the database had the same value, we’re returning 0.
If we saw that the checkpoint is different from what was expected and further away from it, then we’re returning 2.
Otherwise, we’re returning 3, which means that the checkpoint is different from the expected and older.

Essentially, by passing the expected position, we can detect whether we:

already handled the specific position,
have some competing instance of our processor handling our data.

That’s why we’re doing this fancy dance with IF statements and a stored procedure.

Detection assumes that we have a global ordering processing guarantee (thus, tricky bits with transaction ID).

It also shows why global ordering is useful.

By detecting that we’ve already handled a specific position, we can skip processing handling idempotency on the processor level.

By detecting that there’s another processor with the same id processing messages, we can make it more resilient and detect the noisy neighbour issue.

How would that look in the code?

async function handleBatch(messageBatch: RecordedMessage[], context: ProcessorContext): Promise<BatchHandlingResult> {
  const { checkpoint } = messageBatch[messageBatch.length - 1].metadata;

  return context.pool.withTransaction(async (transaction) => {
    for (const message of messageBatch) {
      await context.onMessage(message);      
    };

    // No error was thrown: proceed to store checkpoint of the last processed message
    const result = await storeProcessorCheckpoint(transaction.execute, {
      processorId: context.processorId,
      newCheckpoint: checkpoint,
      lastProcessedCheckpoint: context.lastProcessedCheckpoint,
    });

    if(result.success) {
      await transaction.commit();
    } else {
      // no need to do here, either we already handled it
      // or we have a mismatch of expected and existing checkpoints
      await transaction.rollback();
    }

    return result;
  });
}

type ProcessorContext = {
  processorId: string;
  lastProcessedCheckpoint: bigint | null;
  onMessage: (message: AnyMessage) => Promise<void>;
  pool: ConnectionPool;
}

type BatchHandlingResult =
  | {
      success: true;
      newCheckpoint: bigint | null;
 }
  | { success: false; reason: 'IGNORED' | 'FURTHER' | 'OLDER' };

async function storeProcessorCheckpoint(
  execute: SQLExecutor,
  options: {
    processorId: string;
    newCheckpoint: bigint | null
    lastProcessedCheckpoint: bigint | null;
    partition?: string;
 },
): Promise<BatchHandlingResult> {
  const { result } = await single(
      execute.command<{ result: 0 | 1 | 2 | 3}>(
        SQL`SELECT store_processor_checkpoint(
            ${options.processorId}, 
            ${options.newCheckpoint}, 
            ${options.lastProcessedCheckpoint}, 
            pg_current_xact_id()
        ) as result;`,
      ),
    );

    return result === 1
      ? { success: true, newCheckpoint: options.newCheckpoint! }
      : { success: false, reason: result === 0 ? 'IGNORED' : result === 2 ? 'FURTHER': 'OLDER' };
};

As you can see, thanks to:

global ordering,
checkpoint detection,
storing checkpoint where our side effects will be stored,
transactional capabilities of our end storage,

We can ensure that the entire batch is processed or not. We could even optimise it by storing the checkpoint first and not processing the business logic if there’s a mismatch, then committing only if the logic succeeds.

We’re getting by that generic idempotence check and detection of the noisy neighbour.

Of course, I still believe that idempotence check should happen on the business logic side. But why not both?

Being able to detect a noisy neighbour can help you automatically stop (or pause) one of the competing consumers and avoid inconsistency conflicts.

What are the tradeoffs of this approach?

This will work if we have a global ordering guarantee. Not many messaging solution gives us such. If we have a subscription-based outbox as explained, event store like Emmett, Marten or KurrentDB, Kafka, this will work, but not necessarily for solutions like RabbitMQ, SQS, Google Pub Sub, etc.
This works best if you have transaction capabilities. Batching updates generally improves performance, but sometimes can lead to long-lived transactions; beware of that. The subscription-based solution with a transaction ID also works best if your transactions are short. If they’re open for a long time, it can cause delays.
Still, even without transactions, it will work fine, as long as you’re fine with having retries more often. If the business logic fails and the checkpoint is not committed, it’ll reprocess the already-handled messages from the previously stored messages. Which means that they can be handled more than once, but you should not lose any messages. You may also not fully benefit from the idempotence check for skipping already handled messages.

As always, the choice is yours.

Still, I hope that this article will show you why:

having a global ordering guarantee can be useful,
why and how to checkpoint your processing, how they relate to level codes from old games like Super Frog,
What are the tradeoffs, and how to consider them,
…and that SQL IF statements are sometimes justified. But don’t go wild with them!

And hey, I also hope that’s not something that you’d like to maintain on your own. There are mature tools to deal with such stuff, like Emmett, which implements this for you.

What are your thoughts? Questions? Concerns?

Cheers!

Oskar

Consumers, projectors, reactors and all that messaging jazz in Emmett

Did you know that you can build an event store in one hour? I even did it a few times on the conference stage. Actually, it took me usually around 25 minutes; the rest was mistyping, lame jokes and a bit of explanation. See:

Yet, my final thought was: Kids don’t do it at home.

It’s a fun coding exercise, but using the outcome in production? Not as much fun running and maintaining it. Why though? How hard can it be?

Event Sourcing systems have two phases: appending events and processing them afterwards. The write side gets most of the attention in tutorials and talks - commands, deciders, event stores, optimistic concurrency, as you saw, I’m also one to blame.

Providing the guarantees on the write side is relatively simple, especially if you use a database like PostgreSQL as a storage. You need to provide features like:

appending an event at the end of the stream,
reading all events from the stream,
a guarantee of the ordering within the stream,
being able to read your writes,
strong-consistent, atomic writes and optimistic concurrency.

That can be solved with knowledge about transactions, database design, etc. So again, why so hard?

The processing side is where systems often struggle as they grow. This is where the Event Sourcing solution becomes an Event-Driven Messaging tool. And if you’ve read my previous articles, you know that this can be tricky at times.

How do you reliably process events to build read models? How do you trigger side effects without losing messages? How do you scale processing independently from writes? How do you make it performant and run multiple handlers in parallel?

I’ve been working on the message processing architecture in Emmett for a while now. I’ll try to explain how I designed the split between Consumers and Processors, the problems it solves, and the tradeoffs involved.

Why Split Consumers and Processors?

When processing messages, we already know that someone produced them. We’re on the receiving end. Facts are already known; now we need to do something about them.

When we process them, do we care about the source? Typically, we take the information it gathers and reason about it. For instance, when we received an event indicating that a room reservation was made, we may need to send an email with details to the consumer, update the reservations dashboard, and generate a pro forma invoice. We may have specific logic, depending whether it came from our internal reservation platform or Booking.com, but we know the source from the message payload.

That seems obvious, but it was an important realisation for me. When we’re building a read model in MongoDB, we don’t care if events come from PostgreSQL event store, EventStoreDB, RabbitMQ queue or Kafka topic.

It needs events and the projection logic. Of course, it needs to know the guarantees around: delivery, ordering, idempotency, etc., but besides that? The message’s source doesn’t matter to its logic.

Similarly, a component polling PostgreSQL for messages to publish them doesn’t care what happens to those events - whether they update read models or trigger webhooks is irrelevant to polling logic.

These concerns are orthogonal.

I realised that much of the complexity comes from coupling those two together. We wouldn’t like to change our processing logic because of an internal change in how they’re produced, or vice versa. I concluded that separating them means each can evolve independently. And I came with the initial idea for the split: Consumers and Message Processors.

Consumers are responsible for getting messages from a source and forwarding them to processors. Think of them as the “delivery mechanism.” They handle the “where do events come from” concern. A consumer might connect to:

A PostgreSQL event store, polling the events table,
EventStoreDB, using push-based catch-up subscriptions,
Kafka, consuming from topics,
Any other message source you might have.

Processors are responsible for doing something meaningful with those messages. They handle the “what do we do with messages” concern. A processor might:

Update a read model in PostgreSQL or MongoDB,
Call an external API when certain events occur,
Publish events to Kafka or fire webhooks,
Trigger workflow steps or saga operations.

This separation follows Unix philosophy: small, focused components connected by simple interfaces. Each piece does one thing well. You can plug any processor into any consumer. This gives you flexibility that matters in practice:

You can run the same projection logic against different event sources
Adding new processors doesn’t require changing consumer code
You can test processors in isolation with fake event streams
Consumers and processors can be scaled independently

Let me show you how this looks in practice.

How Consumers Work

I deliberately decided to keep Consumers as dumb as possible. A consumer’s entire job is:

Connect to a message source,
Poll messages in batches or subscribe to notifications (depending on the source specifics).
Forward them to all registered processors.
Go back to step 2.

That’s it. No business logic. No complex state management. No decision-making about what to do with messages. Consumers are essentially routers.

Why this simplicity? I’m a simple guy; I like clear boundaries for responsibility. They help me reason about both how to use the tool and how to handle it when things go wrong. When message delivery breaks, you want to know exactly where to look. With a simple consumer, the question is binary: did it deliver messages or didn’t it? There’s no complex interaction between delivery logic and processing logic to untangle at 3 AM.

For EventStoreDB, the consumer creates a single subscription and fans out messages to all registered processors. For PostgreSQL, it polls the message table in batches, handling ordering guarantees (we’ll get to why that later).

Here’s what a basic consumer setup looks like for PostgreSQL :

const consumer = postgreSQLEventStoreConsumer({
  connectionString,
  processors: [
    shoppingCartDetailsProjector,
    customerAnalyticsProjector,
    orderNotificationReactor
  ]
});

await consumer.start();

Set up looks accordingly for the other sources providing options specific for the source, e.g. for EventStoreDB store, you may want to provide the category stream name:

const consumer = eventStoreDBEventStoreConsumer({
  connectionString,
  from: { stream: '$ce-roomRservations', options: { resolveLinkTos: true } },
});

The consumer receives messages, batches them and forwards them to all processors. Each processor handles messages independently - they don’t know about each other, and they don’t need to.

Why Batching Belongs to Consumers

Who decides how many messages to fetch at once? That’s a tricky question. I think that batching can happen both at the consumer and processor levels. The consumer decides the batch size for the polling or receiving to tune the receiving throughput. The processor can either align with it, using those batches as a safe default or diverge to its specifics. Read more on Why you should batch message processing in my other article.

Different message sources have different optimal batch sizes. PostgreSQL might be efficient with 100-row fetches. EventStoreDB subscriptions don’t have built-in batching; they deliver events as they arrive. Kafka has its own batching semantics. These are all source-specific optimisations we should be able to apply without ending up with the lowest common denominator.

Processors, by default, can just receive batches and process them. Then they can decide whether to split batches into smaller chunks, group them into even larger chunks, or process them as single messages. For instance, PostgreSQL can handle random single updates pretty well, whereas Elastic prefers batching updates.

How Processors Work

If consumers are simple routers, processors are where the interesting work happens. They’re the smarter ones in this relationship. A processor is responsible for:

Processing logic: Actually doing something useful with events. Updating a read model, sending an email, calling an API.
Checkpointing: Tracking which messages have been processed. This is crucial - without it, you’d reprocess everything from the beginning every time you restart.
Error handling: Deciding what to do when processing fails. Retry? Skip? Stop everything?
Idempotency: Doing their best to ensure that reprocessing the same event doesn’t cause problems. Of course, still assuming that handlers should be idempotent, read more in my other article,
Backpressure: They need to be able to tell consumers that they cannot process more messages at the moment, and that the consumer needs to slow down delivery.

Those are general promises and common stuff for the message processing logic. Still, there are multiple reasons why you want to process incoming messages:

Read models (projections) transform events into queryable state. For instance, shopping cart events - ProductItemAdded, ProductItemRemoved, ShoppingCartConfirmed - need to become a document showing current items, quantities, and totals. Something your API can quickly return when a user opens their cart.
Reactions trigger side effects after a business fact has happened. When a shopping cart is confirmed, you may want to send a confirmation email, notify the shipment module, and register a new order. These things need to happen, but they’re not part of the core business logic.
Workflows coordinate multi-step processes across multiple streams. An order might involve payment processing, inventory reservation, and shipping coordination - each with its own state and events.
Integration means forwarding events to other systems. Other services in your systems might need to know about orders. External partners might need webhook notifications. You might publish to messaging systems for downstream consumers.

All of those processing needs a bit different ways to handle reliability, ordering, throughput, etc. Also, all tools we integrate with require a different approach: storing the read model in PostgreSQL will be quite different from forwarding a message to Kafka.

I wouldn’t like to handwave all of those specifics and end up with the lowest common denominator. That’s why I decided to group them into the following archetypes:

projectors,
reactors,
workflows,
allow custom message processors to allow people to tune it fully to their needs,
and in the future, stuff like forwarders, web hooks and others we find useful.

All of them should have a unified API that allows them to be plugged into different consumers, while also embracing differences in message processing and target API specifics.

That’s also why each message processing target (PostgreSQL, EventStoreDB, MongoDB, InMemory, Kafka, SQS, etc.) will have its own implementations.

I believe that this focused responsibility, different archetypes, and specific implementations for different tools will strike the right balance between reusability and avoiding the lowest common denominator. We’ll see if that’s not a famous last words.

Read also more in:

The example projector can look like that:

const projection = pongoSingleStreamProjection({
  collectionName: shoppingCartsSummaryCollectionName,
  evolve,
  canHandle: ['ProductItemAdded', 'ShoppingCartConfirmed'],
  initialState: () => ({
    status: 'pending',
    productItemsCount: 0,
 }),
});


const postgreSQLProjector = postgreSQLProjector({ projection });

const reactor = postgreSQLReactor({
    processorId: 'order-notifications',
    canHandle: ['ShoppingCartConfirmed'],
    eachMessage: (event) => 
      emailService.sendOrderConfirmation(event.data.customerId);    
  }

Native implementations of processors

Different storage requirements require different capabilities, and getting proper guarantees might involve deeper knowledge. For instance, Postgres sequences issues can impact your messaging guarantees. Those are cases where, when you’re starting, you might not anticipate. Test environments may not even catch it; you might realise you’re losing business data when you reach production. That’s why it’s, imho, better to have a tool that solves it rather than trying to maintain it on your own, making technical infrastructure something you need to keep working on instead of your business features. How does Emmett solve them? Let’s discuss them briefly. I’ll try to expand in the future posts about the details.

Resilience

What happens if the processor fails? By default, it stops processing. But only this one, the consumer keeps pushing events to the other processors that can continue. Consumer stops when all their processors are inactive.

Why? Consider this scenario: you have two processors, one updating MongoDB and another updating PostgreSQL. MongoDB becomes temporarily unavailable. Should that stop PostgreSQL updates?

Still, failure behaviour is configurable; your message handler can return:

void/ACK: message processed successfully, continue to the next one.
SKIP: Skip this message, useful for poison messages that consistently fail
STOP: Stop this processor entirely.

Why have skip separate from ACK? Consider a poison message - a message that causes your processor to fail every time. Without skip, you have two bad options: fail forever (blocking all processing) or ACK it (pretending you processed it). With Skip, you can move it to a dead-letter queue for investigation while continuing to process other messages.

For now, Emmett doesn’t support Dead Letter/Poison Message Queues out of the box, but they will be supported in the future. You could already append those events to some specific stream.

In upcoming releases, we’ll also have configurable retry policies based on error type and other factors. Just like we already have for command handlers (e.g. to retry 3 times with exponential backoff for Optimistic Concurrency error).

There’s no easy answer to when to stop and when to skip poison messages. Neither choice is universally correct. A financial system might need all-or-nothing semantics. A social media feed can tolerate inconsistency between views.

That’s also why you can freely group processors within consumers. Best if they share similar resiliency and desired throughput characteristics. If they’re very different, you can always spin up another consumer for the same source and process it differently.

Checkpointing processing

In Emmett, processors own their checkpoints. Each processor independently tracks the last message it processed. The consumer doesn’t maintain any checkpoint state.

When a consumer starts up, it asks all registered processors for their last processed position and starts polling from the earliest one.

It has several benefits:

Independent progress: Processors can move at different speeds. If your MongoDB projector is fast and your analytics processor can get slow at times, they each track their own progress. The slow one doesn’t hold back the fast one.
Isolated failures: If one processor’s checkpoint storage fails, only that processor is affected. Others continue working.
Easy replay: To rebuild a single projection, you just reset that processor’s checkpoint. No need to coordinate with other processors or manage a global position.
Flexibility: Processors can store checkpoints wherever makes sense - in the same database as their read model, in a separate checkpoint table, or anywhere else.
Capability to redistribute the load. As mentioned in the previous points, if you observe that one of the processors is slower or demands more resources, you can freely deploy it separately in a different consumer, and it’ll start where it left off.

The tradeoff? When a consumer restarts, it might poll events that most processors have already seen. If one processor is significantly behind, all processors receive those events again (they just skip them based on their checkpoints). This is why you should group processors by their typical processing pace - don’t put a real-time dashboard projector and a monthly analytics batch processor on the same consumer.

Backpressure: When Processors Can’t Keep Up

Backpressure occurs when processors can’t process messages fast enough for the consumer to deliver them. This is a real operational concern that needs explicit handling.

Emmett doesn’t support it at the moment, but here’s what I’m thinking about it.

There are several strategies, each with tradeoffs:

1. Ignore backpressure: Consumer keeps polling and pushing regardless of processor state.

Pro: Simple, maximum throughput when processors can keep up
Con: Memory grows unbounded, possible OOM, cascading failures

2. Stop on any slowdown: If any processor signals it’s overwhelmed, stop polling.

Pro: Safe, no resource exhaustion
Con: Slowest processor determines overall throughput

3. Force synchronised pace: All processors must process each batch before the next is fetched.

Pro: All processors stay in sync, predictable memory usage
Con: The Slowest processor becomes the bottleneck for all

4. Slow down ingress: Adaptively reduce polling rate based on processor feedback.

Pro: Balances throughput and stability
Con: More complex, needs tuning

5. Rolling buffer: Buffer messages up to a limit, retry delivery to slow processors.

Pro: Absorbs temporary slowdowns, maximises throughput
Con: Needs memory limits, complex failure handling

Different systems need different strategies. Real-time dashboards might use strategy 1 (drop messages rather than lag). Financial transactions might use strategy 3 (consistency over throughput). Event forwarding to Kafka might use strategy 5 (buffer temporary network issues).

I’m leaning toward making this configurable per consumer, with sensible defaults. The default would be a bounded buffer with adaptive polling slowdown.

Scaling: Current State and Future Plans

For now, the big benefit of having dumb consumers is that you can scale them horizontally. Of course, this works for offset-based solutions like event stores and streaming tools like Kafka. It may not always work for systems that remove the message once it’s handled. Still, current consumers are using only event stores as sources; Kafka will likely come next.

You can group processors into consumers by that, reducing the number of polling jobs (one consumer polls/subscribes to one source).

I already mentioned batching, which should also increase the throughput.

Running multiple instances of the same processor causes conflicts. Both process the same events, update the same read models, and corrupt the state. Emmett already has the basic capability to do distributed locking, but it’s not fully plugged yet. This will come in future releases.

For now, checkpointing can detect whether a newer checkpoint is already stored (which can suggest another processor is running) and stop processing.

The recommended approach is to run the consumer as a separate service from the API. Then you can scale it separately. You can also set replicas=1 for the specific consumer to ensure one instance.

Rebuilding Projections

Event Sourcing enables rebuilding read models from events. Bug in projection? Fix code, rebuild. New read model? Populate from history.

With processor-owned checkpoints, you can either rebuild read model from scratch by:

Stopping the processor.
Delete read model data.
Reset the checkpoint to the beginning.
Restart processing.

Or doing blue greeen by:

Creating a new version of your storage (with Pongo, it’s just adding a suffix or prefix to your collection name).
Start consumer since the beginning.
Check if read models are close enough, and stopthe old processor
Start processing.

In Emmett you have even some syntactic sugar on top of consumers and processors to make this easier:

import { rebuildPostgreSQLProjections } from '@event-driven-io/emmett-postgresql';

const rebuilder = rebuildPostgreSQLProjections({
  connectionString,
  projection: shoppingCartsSummaryProjectionV2
});

await rebuilder.start();

This will spin up a new consumer; other consumers and processors continue normally, with their checkpoints unaffected. You can specify the position from which you want to start, and also whether to truncate the end storage.

We’ll need more metrics like gap detection and distributed locking to make it more plug-and-play.

Wrapping Up

The consumer/processor architecture in Emmett is about making event processing concerns explicit and separable:

Consumers handle delivery - getting events from sources to processors. They’re simple by design. When delivery breaks, you know where to look.

Processors handle processing - doing useful things with events. They own their checkpoints, track their own progress, and handle their own failures.

This separation gives you:

Flexibility to mix and match consumers and processors,
Independent scaling of different processing workloads,
Isolated failure domains,
Easy projection rebuilds,
Testability at multiple levels.

The design makes tradeoffs explicit:

Partial progress over all-or-nothing (configurable soon),
Processor-owned checkpoints over global tracking,
Simplicity in consumers, complexity in processors,
Eventual consistency for async operations.

There’s more to build - distributed locking, partitioning, better backpressure handling. There’s still a lot to do, but I believe the foundation is there, and I know real applications are using it already.

I hope that this is a good food for thought, even if you’re not using Emmett. I’m curious about your thoughts and feedback. I’ll try to tackle those cases in more detail in dedicated articles.

If you have questions, feedback, or would like to help me speed up the planned stuff, come chat in the Emmett Discord. We have a small, but welcoming and awesome community.

Cheers!

Oskar

Requeuing Roulette in Event-Driven Architecture and Messaging

I’m always saying that there’s a thin line between good and bad practice, and this thin line is named “Context”.

That’s also true for the (anti-)pattern I’m calling “Requeuing Roulette”. Let’s discuss it today, continuing the “race condition series”:

What’s the Requeuing Roulette? As the name suggests, this technique involves putting a message back into the queue. It also (correctly) suggests that we’re hoping for the best. And sometimes we may be lucky to be true.

The basic primitive for a messaging system is a queue. The producer is putting messages into the queue, and the consumer is getting them on the other end. If everything goes well, the consumer receives them in the order the producer put them (thus, a queue, like a queue in a shop).

If the consumer is not available, the messaging system will try to deliver messages and handle retries for us.

We discussed it in detail in:

Ordering of processing works if we have a single consumer for a single queue. If we have more than one consumer, we lose the ordering guarantee. Why would we want to have more than one consumer? Obviously, to speed up processing. If messages in the queue are not causally correlated, then we can process them in parallel.

What does the smartass “causally correlated” even mean? For instance:

depositing money into a bank account is causally correlated to opening it, as we can’t deposit money if we don’t open it.
depositing money into a bank account is NOT causally correlated to other deposits, as we can deposit as much money as we have (of course, ignoring weird regulations),
money withdrawal is causally correlated to depositing money and other withdrawals, as we need to check the balance, and they may impact it.
Withdrawals and deposits are only causally correlated if they happen on the same bank account; other bank account operations can happen at any time.

You get the idea, aye?

So if we set up a queue to process money transfer events, then it could look as follows:

I assumed that we’re following the advice from the previous article. Besides the event type and payload, we’d also pass the record revision, which represents the logical order of events. It comes from the number incremented with each change. Assuming we’re publishing events after each successful business logic handling, it should be gapless.

You may notice that our queue actually has multiple timelines for each causally correlated message sequence. If we simplify our considerations and assume that all events from a certain account are causally correlated, then we could visualise them as:

It’s fine to process events that are not causally correlated, as by that we’re increasing throughput, not trading off correctness:

But it’s not fine to process messages from the same timeline in parallel, as we may get race conditions when the consumer processing the earlier message will be slower than the one processing the later message.

We actually have two orders in messaging systems:

Queue Order: The order messages are produced to the queue
Processing Order: The order messages are actually consumed and processed.

We already learned that we can detect the out-of-order processing by:

We can also apply techniques like the Phantom record, where we keep data as it comes and take the next steps only when defined conditions are met. And that’s also what I’d recommend in general, but…

But I promised you to talk today about the Requeuing Roulette (anti)pattern, aye? So let’s do it!

If we’re using tools like RabbitMQ, SQS and other classical messaging tooling (so not you Kafka! You’re a streaming or log solution!), then we can put the message back in the queue.

RabbitMQ message ordering documentation states:

Messages published in one channel, passing through one exchange and one queue and one outgoing channel will be received in the same order that they were sent. RabbitMQ offers stronger guarantees since release 2.7.0.

Messages can be returned to the queue using AMQP methods that feature a requeue parameter (basic.recover, basic.reject and basic.nack), or due to a channel closing while holding unacknowledged messages. Any of these scenarios caused messages to be requeued at the back of the queue for RabbitMQ releases earlier than 2.7.0. From RabbitMQ release 2.7.0, messages are always held in the queue in publication order, even in the presence of requeueing or channel closure.

With release 2.7.0 and later it is still possible for individual consumers to observe messages out of order if the queue has multiple subscribers. This is due to the actions of other subscribers who may requeue messages. From the perspective of the queue the messages are always held in the publication order.

The last paragraph seems promising, as it suggests the message will be put back before the next messages, since it was placed in the queue.

Unfortunately, it’s only a best effort. Another place in the documentation states:

When a message is requeued, it will be placed to its original position in its queue, if possible. If not (due to concurrent deliveries and acknowledgements from other consumers when multiple consumers share a queue), the message will be requeued to a position closer to queue head.

So in the worst case, it can even end up like that:

Then the fun will begin. Now we have two messages that we already see will be handled out of order, and we’ll need to requeue them, hoping they land after the message we need to process as the next one. What if we have multiple messages like that? What if our consumer randomly fails when processing our message with revision 12, and we need to requeue it again? How lucky will it be with Requeuing Roulette?

As you can see, the more correlated our messages are and the more we’d like to parallelise them, the more likely we are to face Requeuing Roulette. Typically, we put messages into the same queue so we can process them in order.

When Requeuing Roulette is helpful?

It can be useful if:

We want to get the best parallelism for message processing, and ordering best effort is good enough for us.
Our messages are most of the time not causally correlated or…
Events for the same records/processes are not quickly published one after another, so we could safely retry message before the next one will arrive,
Our consumers are stable, not failing too often.

As you see, those assumptions can be fragile and classical famous last words.

Of course, we can use one of the techniques like:

RabbitMQ routing key, correlation id,
AWS SQS message group id, visibility timeout,
Azure Service Bus sessions,
etc. see Ordering, Grouping and Consistency in Messaging systems for details.

Making this trade-off more in favour of the ordering guarantee or parallelism may reduce it to an acceptable level, but you need to be aware of the risk of an unexpected traffic spike or a different message distribution than you expected.

Dangers of Requeuing Roulette

Even when order doesn’t matter, requeueing has a hidden cost that becomes visible under load.

Suppose you reject the message with requeue set to true. In that case, it can be redelivered to your consumer almost instantly, resulting in a very high workload, since your consumer will reject it again.

Let’s say you have a message that fails because a downstream service is down. You requeue it. It immediately returns to the consumer (maybe even the same one), fails again, and is requeued. This can happen hundreds of times per second. It can also swamp the slow consumer, preventing it from even recovering.

In the worst case, your CPU can be spent processing and requeueing the same 10 messages over and over, while thousands of processable messages sit behind them in the queue (because RabbitMQ will try to put requeued messages before the next messages).

What about Kafka?

Well, in Kafka, this issue doesn’t exist as Messages with the same record key go to the same partition, maintaining order within that partition while allowing parallel processing across partitions.

So Kafka, for the win? Hold your horses!

Only one consumer from the consumer group can handle a specific partition. So, within a single partition, parallelisation isn’t possible. If we map the RabbitMQ queue to Kafka’s partition, then the conclusion can be that Kafka solved it by removing this feature.

Also, when we consume a message from the classical messaging system (like RabbitMQ), it will be removed from the queue. In a streaming solution like Kafka/Pulsar, etc., they will remain in the log until the retention policy kicks in and drops old messages from the partition.

Kafka maintains the offset of the last processed message in each topic partition. You don’t need to requeue messages; you can just rewind the offset to an older position when you want to reprocess messages.

TLDR

The “requeueing roulette” is a symptom of trying to solve a distributed systems problem with a technical solution.

The requeueing roulette is seductive because it promises something impossible: maintaining strict order in a distributed, concurrent system without sacrificing throughput. It’s trying to cheat the fundamental trade-offs of distributed systems.

Still, cheating can take us far enough, but there’s always a danger that we’ll be caught and handcuffed.

If you’re considering using Requeuing Roulette, then consider the other techniques I described in previous articles. I’d treat Requeuing Roulette as a temporary solution and a tradeoff.

In my opinion, if you decide to use it, then the question isn’t whether you’ll abandon it, but how much pain you’ll endure before you do.

The real skill isn’t in making requeueing work - it’s in understanding your actual ordering requirements and choosing the most straightforward solution that meets them. Often, that means accepting that perfect ordering is neither necessary nor worth its cost, especially in the long term.

Read also more in:

Cheers!

Oskar

Handling Events Coming in an Unknown Order

After the last article on Dealing with Race Conditions in Event-Driven Architecture with Read Models, I got such a question from Ben:

You described the scenario where you know what events you should receive, just not the order. But what if you don’t know that? For example, you get an ItemRemovedFromCart event, but the item doesn’t exist in your view of the current state of the cart. Is it an invalid event? Or is there an ItemAddedToCart event that hasn’t come through yet?

That’s a good question, and good questions usually require more depth to give a precise answer. That’s what we’re here for!

Let’s follow up and discuss how to determine whether we have complete information for our events!

What we learned so far

Communication in messaging systems works like a department store. It has multiple cash registers and separate queues for each of them. You can only guess which person will be handled in a specific queue: first in, first out. Between queues, you only know that the slowest will be the one you’re standing in.

Of course, you could put a single cash register and a single queue, and everything would be sequential. Such a setup can work for small groceries with few customers. Tho, for a supermarket, it’d end up with an extremely long waiting queue.

You can think about a single module as a single cash register in a department store or a small grocery store. Inside it, you can get strict ordering of processing, but not in the relationship with the outside world. For instance, you may know that on Monday morning, you’re getting the fresh fruits delivery, and at noon, you’re getting the dairy delivery. And it’s typically like that, but from time to time, because of the fruit delivery delay, you may get your fresh dairy first.

Is it an issue? Not a huge one, as you just want them to come asap so you have fresh stuff to sell. When would it be an issue? If you were running the Milk Shake Cafe and needed both to make your special strawberry shake recipe.

In many systems, ordering is not a key concern, especially when we partition the workload. The issue may arise when we need to correlate separate actions.

In the last article, we discussed the payment verification workflow. To make the final decision, we needed to correlate data from the external payment gateway with our own modules, which calculate fraud scores, check limits, and assess risk. Only after receiving data on available merchant limits and the fraud assessment score could we make the final decision. Those pieces of information could return to us at different times and in a different order.

To resolve it, we were just gathering and aggregating data as they went. Then, after each step, we checked whether we now have all the data. If we had, we were making the final decision; if not, we were storing it as is, assuming that at some point, data would arrive.

If we model that as the workflow, then it’d look like that:

function decide(
  current: PaymentVerification | null,
  event: PaymentVerificationEvent
):
  | PaymentVerification
  | { document: PaymentVerification; events: VerificationEvent[] } {
  current = current ?? {
    paymentId: event.paymentId,
    initialState,
  }

  switch (event.type) {
    // (...) other event handlers
    case 'MerchantLimitsChecked': {
      const updated = {
        ...current,
        merchantLimits: {
          withinLimits: event.withinLimits,
          dailyRemaining: event.dailyRemaining,
          checkedAt: event.checkedAt,
        },
        lastUpdated: event.checkedAt,
      };

      return tryCompleteVerification(updated, event);
    }
    case 'FraudScoreCalculated': {
      if (
        current.fraudAssessment &&
        event.calculatedAt <= current.fraudAssessment.assessedAt
      )
        return current;

      const updated = {
        ...current,
        fraudAssessment: {
          score: event.score,
          riskLevel: event.riskLevel,
          assessedAt: event.calculatedAt,
        },
        lastUpdated: event.calculatedAt,
      };

      return tryCompleteVerification(updated, event);
    }
  }
};

And:

function tryCompleteVerifications(
  current: PaymentVerification,
  event: PaymentVerificationEvent
):
  | PaymentVerification
  | { document: PaymentVerification; events: VerificationEvent[] } {
  // Ignore if we already made decision
  if (current.decision)
    return current;

  // Check if we now have BOTH critical pieces
  if (!current.fraudAssessment || !current.merchantLimits)
    // Don’t have both yet - stay in processing
    return {
      ...current,
      status: 'processing',
      dataQuality: 'processing',
    };

  const decision =
    current.fraudAssessment.riskLevel === 'high'
      ? {
          approval: 'declined',
          reason: 'High fraud risk',
          decidedAt: event.checkedAt,
        }
      : !current.merchantLimits.withinLimits
      ? {
          approval: 'declined',
          reason: 'High fraud risk',
          decidedAt: event.checkedAt,
        }
      : {
          approval: 'approved',
          reason: 'Verified',
          decidedAt: event.checkedAt,
        };

  return {
    document: {
      ...current,
      status: decision.approval,
      decision,
    },
    events: [
      {
        type: 'PaymentVerificationCompleted',
        data: decision,
      },
    ],
  };
};

This works fine, as we know precisely which steps need to happen, so we know what we’re waiting for. And that’s how we made the full loop to Ben’s question. What if we didn’t know which steps we’re waiting for?

How to know what we don’t know?

Let’s have a look at the case brought by Ben: the e-commerce flow. First, we complete the shopping cart by adding and removing items, then we confirm it. The example event flow could look as follows for the online food ordering:

ItemAddedToCart     (cartId: 1, name: Pizza Napoletana) 
ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
CartConfirmed       (cartId:1, confirmedAt: 2025-11-03 11:44:27)

We see here that someone added the first Pizza, then maybe accidentally added it again, corrected their mistake, and confirmed the order.

Then, if that was an online ordering system and we had it integrated with the kitchen ordering, then we could get those events in a different order, for instance:

ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
CartConfirmed       (cartId: 1, confirmedAt: 2025-11-03 11:44:27)
ItemAddedToCart     (cartId: 1, name: Pizza Napoletana) 
ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)

We see that someone removed one Pizza from their shopping cart, which suggests that some information is missing. When we get a confirmation event, we still know that there’s more to come, as an order with a removed item doesn’t make sense. The same goes for the information that one pizza was added; when we correlate it with the removal event having the same cart identifier, we still see zero items in the shopping cart. Once we get the next event, we will finally know that we have more than one item in our shopping cart.

Can we then proceed? Maybe yes and maybe no. For this particular order, it’d be correct, but what if our real order:

ItemAddedToCart     (cartId: 1, name: Pizza Napoletana) 
ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
ItemAddedToCart     (cartId: 1, name: Spaghetti Carbonara)
ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
CartConfirmed       (cartId: 1, confirmedAt: 2025-11-03 11:44:27)

Also, since messaging systems retry to ensure delivery, how would we know that those “doubled” events for adding or removing are actually distinct events and not just retries?

For instance, in such a delivery case:

ItemAddedToCart     (cartId: 1, name: Pizza Napoletana) 
ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
CartConfirmed       (cartId: 1, confirmedAt: 2025-11-03 11:44:27)
ItemAddedToCart     (cartId: 1, name: Spaghetti Carbonara)
ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)

Let’s discuss a few strategies to deal with that!

External vs Internal events

Doctor, it hurts when I bend my arm this way.

Then don’t bend it this way

One of the most common mistakes we learn too late is separating our events into internal and external (or private and public). Internal information can and should be more granular. We need it to be precise in capturing the business context and making our decision.

Yet, other parts of our system don’t need to know all of that. Is the kitchen interested in the details of all the changes procrastinating customer made to their shopping cart? No, they just want the final information on which meal they need to prepare.

So in our example, if we published to the outside world just:

CartConfirmed  {
    cartId: 1, 
    items: [ { name: "Spaghetti Carbonara" } ],
    confirmedAt: "2025-11-03 11:44:27"
}

Such a type of event is also called a Summary Event. We should not mistake it with the latest state. It’s still an event because it tells what has happened business-wise. It gathers all the information needed for other modules and summarises the changes. And no more than that. It should still be as small as possible and expose only the information that other modules need. It’s a contract made between different teams. I wrote about it in detail Internal and external events, or how to design event-driven API.

We can define such an internal event API as:

type ItemAddedToCart = {
  type: 'sc:int:ItemAddedToCart';
  data: {
    cartId: string;
    productItem: ProductItem;
  };
};

type ItemRemovedFromCart = {
  type: 'sc:int:ItemRemovedFromCart';
  data: {
    cartId: string;
    productItem: ProductItem;
  };
};

type CartConfirmed = {
  type: 'sc:int:CartConfirmed';
  data: {
    cartId: string;
    confirmedAt: Date;
  };
};

type ShoppingCartEvent =
  | ItemAddedToCart
  | ItemRemovedFromCart
  | CartConfirmed;

type ProductItem = {
  productId: string;
  quantity: number;
}

and public as:

type CartOpened = {
  type: 'sc:ext:CartOpened';
  data: {
    cartId: string;
    openedAt: Date;
  };
};

type CartConfirmed = {
  type: 'sc:ext:CartConfirmed';
  data: {
    cartId: string;
    productItems: { productId: string; quantity: number }[];
    confirmedAt: Date;
  };
};

type ShoppingCartExternalEvent = CartOpened | CartConfirmed;

As you see, we can even have more than one summary event, and not even be one-to-one with an internal event. Maybe we also have an analytics module that analyses how long it takes the user to make a final decision after adding the first product. Then we may decide to expose such an event, hiding the details of the internal flow. We’re also defending ourselves and minimising the need for versioning when flow changes.

Ok, but how to map internal events into external?

We can enrich them using such a function:

import type { ShoppingCartExternalEvent } from './shoppingCart.external';
import type { ShoppingCart, ShoppingCartEvent } from './shoppingCart.internal';

const enrich = (
  event: ShoppingCartEvent,
  state: ShoppingCart | null,
): ShoppingCartExternalEvent | [] => {
  switch (event.type) {
    case 'sc:int:ItemAddedToCart':
      return state == null
        ? {
            type: 'sc:ext:CartOpened',
            data: {
              cartId: event.data.cartId,
              openedAt: new Date(),
            },
          }
        : [];
    case 'sc:int:CartConfirmed':
      return {
        type: 'sc:ext:CartConfirmed',
        data: {
          cartId: event.data.cartId,
          productItems: state?.productItems ?? [],
          confirmedAt: event.data.confirmedAt,
        },
      };
    default:
      return [];
  }
};

We can then subscribe to internal events in our module, load the state (best to build it from events if we’re using Event Sourcing), and publish enriched events externally.

If we’re using messaging, this means also separating queues/topics. If you’re using Kafka both for internal and external communication, then you should separate topics and have two different topics for outgoing communication, e.g.:

‘carts:events:int’
‘carts:events:out’.

Similarly, for RabbitMQ or similar tools, you should have separate queues for internal and external communications.

This is important, as you can now:

Publish messages that other modules need, decreasing the number of issues with ordering,
Have enrichment as an anti-corruption layer for your internal process changes,
You can have different scaling capabilities for internal and external events. Maybe for internal, you don’t even need a messaging system, maybe outbox or event store subscriptions will be enough? Maybe you could cut costs using AWS SQS for internal communication and AWS Kinesis for cross-module?
You can now define different security for those topics and retention policies.

Sweet, right?

It’s not me, it’s them

Maybe it’s sweet enough for you, but you may also say:

But Oskar, I’m not producing those events, it’s the other team and the external service. If I was responsible for that, I’d go this way, but I can’t change it how messages are published.

I could handwave it and say I pity you, but well, this actually can happen. Let’s see what else we could do about it.

The first idea could be: Let’s add timestamps!

Let’s see how it looks for our example:

11:40:10 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana) 
11:40:10 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
11:42:13 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
11:43:18 - ItemAddedToCart     (cartId: 1, name: Spaghetti Carbonara)
11:44:23 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
11:44:27 - CartConfirmed       (cartId: 1)

And the out of order delivery:

11:40:10 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana) 
11:40:10 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
11:44:23 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
11:44:27 - CartConfirmed       (cartId: 1, confirmedAt: 2025-11-03 11:44:27)
11:43:18 - ItemAddedToCart     (cartId: 1, name: Spaghetti Carbonara)
11:42:13 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)

Would that help? No, because how would we know, based on timestamps, that there will be two more events after confirmation? Timestamps only tell us when a certain operation happens. They could help us order items that were delivered, but they won’t help us know what we’re missing. Because how do we know that there’s a gap in our knowledge? Within a minute, one could do nothing and order or remove a few more items. Also, timestamps might work if the data is coming from the same node, but we don’t have any guarantees across nodes. Read more about clock drift.

What we actually need is the logical clock. One that increments after each operation. So something like:

1 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana) 
2 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
3 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
4 - ItemAddedToCart     (cartId: 1, name: Spaghetti Carbonara)
5 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
6 - CartConfirmed       (cartId: 1)

If we had such, then our delivery would look as follows:

2 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana) 
1 - ItemAddedToCart     (cartId: 1, name: Pizza Napoletana)
5 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)
6 - CartConfirmed       (cartId: 1, confirmedAt: 2025-11-03 11:44:27)
4 - ItemAddedToCart     (cartId: 1, name: Spaghetti Carbonara)
3 - ItemRemovedFromCart (cartId: 1, name: Pizza Napoletana)

If this number were monotonic and gapless, we’d know that if the event has number 3, then we have completeness of information if we received three events; if not, then we’re missing something.

We still need to define the completion criteria and determine, from a business perspective, where we can make a decision or proceed to the next step. Here we know that we can start preparing a meal when the shopping cart is confirmed.

In this case, we got the events in the following order: 2, 1, 5, 6.

We know we’re missing events 3 and 4, so we need to wait for them. Only when we receive them can we proceed. Ok, but how to do it?

What if we kept a list of pending events in our data model? Let’s try that!

Our kitchen order could look as follows:

type KitchenOrder = {
  orderId: string;
  productItems: ProductItem[];
  status: 'Incomplete' | 'Ready' | 'InPreparation';
};

type KitchenOrderCommand =
  | {
      type: 'AddItem' | 'RemoveItem';
      productId: string;
      quantity: number;
    }
  | {
      type: 'Confirm';
      orderId: string;
    };

type ProductItem = {
  productId: string;
  quantity: number;
};

When storing it, we could store it with additional metadata:

type DocumentWithPendingCommands<State, Command> = State & {
  metadata: {
    lastProcessedRevision: number;
    pendingCommands: PendingCommand<Command>[];
  };
};

As you see, besides the regular data, we have two other properties: pending commands and last processed revision.

You can think about pending commands as your git repository on your local disk. It contains the list of all operations that you’ll eventually commit. The rest of the data is like the remote git repository. They will be updated when you push your changes there. Then, the last processed revision will be updated with the revision of the last applied command.

The code for that workflow could look as follows:

function handle(
  event: ShoppingCartEvent,
  document: DocumentWithPendingCommands<
    KitchenOrder,
    KitchenOrderCommand
  > | null,
): DocumentWithPendingCommands<KitchenOrder, KitchenOrderCommand> {
  const { metadata, ...data } = document ?? {
    metadata: {
      lastProcessedRevision: 0,
      pendingCommands: [],
    },
  };

  const state: KitchenOrder = {
    orderId: event.data.cartId,
    productItems: [],
    status: 'Incomplete',
    ...data,
  };

  const updated = {
    ...state,
    metadata: {
      lastProcessedRevision: metadata.lastProcessedRevision,
      pendingCommands: [
        ...metadata.pendingCommands,
        mapToPendingCommand(event),
      ],
    },
  };

  return handlePendingCommands(updated, decide);
}

It takes the current state of the kitchen order. If it doesn’t exist, then we need to set it up with default data. We’re also appending a pending command that’s built from an event.

function mapToPendingCommand(
  event: ShoppingCartEvent,
): PendingCommand<KitchenOrderCommand> {
  switch (event.type) {
    case 'sc:int:ItemAddedToCart':
      return {
        type: 'AddItem',
        productId: event.data.productItem.productId,
        quantity: event.data.productItem.quantity,
        metadata: {
          revision: event.metadata?.revision,
        },
      };
    case 'sc:int:ItemRemovedFromCart':
      return {
        type: 'RemoveItem',
        productId: event.data.productItem.productId,
        quantity: event.data.productItem.quantity,
        metadata: {
          revision: event.metadata?.revision,
        },
      };
    case 'sc:int:CartConfirmed':
      return {
        type: 'Confirm',
        orderId: event.data.cartId,
        metadata: {
          revision: event.metadata?.revision,
        },
      };
  }
}

Why not use a regular event here? We could. The benefit is that then we’d have all the data stored as it came. This could make troubleshooting and correction easier. Downside? We’re coupling the event with our internal business logic. Also, keeping the whole event payload increases the size of our data. The choice is yours.

As you see, we’re taking revision from the event metadata. We’ll get to that later, how to fill it on the producer side. Now, let’s focus on the workflow.

Let’s see what processing pending items looks like:

function handlePendingCommands<State, Command>(
  document: DocumentWithPendingCommands<State, Command>,
  decide: (command: Command, state: State) => State,
): DocumentWithPendingCommands<State, Command> {
  const { metadata, ...data } = document;

  const commandsToHandle = getCommandsReadyToHandle(
    metadata.pendingCommands,
    metadata.lastProcessedRevision,
  );

  // Nothing to do see here, please disperse
  if (commandsToHandle.length === 0) return document;

  let state = data as State;
  for (const command of commandsToHandle) {
    state = decide(command, state);
  }

  const lastCommand = commandsToHandle[commandsToHandle.length - 1];

  return {
    ...state,
    metadata: {
      lastProcessedRevision: lastCommand.metadata.revision,
      pendingCommands: metadata.pendingCommands.filter(
        (a) => a.metadata.revision > lastCommand.metadata.revision,
      ),
    },
  };
}

We need to get the commands ready to be handled. For instance, if we already had commands with the following revisions 2, 6, 1, 4, and now we got an action with number 3, then that means:

We can process actions 1, 2, 3, 4
Action 6 remains, as we’re missing 5.

The filtering can look as follows:

function getCommandsReadyToHandle<Command>(
  pending: PendingCommand<Command>[],
  lastProcessedRevision: number,
): PendingCommand<Command>[] {
  return (
    pending
      // filter out commands that have already been processed
      .filter((cmd) => cmd.metadata.revision > lastProcessedRevision)
      // sort by revision to ensure correct order
      .sort((a, b) => a.metadata.revision - b.metadata.revision)
      // only take commands that are consecutive in terms of revision
      .reduce<PendingCommand<Command>[]>((acc, command) => {
        const lastRevision = acc[acc.length - 1]?.metadata.revision;

        return !lastRevision || command.metadata.revision === lastRevision + 1
          ? [...acc, command]
          : acc;
      }, [])
  );
}

When we filter them out and have actions to process, we need to run the actual logic for each of them. For our case this could look like that:

function decide(
  command: KitchenOrderCommand,
  order: KitchenOrder,
): KitchenOrder {
  switch (command.type) {
    case 'AddItem':
    case 'RemoveItem': {
      if (order.status !== 'Incomplete') return order;

      const updatedItems = new Map(
        order.productItems.map((item) => [item.productId, item.quantity]),
      );
      const current = updatedItems.get(command.productId) ?? 0;

      const multiplier = command.type === 'AddItem' ? 1 : -1;
      const updated = current + multiplier * command.quantity;

      if (updated > 0) {
        updatedItems.set(command.productId, updated);
      } else {
        updatedItems.delete(command.productId);
      }

      return {
        ...order,
        productItems: Array.from(updatedItems.entries()).map(
          ([productId, quantity]) => ({
            productId,
            quantity,
          }),
        ),
      };
    }
    case 'Confirm':
      if (order.status !== 'Incomplete') return order;

      return {
        ...order,
        status: 'Ready',
      };
  }
}

type KitchenOrderCommand =
  | {
      type: 'AddItem' | 'RemoveItem';
      productId: string;
      quantity: number;
    }
  | {
      type: 'Confirm';
      orderId: string;
    };

After processing each command, we’re returning the document with the updated state, filtered-out processed commands, and the last processed revision set to the revision of the last processed command.

Not that big a hassle as it may seem, but…

What’s revision and how to get it?

A revision needs to be created on the producer side. The best is to use Optimistic Concurrency for that. If you don’t know what optimistic concurrency or locking is, then you should. Check my intros:

Essentially, if you’re:

using a typical implementation of optimistic concurrency where each record/document change increments its version/revision,
publishing an event after each business operation.

Then you can use this incremented state revision and pass it in your events metadata. Having that, you’d know precisely on which revision it was recorded and get gapless monotonic numeration. It’d also ensure that each message is produced in a certain order, as operations on the specific record will be sequential. Read also more on how revision can help in Dealing with Eventual Consistency and Idempotency in projections.

But Oskar, what if I have more than one record?

Well, then you either need to store multiple revisions. But this won’t help if you need to correlate data between them, as revision is monotonic and gapless for the specific record.

What about global positions? Well, they’re useful for knowing the order of things, but they won’t help here, as they’re monotonic but may have gaps.

TLDR

Proper modelling in Event-Driven Architecture can spare you a lot of complicated implementation tricks.

If you

define essential events for your process,
ensure that they have completeness of information,
shape contracts and communication between modules, respecting the internal and external split.

Then, when things get easier to handle, we can define conditions that tell us when to take action.

Still, sometimes we may:

have strict ordering needs,
be using a queue that doesn’t give us an ordering guarantee,
need to adjust to the other teams.

Then, using revision can be a decent option to solve things in an organised way.

If you’re dealing with such issues, I’m happy to help you through consulting or mentoring. [email protected]">Contact me and we’ll find a way to unblock you!

Read also more in:

Cheers!

Oskar

Dealing with Race Conditions in Event-Driven Architecture with Read Models

My events came out of order! What should I do?!

Are you familiar with the term “phantom record” and its benefits? No? Let me explain it to you today.

Everyone has a plan until they get punched in the mouth. Design, architecture, and modelling are important, but it’s the actual code that reaches production. That’s also the place where we see all those nasty issues that we haven’t foreseen, like: race conditions, eventual inconsistency, idemNotency, etc.

We realise that our BBC Architecture is just a Box-Box Cylinder on paper; in reality, things get messy. We go from a 2D layout to a 3D or even a 4D view.

We see that our business processes are not so linear and predictable as we draw them. Of course, we have somewhere in the back of our mind that subprocesses can go in parallel, but somehow our tests go down the happy path. Then bugs and incidents come in.

We learn the hard way that real processes have delays; they run in parallel. For instance, when you process a payment, fraud checking, risk assessment, and merchant validation happen simultaneously. That’s efficiency, not a flaw.

And the sad part is that it’s our role to provide this efficiency. Yelling at event-driven clouds won’t help.

Race Conditions in EDA

When we distribute our systems, we’re getting better isolation of failure, we can deploy changes at a different pace, but… But we also get communication going at various paces. We can no longer get a unified view of the business actions in the order of appearance. Now each service has its own linearity.

To integrate services predictably, we’re using messaging tools. They provide us with durability, retries when the recipient is unavailable, and ensure that information flows and data are delivered effectively. Still, they’re not magical creatures; they have their limits. We can’t cheat physics.

Between one service and the other, we’re putting in a queue. That’s the place where we can achieve an ordering guarantee. Typically, one queue, one consumer means that we can achieve ordering. Still, not all tools are giving us that.

Out of order issues may happen when the RabbitMQ queue has multiple consumers racing for messages. When we add consumers, we get better throughput, but we’re risking ordering issues.

Out of order issues may also happen when you’re using tools like SQS or Google PubSub, which only guarantee best-effort ordering.

Or when your outbox pattern deletes processed messages and loses sequence.

Or when network delays shuffle carefully ordered streams.

It’s safe to say that you won’t get any ordering guarantee between different queues. Since queues represent communication flows between modules, we should not assume strict ordering in cross-module communication.

Events vs Rumours?

Continuing with the payment verification process example: Fraud module publishes to one queue. Risk assessment for another. The payment gateway to a third. Messages arrive at your module in the order your system received them, not the order they were created. Fraud detection service flags a high-risk payment at 10:00:01. The payment initiation event from the gateway, created at 10:00:00, arrives at 10:00:02. The fraud score arrives before the payment exists in your system. You might think sorting by timestamp solves this. It doesn’t. Clock skew between services means timestamps lie. Each service node has its own time. It may be similar, but if you have a system with high throughput, the skew may be significant enough, causing more issues than actual help.

But, well, that’s the same case when you’re getting information from the outside world. You can get multiple friends sending you some news, you can get them at different paces, and read them in a different order. You may get obsolete news and then newer, but it can go the other way round. You’re only sure of which you received those messages; to get the truth, you need to correlate that information and do fact-checking based on some rules (e.g. which source is more reliable, newer, etc). Then you can deduce the actual information.

We’re saying that in the event-driven world, events are facts. But that’s just half the truth.

Those events that we store/publish are facts for us, or at least represent the current state of our knowledge.

The events from external systems are rumours at best. We need to interpret them to make them (our) facts.

Ok, but what can we do? Sit and cry?

Of course, we can do more. We’ll discuss today a simple technique with Read Models that can take you far enough.

Let’s say that you’re using Event Sourcing in your system. You take all the messages coming from other systems, and you store them in your event store.

Event Sourcing assumes you can rebuild state by replaying events in sequence. When FraudScoreCalculated arrives before PaymentInitiated, the payment doesn’t exist. You can’t apply a fraud score to nothing. You realise that when your carefully designed domain model throws exceptions. This should never happen, aye?

This isn’t specific to Event Sourcing. Even if you don’t use it but just update read models directly from events, you have the same problem. Your read model update handler for FraudScoreCalculated looks for a payment document to update. No document exists. The update fails.

Such issues are a recurring theme in my consulting, as they are a common concern for my customers. My advice is usually: Don’t try to fight them.

Store data as it arrives and “denoise” on your side. Interpret them and save your own “facts”. Acknowledge the need for a split between internal and external events. Use an Anti-Corruption Layer (ACL) pattern to protect from external chaos.

Read models as Anti-Corruption Layer

Read models can be used as such an ACL. A read model document can have a partial state, with optional fields that are filled as events arrive. You can process each event, updating the fields it can, ignoring those it can’t, and making decisions based on the available data.

Let’s explore how to apply this in practice using our payment orchestration scenario. Let’s say we’re gathering information about the payment verification process, displaying its state, and performing follow-up operations once the process is complete.

When we trigger payment, we need to correlate data from the external payment gateway with our own modules, which calculate fraud scores, check limits, and assess risk.

We could define events using TypeScript, as:

type PaymentVerificationEvent =
  | PaymentInitiated
  | FraudScoreCalculated
  | RiskAssessmentCompleted
  | MerchantLimitsChecked
  | PaymentCompleted
  | PaymentDeclined;

Having

type PaymentInitiated = {
  type: "PaymentInitiated";
  data: {
    paymentId: string;
    amount: number;
    currency: string;
    gatewayId: string;
    initiatedAt: Date;
  };
};

type FraudScoreCalculated = {
  type: "FraudScoreCalculated";
  data: {
    paymentId: string;
    score: number;
    riskLevel: "low" | "medium" | "high";
    calculatedAt: Date;
  };
};

type RiskAssessmentCompleted = {
  type: "RiskAssessmentCompleted";
  data: {
    paymentId: string;
    riskScore: number;
    factors: string[];
    assessedAt: Date;
  };
};

type MerchantLimitsChecked = {
  type: "MerchantLimitsChecked";
  data: {
    paymentId: string;
    withinLimits: boolean;
    dailyRemaining: number;
    checkedAt: Date;
  };
};

type PaymentCompleted = {
  type: "PaymentCompleted";
  data: { paymentId: string; completedAt: Date };
};

type PaymentDeclined = {
  type: "PaymentDeclined";
  data: { paymentId: string; reason: string; declinedAt: Date };
};

Let’s say the fraud system flagged the payment as high-risk before it even existed in your system. Approval happened before the risk assessment was completed. The example race condition can look as follows:

10:15:32.123 - FraudScoreCalculated (score: 85, high risk)
10:15:32.145 - PaymentInitiated (amount: $500)
10:15:32.167 - PaymentCompleted (approved by automated system)
10:15:32.201 - RiskAssessmentCompleted (risk: medium)
10:15:32.234 - MerchantLimitsChecked (within limits)

If we assume that things go as on the whiteboard, and handlers can’t process events for non-existent payments, then we’ll be the ones who get punched in the mouth by reality. They will fail to find documents to update.

The first step is to embrace that we have a problem. Actually, it’s not a problem, but rather a challenge - a scenario we just need to support.

We can start by defining a document that represents a payment’s verification state. The document should include optional fields since you can’t predict which event will arrive first. The projection function processes each event and updates this document, creating it if necessary.

It can look as follows:

type PaymentVerification = {
  paymentId: string;
  payment?: Payment;
  fraudAssessment?: FraudAssessment;
  riskEvaluation?: RiskEvaluation;
  merchantLimits?: MerchantLimits;
  decision?: Decision;
  status: "unknown" | "processing" | "approved" | "declined";
  completionPercentage: number;
  lastUpdated: Date;
  dataQuality: "partial" | "sufficient" | "complete";
};

type Payment = {
  amount: number;
  currency: string;
  gatewayId: string;
  initiatedAt: Date;
};

type FraudAssessment = {
  score: number;
  riskLevel: "low" | "medium" | "high";
  assessedAt: Date;
};

type RiskEvaluation = {
  score: number;
  factors: string[];
  assessedAt: Date;
};

type MerchantLimits = {
  withinLimits: boolean;
  dailyRemaining: number;
  checkedAt: Date;
};

type Decision = {
  approval: "approved" | " declined";
  reason: string;
  decidedAt: Date;
};

Besides the optional data that we’ll gradually fill as events arrive, we have some mandatory fields. The obvious one is paymentId, we need to be able to correlate upcoming data from events. If you have a look at them, all of them have such information. Thanks to that, we know which payments we are verifying. If we’re missing it, then we won’t be able to correlate upcoming data. Then we’re indeed doomed.

Of course, sometimes things can get harder. We do not always have a certain id, sometimes some other data, like external id, idempotence key, correlation id, whatever id. Still, we need to have some field, or multiple fields that allow us to point to that for this event we need to update that document.

How do we update our read model? We need a function that takes the current state (or null if it doesn’t exist) and the event we apply on top of it, returning the new state.

It can look as follows:

function evolve (
  current: PaymentVerification,
  event: PaymentVerificationEvent
): PaymentVerification | null {
  current = current ?? {
    paymentId: event.paymentId,
    initialState,
  }

  switch (event.type) {
    case "PaymentInitiated": 
      return onPaymentInitiated(current, event);
    case "FraudScoreCalculated": 
      return onFraudScoreCalculated(current, event);
    case "RiskAssessmentCompleted": 
      return onRiskAssessmentCompleted(current, event);
    case "MerchantLimitsChecked": 
      return onMerchantLimitsChecked(current, event);
    case "PaymentCompleted": 
      return onPaymentCompleted(current, event);
    case "PaymentDeclined": 
      return onPaymentDeclined(current, event);
};

const initialState: PaymentVerification = { 
  paymentId: undefined!, 
  status: 'unknown', 
  completionPercentage: 0, 
  lastUpdated: new Date(), 
  dataQuality: 'partial' 
};

// Using Emmett it could be defined as
export const paymentVerificationProjection = pongoSingleStreamProjection({
  // collection to where we store projection
  collectionName: 'paymentVerification',
  getDocumentId: (event) => event.data.paymentId,
  evolve,
  canHandle: [
    'PaymentInitiated',
    'FraudScoreCalculated',
    'RiskAssessmentCompleted',
    'MerchantLimitsChecked',
    'PaymentApproved',
    'PaymentDeclined',
  ],
  // use this state when there's no document in collection of certain id
  initialState: () => initialState,
});

And yes, even if the data is not in the state we expect, we can always create the Phantom Record that represents the best state we can in the conditions we have.

What if fraud scoring arrives first?

function onFraudScoreCalculated(
  current: PaymentVerification,
  { data: event }: FraudScoreCalculated): PaymentVerification {
  // Ignore event if we're already ahead of it
  if (current.fraudAssessment && event.calculatedAt <= current.fraudAssessment.assessedAt)
    return current;

  const updated = {
    ...current,
    fraudAssessment: {
      score: event.score,
      riskLevel: event.riskLevel,
      assessedAt: event.calculatedAt,
    },
    lastUpdated: event.calculatedAt,
  }

  if (event.riskLevel !== 'high')
    return {
      ...updated,
      status: 'declined',
      decision: {
        approval: 'declined',
        reason: `High fraud risk detected: score ${event.score}`,
        decidedAt: event.calculatedAt,
      },
    };

  return updated;
}

I told you that you should not use a timestamp, but I did it on my own. We cannot always be prim and proper. When doing ACL, sometimes we can achieve just good enough results. If the event doesn’t have any logical timestamp, then we need to live with what we have. Here, we’re assuming that FraudScoreCalculated always comes from the same source, and then we assume that timestamps have the chance to be consistent. With that, we can use them to see if we haven’t already applied this information, or the newer one. If it’s older, then we just don’t make any changes. Sometimes you’ve got to do what you’ve got to do.

Now, lets have a look on the payment initiation when fraud scoring might have already completed:

function onPaymentInitiated(
  current: PaymentVerification,
  { data: event }: PaymentInitiated): PaymentVerification {
  // Ignore if we already handled this event
  if (current.payment)
    return current;

  return {
    ...current,
    payment: {
      amount: event.amount,
      currency: event.currency,
      gatewayId: event.gatewayId,
      initiatedAt: event.initiatedAt,
    },
    lastUpdated: event.initiatedAt,
  };
};

If fraud scoring was completed first, current already has fraud data. The handler merges payment details into the current state.

We can also mark that in the field that I called dataQuality. It may not be the perfect name, but I used it to indicate that we can have a field that lets us decide whether data is as expected, or if we hit reality. Based on that, we can decide how to display it, or whether to display it at all. Thus, the name “phantom record”. It may look like phantom as it’s missing some data but it’s the best data we have.

function onPaymentCompleted(
  current: PaymentVerification,
  { data: event }: PaymentCompleted): PaymentVerification {
  // Ignore if we already made decision
  if (current.decision)
    return current;

  const decision = 
    current.fraudAssessment?.riskLevel === 'high' 
      ? {
        approval: 'declined',
        reason: `Approval attempted but overridden by fraud (score: ${current.fraudAssessment.score})`,
        decidedAt: event.approvedAt,
      } : {
        approval: 'approved',
        reason: `Approved by ${event.approvedBy}`,
        decidedAt: event.approvedAt,
      };

  return {
    ...current,
    status: decision.approval,
    decision,
    lastUpdated: event.approvedAt,
  };
}

Waiting for Dependencies

Some decisions need multiple pieces. The MerchantLimitsChecked handler waits for both fraud assessment and merchant limits. If we receive merchant limits first and don’t have a fraud assessment yet, we simply store the information; the same applies if it goes the other way. When we eventually get both, we can make the final update, e.g. about the final payment approval or decline.

function onMerchantLimitsChecked(
  current: PaymentVerification,
  { data: event }: MerchantLimitsChecked): PaymentVerification {
  // Ignore if we already handled this event
  if (current.merchantLimits)
    return current;

  const updated = {
    ...current,
    merchantLimits: {
      withinLimits: event.withinLimits,
      dailyRemaining: event.dailyRemaining,
      checkedAt: event.checkedAt,
    },
    lastUpdated: event.checkedAt,
  };

  // Check if we now have BOTH critical pieces
  if (!updated.fraudAssessment || !updated.merchantLimits)
    // Don't have both yet - stay in processing
    return {
      ...updated,
      status: "processing",
      dataQuality: "processing",
    };

  // Both present - can make final decision
  const decision =
    updated.fraudAssessment.riskLevel === "high"
      ? {
          approval: "declined",
          reason: "High fraud risk",
          decidedAt: event.checkedAt,
        }
      : !updated.merchantLimits.withinLimits
      ? {
          approval: "declined",
          reason: "High fraud risk",
          decidedAt: event.checkedAt,
        }
      : {
          approval: "approved",
          reason: "Verified",
          decidedAt: event.checkedAt,
        };
        
  return {
    ...updated,
    status: decision.approval,
    decision,
  };
}

And now we’re reaching the grey area, where this can become an anti-pattern. I even made this mistake when writing this article. I wrote initially, “we can make the final decision”, but then changed it to “we can make the final update”. Why?

Projections, ACL responsibility and Workflows

I’m sure you’ve used or seen Anti-Corruption Layers. They’re usually places where all the weird logic lands. We want to protect our core logic, pushing all hacks to one, hidden place. This strategy makes some sense; that’s why it’s called Anti-Corruption Layer. Yet…

Yet, we should not push it to the limits, as we’ll end up with an unmaintainable beast. With the last example, we’ve reached or even passed the limits of what projection should be responsible for. The projection should just interpret upcoming information and store the result. It should not make decisions. Business logic is responsible for making decisions.

What we actually ended up with in the last step is a form of process manager, or workflow, so a state machine that listens to events, gets its current state and makes further decisions. And what’s the best way to inform the outside world about making new decisions? Well, producing a new event.

We could define an event as:

type PaymentVerificationCompleted = {
  type: "PaymentVerificationCompleted";
  data: { 
    approval: "approved" | " declined";
    reason: string;
    decidedAt: Date; 
  };
}

As we’re making decisions, let’s be frank and rename evolve function to decide and produce a new event when we distilled it from the messy outside world?

It could look like that:

function decide(
  current: PaymentVerification | null,
  event: PaymentVerificationEvent
):
  | PaymentVerification
  | { document: PaymentVerification; events: VerificationEvent[] } {
  current = current ?? {
    paymentId: event.paymentId,
    initialState,
  }

  switch (event.type) {
    // (...) other event handlers
    case "MerchantLimitsChecked": {
      const updated = {
        ...current,
        merchantLimits: {
          withinLimits: event.withinLimits,
          dailyRemaining: event.dailyRemaining,
          checkedAt: event.checkedAt,
        },
        lastUpdated: event.checkedAt,
      };

      return tryCompleteVerification(updated, event);
    }
    case "FraudScoreCalculated": {
      if (
        current.fraudAssessment &&
        event.calculatedAt <= current.fraudAssessment.assessedAt
      )
        return current;

      const updated = {
        ...current,
        fraudAssessment: {
          score: event.score,
          riskLevel: event.riskLevel,
          assessedAt: event.calculatedAt,
        },
        lastUpdated: event.calculatedAt,
      };

      return tryCompleteVerification(updated, event);
    }
  }
};

We update the state and try to complete verification if we gathered both merchant limits and risk assessment. This could be encapsulated in a dedicated method, as the logic is the same now matter which event came first:

function tryCompleteVerifications(
  current: PaymentVerification,
  event: PaymentVerificationEvent
):
  | PaymentVerification
  | { document: PaymentVerification; events: VerificationEvent[] } {
  // Ignore if we already made decision
  if (current.decision)
    return current;

  // Check if we now have BOTH critical pieces
  if (!current.fraudAssessment || !current.merchantLimits)
    // Don't have both yet - stay in processing
    return {
      ...current,
      status: "processing",
      dataQuality: "processing",
    };

  const decision =
    current.fraudAssessment.riskLevel === "high"
      ? {
          approval: "declined",
          reason: "High fraud risk",
          decidedAt: event.checkedAt,
        }
      : !current.merchantLimits.withinLimits
      ? {
          approval: "declined",
          reason: "High fraud risk",
          decidedAt: event.checkedAt,
        }
      : {
          approval: "approved",
          reason: "Verified",
          decidedAt: event.checkedAt,
        };

  return {
    document: {
      ...current,
      status: decision.approval,
      decision,
    },
    events: [
      {
        type: "PaymentVerificationCompleted",
        data: decision,
      },
    ],
  };
};

We’re returning not only the new state, but also the new event. It can be published to the local module queue or just stored inside the event store.

Essentially, we’re making a chaotic outside world linear based on the order of our observations. We can’t change the outside world, but we can at least know why and where we’ve made our decisions.

This is actually the same pattern I showed during the webinar on modelling and implementing distributed processes:

We aggregate data until we have enough of it to make the next decision. Is it perfect? Nah, but sometimes best you can is good enough.

When This Approach Works (And When It Doesn’t)

This approach works when:

Your business logic can handle partial data: Payment approval can often proceed with fraud score and limits check, even if risk assessment is pending. E-commerce fulfilment can start when payment is confirmed, even if the recommendation engine results are still processing.

Eventual consistency is acceptable: The payment dashboard might show “processing” for a few hundred milliseconds while verification completes. Most business users can tolerate brief inconsistency in exchange for system responsiveness.

Decisions can be corrected or refined: A payment approved with partial data can be flagged for additional review when complete risk data arrives. An order can be expedited or delayed based on a complete customer analysis.

You have clear business rules for conflicts: When fraud assessment contradicts approval, fraud wins. When risk assessment arrives late, it updates data quality but doesn’t reverse committed decisions.

This approach struggles when:

Perfect consistency is required: Financial accounting, legal compliance, and safety-critical systems often can’t tolerate any inconsistency, even briefly.

Business logic requires complete data: Some decisions genuinely can’t be made with partial information. Credit limit increases might require a complete financial analysis before any approval.

Rollback costs are high: If reversing a decision is expensive or impossible, you need stronger ordering guarantees before committing.

Users can’t tolerate uncertainty: Some interfaces need to show definitive status immediately, not “processing” or “partial data available.”

Conclusion: Embracing the Chaos

The world is chaotic, and we can’t stop the chaos, but we can stop fighting the chaos. External events are rumours about what happened in other systems. Your read model can store the information you derive from those rumours. The evolve function processes whatever arrives, in whatever order, building state incrementally, making your local interpretation.

Undeniably, it’s a workaround of some sort. But it’s also an acknowledgement that distributed systems don’t guarantee order across boundaries. When you can’t control the topology, you adapt on your side. Store data as it arrives. Denoise in your projections. Create clean internal events for downstream systems.

Sometimes the proper solution is fixing your message topology. Route related events to the same partition. Use predictable identifiers for correlation. But when external systems constrain your options, when organisational boundaries limit your control, when legacy integrations force your hand - you’ve got to do what you’ve got to do.

Build your local models and live with partial state. Process events in any order. Make decisions with available data. That’s how you build reliable systems on unreliable foundations.

If you’re dealing with such issues, I’m happy to help you through consulting or mentoring. [email protected]">Contact me and we’ll find a way to unblock you!

Read also more in:

Cheers!

Oskar

How to build MongoDB Event Store

I have always said that MongoDB is not the best choice for event storage, and guess what? I just released the stable version of the MongoDB event store in Emmett.

And let me explain today how to do it. It’ll be a detailed article, but I think it’s an interesting piece, so fasten your seat belts!

A few things are needed for that. The first and most important thing is to have a motivated contributor!

I was lucky to have such with Alexander Lay-Calvert. He did most of the hard work; I was helping conceptually and with final touches. So, what you read today is a summary of our conjoined effort with a majority made by Alex.

What else is needed? Let’s discuss that today!

Event stores as key-value stores

In this article, we’ll use the canonical definition of event sourcing.

Event Sourcing is about making decisions, capturing their outcomes (so events) and using them to make further decisions (so events are the state). We do that by reading all the events we recorded already for the specific process/entity. We apply them and form in-memory decision model that we use to check our business rules, and deciding whether we record the next events.

Event Streaming is about moving information from one place to another and integrating multiple components. Read more about in Event Streaming is not Event Sourcing!.

Event stores are not messaging tools. They may have similar capabilities as Event Streaming solutions, but the focus is different:

event stores are focus on consistency, durability and quality of data,
event streaming solutions (like Kafka) are focus on delivery, throughput and integration.

Event stores are key-value databases At least logically. In relational databases, records are called rows; in document databases, documents; in Event Sourcing, they’re called streams.

In event stores, the stream is built from:

the key represents a record identifier (e.g. order id, invoice number, car license plate number, etc.),
the value represents a sequence of business events that happened for the specific record (e.g. OrderInitiated, ShipmentScheduled, OrderCompleted).

Simple as that, still, most of the time, we were taught that event stores are append-only logs looking more or less like that:

Indeed, physically, most of them are append-only logs. See: Emmett PostgreSQL storage, EventStoreDB, Marten, Axon Server. But well, the same can be said for the relational databases. Internally, each operation (INSERT/UPDATE/DELETE) is appended to the Write-Ahead/Transaction Log. Then, upon transaction commit, they’re applied to specific tables.

I haven’t seen many (if at all) introductions to relational databases explaining them as append-only logs, and that’s fine, as it wouldn’t be accessible. We should also stop doing that for event stores.

Again, even if internally, our event stores are append-only logs, then still each event is logically mapped to specific streams:

So again, logically, our event stores are key-value databases:

And that’s great, as it’s much easier to reason about the guarantees we should expect from our store after realising that.

Plus, it allows us to implement event stores on top of a bit more exotic storage like MongoDB, DynamoDB, CosmosDB or Blob Storage.

Ok, enough intro! Let’s discuss how to do it using the MongoDB example!

Basic Event Stream Definition

Let’s try to make the definition of our event stream more formal. Formal? Yes, let’s create the code with type definitions to express our requirements more precisely. Prototyping is an underestimated design skill.

I’ll use TypeScript because it’s a decent language for that. I won’t use much fancy structure, so it should be understandable. If not, paste it into the ChatGPT and translate it into your favourite language. That’s how we code today, aye?

We said that the event store is a key-value database and that records are called streams. Let’s write that down then:

type Stream = unknown;

type StreamName = unknown;

type EventStore = Map<StreamName, Stream>;

It reflects our knowledge; the shape of Stream and type of Stream Name are unknown to us, but we already know that Event Store is a database, a map where Stream Name is a key and Stream is a value.

Let’s now describe the Stream: it represents a sequence of business events that happened for a specific record.

type Event = unknown;

type Stream = Event[];

type StreamName = unknown;

type EventStore = Map<StreamName, Stream>;

Easy peasy, boom! We’re done!

Ok, not quite. Let’s do a reality check with…

Requirements for event store

What are the requirements, then? In my opinion, at least:

Appending event at the end of the stream. We need to be able to record new business facts for a specific entity,
Reading all events from the stream. That’s a must to get the current entity state from events. We read all recorded events and apply them to get the current state.
The guarantee of the ordering within the stream. It’s important to know the order of events in our process. Both in getting the state and integrating it with other workflows.
Being able to read your writes. As event stores are databases, you should get the new events right after you append them.
Strong-consistent, atomic writes and optimistic concurrency. It’s a must to know whether we’re making a decision based on the latest state and whether conflicting changes have been detected. In other words, ensuring we won’t end up in the wrong state.
Store both event data and metadata. Event data is obvious; it gathers all information related to the business fact that happened. Metadata is needed for the generic handling, such as telemetry data.

The 2nd tier of event stores features (so great to have but not must-haves):

Being able to subscribe to notifications about newly appended events (best if it’s push-based). Also, having at-least once delivery guarantee. That’s important for integrating different business workflows from different streams.
Built-in projections for building read models. We need to have performant queries. Reading all events from the stream is fine for business logic but not good enough for querying.
Global ordering of events. It’s both useful for building read models from multiple streams and integrating workflows.
Streams archiving. So, being able to move the obsolete events (e.g. from completed workflows) to some cold storage. The fact that we record all business information doesn’t mean that we need to keep them forever.

We could expand our Event Store definition into:

interface EventStore {
  readStream(streamName: StreamName): Event[];

  appendToStream(streamName: StreamName, events: Event[]): void;
}

And show pseudo-code implementation as:

function getEventStore(): EventStore {
  const streams = new Map<StreamName, Stream>();

  const appendToStream = (streamName: StreamName, events: Event[]): void => {
    const currentEvents = streams.get(streamName) ?? [];

    streams.set(streamName, [...currentEvents, ...events]);
  };

  const readStream = (streamName: string): Event[] => {
    return streams.get(streamName) ?? [];
  };

  return {
    appendToStream,
    readStream,
  };
}

Now, let’s validate how that fits into the MongoDB capabilities.

MongoDB specifics

Document databases are key-value databases whose values have a defined structure. That is why they are called documents. We can compare them to paper applications we send to some government departments. Documents, just like paper ones, can hold various fields and types of data. Usually, they are stored as JSON-like objects.

MongoDB is a document database. Documents are organised into collections, which are similar to tables in relational databases but without a fixed structure. That means that each document in a collection can have a different set of fields, allowing for a more adaptable and dynamic data model. By storing related data in a single document, MongoDB reduces the need for complex joins, making data retrieval faster and simpler for many applications.

As I wrote in my other article, contrary to common belief, document data is structured but just less rigidly. We should define the schema. It should be denormalised, and it is best not to contain cross-document references. It can have relations but not be enforced strictly in a relational way.

Of course, this lack of a fixed schema can also be seen as a downside when maintaining advanced data consistency and integrity, as there are fewer built-in constraints than in relational databases. Relational databases might be more suitable for applications that demand strict data relationships and complex transactions.

Storing event as a document

The first idea could be to store events as documents in the events collection.

That could work, as each event type will help a different data set. You should expect different information in EmployeeOnboardingCompleted and RoomReservationRejected. MongoDB’s flexible schema would benefit us.

When appending events, we would insert a new document. We could nest data and metadata inside the document payload.

How would we read the events from the specific stream? We could keep the reference for the stream name in the event metadata. MongoDB allows indexing documents, so technically this could work.

We could type our event as:

type Event = {
  type: string,
  data: unknown,
  metadata: {
    streamName: StreamName
  }
}

Still, even with indexes, it’s not an optimised way to use MongoDB; we’ll be storing thousands or millions of events, and most of them will have different stream names. That means high cardinality of values, which is not optimal for indexing.

Also, how would we enforce the consistency and concurrency guarantees? How would we enforce optimistic concurrency so that when two conflicting events append, the first succeeds and the other fails? MongoDB allow conditional checks when storing new data, but only for a single document.

Relational databases can handle that better, as they’re designed to handle such relations and multi-record transactions. So, for relational databases, a single table with events as rows works pretty well, but not here. See more in:

Storing an event as a document will not give us proper guarantees and performance, so it’s a no-go. Document databases work best if we access them by document id. That’s how we can enforce consistency and concurrency and make it performant.

What if we were…

Storing event stream as a document

Let’s look again at our type definition:

type StreamName = string;

type Event = {
  type: string;
  data: unknown;
  metadata: {
    streamName: StreamName;
  };
};

type Stream = Event[];

What if we just stored the Stream as a document containing events? We could also store additional metadata like:

stream type (e.g. Room Reservation, User Onboarding etc.),
stream position representing the number of events in the stream; this could also be used for the optimistic concurrency check,
diagnostic information like when the stream was created, when it was last updated, etc.

Having that, we could redefine our stream type definition into:

type Stream = {
  events: Event[];
  metadata: {
    streamType: string;
    streamPosition: number;
    createdAt: Date;
    updatedAt: Date;
 };
};

Let’s adjust our dummy Event Store implementation, as it’ll keep our focus on the design discussion without additional noise. No worries, we’ll get to the real MongoDB.

function getEventStore(): EventStore {
  const streams = new Map<StreamName, Stream>();

  const appendToStream = (streamName: StreamName, events: Event[]): void => {
    const now = new Date();

    // 1. Get the current stream or set the default
    const stream = streams.get(streamName) ?? {
      events: [],
      metadata: {
        streamType: undefined!, // 🤔 how to get it?
        createdAt: now,
        streamPosition: 0,
      },
    };

    const streamExists = stream.metadata.streamPosition > 0;

    // 2. Append events
    stream.events.push(...events);
    /// and update stream metadata
    stream.metadata.streamPosition += events.length;
    if (streamExists) stream.metadata.updatedAt = now;

    streams.set(streamName, stream);
  };

  const readStream = (streamName: StreamName): Event[] => {
    return streams.get(streamName)?.events ?? [];
  };

  return {
    appendToStream,
    readStream,
  };
}

Not surprisingly, we now need to use the nested structure. When appending events, we create a new stream if it doesn’t exist. Then, we append events to the nested collection and update metadata. That sounds good, but how do we get the stream type?

Stream Type

There are multiple ways to do it:

we could add additional options parameters that’d contain the name,
if we’re using languages like C# or Java, we could add the generic parameter with type and get its name,
we can change the first parameter to be an object containing stream name and stream type,
we can use the URN-bases format of Stream Name to include a stream type, e.g. “{streamType}:{streamId}“.

All of them can be valid, but my personal preference is the last one. Why? Again, event stores are key-value databases. For them, the key to consistency is a proper key strategy. We’d like to enforce our stream’s uniqueness for each type of stream. It’s okay if different stream types share the same id (e.g., the date when a reservation was made or an order was placed) as long as the combined stream name is unique.

We could then redefine our stream name to:

type StreamType = string;
type StreamName<T extends StreamType = StreamType> = `${T}:${string}`;

I use the nice TypeScript feature called Literal Types. Essentially, it allows us to define the strongly typed keys of a certain format. Thanks to that, the TypeScript compiler won’t let us provide stream names that don’t follow our format.

Now, we can add the function to parse our formatted stream name:

function fromStreamName<T extends StreamType>(
  streamName: StreamName<T>,
): StreamNameParts<T> {
  const parts = streamName.split(':') as [T, string];
  return {
    streamType: parts[0],
    streamId: parts[1],
  };
}

type StreamNameParts<T extends StreamType = StreamType> = {
  streamType: T;
  streamId: string;
};

We could use this function in our append method to get the stream type. Let’s skip that and focus on the final part, which is related directly to MongoDB.

Stream Id

If our stream name is a string of a specific format, potentially allowing any string as stream ID, then how would we fit into MongoDB ObjectID, which is used as document id?

From MongoDB docs:

ObjectIds are small, likely unique, fast to generate, and ordered. ObjectId values are 12 bytes in length, consisting of:

A 4-byte timestamp, representing the ObjectId’s creation, measured in seconds since the Unix epoch.

A 5-byte random value generated once per process. This random value is unique to the machine and process.

A 3-byte incrementing counter, initialised to a random value.

ObjectId doesn’t allow the provision of any string, so we need to work around it. We can add the property with the stream name and index it. It won’t be perfect, but it will be fine for most cases.

Our adjusted type definition will look as follows:

type Stream<
  Type extends StreamType = StreamType,
  Name extends StreamName<StreamType> = StreamName<StreamType>,
> = {
  streamName: Name;
  events: Event[];
  metadata: {
    streamId: string;
    streamType: Type;
    streamPosition: number;
    createdAt: Date;
    updatedAt?: Date;
 };
};

As you can see, this clearly shows that our stream has its type and the name derived from it.

The updated dummy implementation will look as follows:

function getEventStore(): EventStore {
  const streams = new Map<StreamName, Stream>();

  const appendToStream = <T extends StreamType = StreamType>(
    streamName: StreamName<T>,
    events: Event[],
  ): void => {
    const now = new Date();

    const { streamId, streamType } = fromStreamName(streamName);

    // 1. Get the current stream or set the default
    const stream = streams.get(streamName) ?? {
      streamName,
      events: [],
      metadata: {
        streamId,
        streamType: streamType,
        createdAt: now,
        streamPosition: 0,
      },
    };

    const streamExists = stream.metadata.streamPosition > 0;

    // 2. Append events
    stream.events.push(...events);
    /// and update stream metadata
    stream.metadata.streamPosition += events.length;
    if (streamExists) stream.metadata.updatedAt = now;

    streams.set(streamName, stream);
  };

  const readStream = (streamName: StreamName): Event[] => {
    return streams.get(streamName)?.events ?? [];
  };

  return {
    appendToStream,
    readStream,
  };
}

We’re almost there: finally implementing it in MongoDB! The final decision remains to be made.

Streams Collection layout

Ok, we already know what our Stream-as-a-Document representation should look like, and we’re feeling comfortable with it. Let’s discuss which collection we should store it in.

The most obvious choice would be to have a single collection with all streams. That should work, as Mongo works well with even a large number of documents.

Still, as we already know that streams have types, we could keep all the streams of the specific type in dedicated collections. That should help to optimise stream usage, potentially apply precise indexing strategies, and make other optimisations.

We could also allow custom collection resolution. For instance, if you’d like to separate storage per application module.

Or separate storage for each customer?

Or maybe use a single database but combine the stream type with the customer name, or use another sharding strategy. Why not?

Moreover, we can also allow users to select different Mongo databases for storage and logical separation.

I recommend using collection per stream type as the default choice, but two other approaches are also valid depending on the needs, so why not allow them all? Let’s do it!

We’ll start by defining the options that users can provide to select our storage layout:

Here are the single collection options:

type CollectionPerStreamTypeStorageOptions = {
  type: 'COLLECTION_PER_STREAM_TYPE';
  databaseName?: string;
};

Collection per stream type:

type SingleCollectionStorageOptions = {
  type: 'SINGLE_COLLECTION';
  collectionName?: string;
  databaseName?: string;
};

And a bit more advanced custom resolution

type CustomStorageOptions = {
  type: 'CUSTOM';
  collectionFor: <T extends StreamType>(streamType: T) => CollectionResolution;
};

type CollectionResolution = {
 databaseName?: string;
  collectionName: string;
};

We allow the user to provide the function that will return the database name and collection name based on a defined algorithm.

Let’s group it in the single union type marking that the need to provide one of those options:

type StorageOptions =
  | SingleCollectionStorageOptions
  | CollectionPerStreamTypeStorageOptions
  | CustomStorageOptions;

The collection resolution could look as follows:

function resolveCollectionAndDatabase<T extends StreamType>(
  streamType: T,
  options: StorageOptions,
): CollectionResolution {
  switch (options.type) {

    case 'SINGLE_COLLECTION':
      return {
        collectionName: options.collectionName ?? defaultCollectionName,
        databaseName: options.databaseName,
      };

    case 'COLLECTION_PER_STREAM_TYPE':
      return {
        collectionName: streamType,
        databaseName: options.databaseName,
      };

    case 'CUSTOM':
      return options.collectionFor(streamType);
  }
}

const defaultCollectionName = 'streams';

I think that’s straightforward, but to recap:

For single collection storage, we resolve the collection name based on the provided value or the default name.
For collection per stream type, we use the stream type name as the collection name.
For custom resolution, we take whatever the user’s creativity gives us.

What would you say if we finally use MongoDB?

Implementing MongoDB event store

Let’s define the function that will provide us with the real MongoDB connection. It can look as follows:

function collectionFor<T extends StreamType>(
  streamType: T,
  client: MongoClient,
  options: StorageOptions,
): Collection<Stream<T>> {
  const { collectionName, databaseName } = resolveCollectionAndDatabase(
    streamType,
    options,
  );

  const db = client.db(databaseName);

  return db.collection<Stream<T>>(collectionName);
}

We use stream type and MongoDB client to resolve the database and return collection based on the storage resolution.

Of course, we could add caching, automatic indexing on stream name, etc. Emmett does all of that, but let’s keep it out of this for brevity.

With the collection resolution, we can update our event store implementation so that it is no longer dummy and use MongoDB.

As we’ll be doing now a serious database operation, let’s change the Event Store definition into:

interface EventStore {
  readStream(streamName: StreamName): Promise<Event[]>;

  appendToStream(streamName: StreamName, events: Event[]): Promise<void>;
}

Yes, I just changed the result types to return Promise, making our operations asynchronous.

Now, let’s define our event store setup. We’ll need to take MongoDB client and storage options.

type MongoDBEventStoreOptions = {
  client: MongoClient;
  storage: StorageOptions;
};

function getEventStore(options: MongoDBEventStoreOptions): EventStore {
  const { client, storage } = options;

  // (...) the rest of the code. We'll get here
}

Here, we assume that the injected client is connected. Of course, that’s a simplified version; in Emmett, we support passing clients in any state, providing the connection string, and handling the client lifetime internally.

Reading stream from MongoDB document

Let’s see what our updated read stream will look like:

function getEventStore(options: MongoDBEventStoreOptions): EventStore {
  const { client, storage } = options;

  const readStream = async <T extends StreamType = StreamType>(
    streamName: StreamName<T>,
  ): Promise<Event[]> => {
    const { streamType } = fromStreamName(streamName);

    // Resolve collection
    const collection = collectionFor<T>(streamType, client, storage);

    // Read events from the stream document
    const stream = await collection.findOne<WithId<Pick<Stream<T>, 'events'>>>(
      {
        streamName: { $eq: streamName },
      },
      {
        // just reading events
        projection: {
          events: 1,
        },
      },
    );
    return stream?.events ?? [];
  };

  // (...) appendToStream

  return {
    appendToStream,
    readStream,
  };
}

As you can see, it looks the same as our dummy implementation. The difference is that we’re using the MongoDB API. As we just want to read events, we can benefit from projection that allows us to select only events without all stream metadata. It is a small but nice performance improvement.

Appending to the stream in the MongoDB document

Let’s now implement our append method! It’ll look as follows:

function getEventStore(options: MongoDBEventStoreOptions): EventStore {
  const { client, storage } = options;

  const appendToStream = async <T extends StreamType = StreamType>(
    streamName: StreamName<T>,
    events: Event[],
  ): Promise<void> => {
    const now = new Date();

    const { streamId, streamType } = fromStreamName(streamName);

    // Resolve collection
    const collection = collectionFor<T>(streamType, client, storage);

    // Append events upserting the document
    await collection.updateOne(
      {
        streamName: { $eq: streamName },
      },
      {
        // append events
        $push: { events: { $each: events } },
        // increment stream position
        $inc: {
          'metadata.streamPosition': events.length,
        },
        // set default metadata
        $setOnInsert: {
          streamName,
          'metadata.streamId': streamId,
          'metadata.streamType': streamType,
          'metadata.createdAt': now,
        },
        // update metadata
        $set: {
          'metadata.updatedAt': now,
        },
      },
      { upsert: true },
    );
  };

  // (...) readStream

  return {
    appendToStream,
    readStream,
  };
}

Again, logically it works as our dummy implementation. The difference is in using MongoDB API. As we make append-only changes, we don’t need to read anything from the stream.

We’re using updateOne to do an upsert. MongoDB has a nice way of making atomic updates, such as appending to an array or incrementing a value. Thanks to that all of those changes will be applied to our document as a single atomic operation.

Optimistic Concurrency

The final bit to provide the proper consistency guarantees is optimistic concurrency. You can read my general introduction and technical implementation guidance.

We want to detect conflicting updates and race conditions. In other words, it handles concurrency correctly.

Optimistic concurrency is called optimistic, as we assume that conflict situations will be rare. A conflict arises when two users try to change the same record at the same time. In our case, that means appending an event to a stream. When this happens, we will only allow the first user to update the state. All other appends will be rejected.

For detection of whether the conflicting change was made, we’ll use a stream version as it is incremented with each append.

What does an optimistic concurrency implementation look like?

Return the current stream position while reading events.
Append the event, sending the current position as the expected one.
Check if the position in the stream equals the expected position sent in the append operation.
If they match, allow appending and set a new entity stream position.
If not, throw or return an error.

To be able to do it, we need to modify our event store definition:

interface EventStore {
  readStream(streamName: StreamName): Promise<ReadStreamResult>;

  appendToStream(
    streamName: StreamName,
    events: Event[],
    options?: AppendToStreamOptions,
  ): Promise<AppendToStreamResult>;
}

type ReadStreamResult = { currentStreamPosition: number; events: Event[] };

type AppendToStreamOptions = { expectedStreamPosition: number };
type AppendToStreamResult = { nextStreamPosition: number };

We added result types and extended them with the expected stream position information. We also added an optional parameter to provide the expected position during the append operation.

Let’s now apply that to our read stream method:

function getEventStore(options: MongoDBEventStoreOptions): EventStore {
  const { client, storage } = options;

  const readStream = async <T extends StreamType = StreamType>(
    streamName: StreamName<T>,
  ): Promise<ReadStreamResult> => {
    const { streamType } = fromStreamName(streamName);

    // Resolve collection
    const collection = collectionFor<T>(streamType, client, storage);

    // Read events from the stream document
    const stream = await collection.findOne<
      WithId<Pick<Stream<T>, 'events' | 'metadata'>>
    >(
      {
        streamName: { $eq: streamName },
      },
      {
        // just reading events
        projection: {
          events: 1,
          'metadata.streamPosition': 1,
        },
      },
    );
    return {
      events: stream?.events ?? [],
      currentStreamPosition: stream?.metadata.streamPosition ?? 0,
    };
  };

  // (...) appendToStream 

  return {
    appendToStream,
    readStream,
  };
}

Not much has changed; we’re just reading the stream position and returning it (defaulting to 0).

How would the append to stream method change?

function getEventStore(options: MongoDBEventStoreOptions): EventStore {
  const { client, storage } = options;

  const appendToStream = async <T extends StreamType = StreamType>(
    streamName: StreamName<T>,
    events: Event[],
    options?: AppendToStreamOptions,
  ): Promise<AppendToStreamResult> => {
    const now = new Date();

    const { streamId, streamType } = fromStreamName(streamName);

    // Resolve collection
    const collection = collectionFor<T>(streamType, client, storage);

    let expectedStreamPosition: number;
    if (options?.expectedStreamPosition) {
      // 1.a. Use provided expected stream position, or...
      expectedStreamPosition = options.expectedStreamPosition;
    } else {
      // 1.b. Get the current stream version
      const currentStream = await collection.findOne<
        WithId<Pick<Stream<T>, 'metadata'>>
      >(
        { streamName: { $eq: streamName } },
        {
          // just reading the stream position
          projection: {
            'metadata.streamPosition': 1,
          },
        },
      );

      // and use it as the expected one
      expectedStreamPosition = currentStream?.metadata.streamPosition ?? 0;
    }

    const nextStreamPosition = expectedStreamPosition + events.length;

    // 3. Append events upserting the document
    const updatedStream = await collection.updateOne(
      {
        streamName: { $eq: streamName },
        'metadata.streamPosition': expectedStreamPosition,
      },
      {
        // append events
        $push: { events: { $each: events } },
        // set default metadata
        $setOnInsert: {
          streamName,
          'metadata.streamId': streamId,
          'metadata.streamType': streamType,
          'metadata.createdAt': now,
        },
        // update metadata
        $set: {
          'metadata.streamPosition': nextStreamPosition,
          'metadata.updatedAt': now,
        },
      },
      { upsert: true },
    );

    // 3. In case of expected version mismatch, throw an error
    if (updatedStream.upsertedCount === 0) {
      throw new Error(`Stream version doesn't match`);
    }

    return { nextStreamPosition };
  };

  return {
    appendToStream,
    readStream,
  };
}

A bit more is happening here. After we resolve the collection by the stream type, we either take the provided expected position from options (if it was provided) as the current one or read it from the stream document.

Once we have it, we pass it on to the update statement.

If the document with a specified stream name and position wasn’t found, that means the stream doesn’t exist or the stream position is different. Even though MongoDB will try to insert the document, it’ll fail as we have the unique index constraint on the stream name. If no document was upserted, then that means that there was an optimistic concurrency conflict, and we can throw an error.

Once we have it, we pass it on to the update statement.

Together with atomic updates, that gives strong guarantees to our MongoDB event store. Of course, we should consider adding retries in case of concurrency error, but you can read about it in my other article.

Tradeoffs

While we showed that it’s perfectly doable to build an event store on top of MongoDB, there are some caveats worth highlighting.

First, MongoDB’s atomic updates and transactions work best on single documents, so if you rely on cross-document or multi-stream consistency, you’ll need to coordinate or accept weaker guarantees carefully. That’s not a huge issue for appends, as we should keep our event streams denormalised and focused on business process. See more in my talk:

Still, that becomes an issue for updating read models: no global ordering out of the box. Our single-document-per-stream approach ensures ordering within one stream, but nothing enforces a globally monotonic event sequence across all streams. You could try to simulate a global ordering by tapping into MongoDB’s oplog or Change Streams, but it’s still more complex (and less straightforward) than dedicated event stores that offer a built-in global offset.

In Emmett, we provided the option to have inline projections stored in the stream document, together with events. That allows atomic updates in the same operation as events append. I’ll expand on it in the follow-up post.

There’s also a valid concern raised by Robert Kawecki. The maximum size of the MongoDB document is 16MB. This is, actually, more than the raw JSON size, as BSON used in MongoDB is a binary format. If we keep our streams short, that should be sufficient for most cases. The stream per document will need to be chunked into multiple documents. The proposed structure could be expanded to include chunk numbers and setting a unique index on streamName and chunk number. That will allow moving events to the new document once the size is reached. We may also need to use snapshots. I’ll cover this 2nd-day issue in the dedicated blog article.

Lastly, from a durability perspective, past Jepsen tests have flagged edge cases under certain configurations and failover scenarios, indicating you’ll want to pay close attention to cluster setup and operational practices. None of that rules MongoDB out—it just means that if you absolutely need a strict global ordering or bulletproof multi-document consistency, a specialized event store (or a fully ACID relational system) might be a better fit.

Otherwise, if per-stream concurrency and ordering suffice, this MongoDB-based strategy can still be a solid choice.

TLDR

I’d still use PostgreSQL event store as the default choice, but MongoDB implementation appeared surprisingly good.

Despite MongoDB limitations, many real-world applications do not require global event ordering or the highest levels of transactional consistency. If your workloads are primarily focused on per-stream ordering and optimistic concurrency within a single document, the described MongoDB strategy can be surprisingly effective.

You can achieve a decent event store solution without deploying an entirely new infrastructure by treating each stream as a document, using upserts for atomic event append operations, and relying on MongoDB’s flexible schema.

If MongoDB is already part of your tech stack and the outlined constraints are not deal-breakers, this approach can deliver a pragmatic, production-friendly solution that balances performance, simplicity, and developer familiarity.

I hope that this guide showed you how to design a proper event store on top of a document and a key-value database. Still, even with that it requires additional work to make it properly performant, reliable and accessible.

If you’re in the Node.js space, don’t reinvent the wheel; just run:

npm install @event-driven-io/emmett-mongodb

And use it.

You can also check the fully working WebApi sample using it. You’ll find it here.

If you’re not in Node.js land and want me to help you build it [email protected]">contact me, I’ll try to help. You can have a look at Emmet’s implementation.

Please also share with me your thoughts, and share the article with your friends if you enjoyed it! That’d be much appreciated, as writing it took me a hellova time.

If you liked this article, you may also find those interesting:

Cheers!

Oskar

Idempotent Command Handling

“Is your command handling idempotent?”

Sounds like a douchebag question to ask. Actually, that’s not a question but a statement that we want to sound smart.

Maybe a definition from Wikipedia could help?

Idempotence (UK: /ˌɪdɛmˈpoʊtəns/, US: /ˈaɪdəm-/) is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application.

Meh, even more douchy.

Yet, if we frame it as:

“How do you handle business logic that should only be run once when the record doesn’t exist? For instance: creating a user on the first call or ensuring that we’re checking in guests to the hotel only once.

Or:

Should checking out a guest fail or succeed if the guest was already checked out?

Now we’re talking! We’re bringing the discussion back. We can reason and discuss specific cases.

Let’s say that we have the Guest Stay finances workflow explained in the How TypeScript can help in modelling business workflows. It starts by checking in guests; then we record charges (e.g. night stay, beer at the bar, massage at SPA) and pay for them. Guests can check out if our balance is settled (so the difference between charges and payments equals zero).

Now, we could start with the naive implementation that throws exceptions when the business rule is not fulfilled. For check-in, this could look as follows:

function checkIn (
  command: CheckIn,
  state: GuestStayAccount,
): GuestCheckedIn {
  const { data: { guestId, roomId }, metadata } = command;

  if (state.status === 'CheckedIn')
    throw new IllegalStateError(`Guest is already checked-in!`);

  if (state.status === 'CheckedOut')
    throw new IllegalStateError(`Guest account is already checked out`);

  const now = metadata?.now ?? new Date();

  return {
    type: 'GuestCheckedIn',
    data: {
      guestId,
      roomId,
      guestStayAccountId: toGuestStayAccountId(guestId, roomId, now),
      checkedInAt: now,
    },
  };
};

I designed my model to express possible states. The guest state can be either not existing, checked in or checked out. Essentially it’s a state machine.

type NotExisting = { status: 'NotExisting' };

type CheckedIn = { status: 'CheckedIn'; balance: number };

type CheckedOut = { status: 'CheckedOut' };

type GuestStayAccount = NotExisting | CheckedIn | CheckedOut;

const initialState = (): GuestStayAccount => ({
  status: 'NotExisting',
});

Thanks to that, I can make the explicit check, ensuring that I assume I can perform the check-in only when the guest’s stay does not exist. Otherwise, I’m throwing exceptions with meaningful messages.

That could work for the basic scenario. Yet, even if we’re designing the business logic or backend code, we should always consider user experience. Throwing exceptions helps us ensure that we won’t end up in the wrong state, which is a minimum effort we should make. Yet, undoubtedly, it doesn’t provide a good user experience. Some can say:

No worries, we’ll return the Result instead of throwing an exception!

That’s sweet, but for me, it’s Potato vs Potahto.

Of course, I understand the principles of not driving our logic by exceptions, but we can map exceptions and results to specific HTTP statuses, pop-ups with an error message, etc.

Whether we throw exceptions or return Result.Error, it’ll be the same (bad) user experience. The flow is stopped with an error. It’s “just” syntactic sugar we prefer.

If we haven’t done that yet, it’s about time to ask a business expert what should happen if someone tries to check in twice. If you think that this can’t ever happen, then think about scenarios like:

the hotel clerk clicks check-in twice,
someone uses the check-in booth,
we have a quick check-in workflow, where automatically, a guest account is created, a confirmed reservation, and checked in,
we’re integrating with external systems that can send us a web-hook multiple times,
the process is asynchronous or initiated in other modules, and queues can retry command delivery.

It may appear that it’d be a better experience to succeed silently if the check-in has already been made. So, we accept the command and say, “Yeah, fine.” How to do it?

The easiest option is to just remove if statement:

if (state.status === 'CheckedIn')
  throw new IllegalStateError(`Guest is already checked-in!`);

Still, if we just do that, then business logic would go forward, and it’d return GuestCheckedIn again. It’d be stored, and that’s not necessarily where we’d like to end up. We’ve already checked in, and we don’t care how many more times someone tries to do it again. What other options do we have, then?

We should have a way to say that we’re ignoring the command. There are multiple options to achieve it, for instance, by returning:

null,
result type informing that we’re ignoring processing,
explicit event telling that error condition was achieved,
empty array or events,
another creative way you came up with.

Again, it’s a bit potato vs potahto discussion, highly dependent on your preferences. Let me go through mine. Returning null, in this case, could be meaningless for other people unless that’s our general convention. Returning explicit events is fair but sometimes a bit too heavy; we’ll discuss that later in more detail. Result type informing that we’re ignoring processing would be best if we’re not doing Event Sourcing, as we need to know whether we should update the state with the new one. As I’m doing Event Sourcing here, my safe default would be an empty array of events. In Event Sourcing, logic should return one or an array of events, so this should not introduce additional complexity.

An updated method will look as follows:

function checkIn (
  { data: { guestId, roomId }, metadata }: CheckIn,
  state: GuestStayAccount,
): GuestCheckedIn | [] {
  if (state.status === 'CheckedIn')
    return [];

  if (state.status === 'CheckedOut')
    throw new IllegalStateError(`Guest account is already checked out`);

  const now = metadata?.now ?? new Date();

  return {
    type: 'GuestCheckedIn',
    data: {
      guestId,
      roomId,
      guestStayAccountId: toGuestStayAccountId(guestId, roomId, now),
      checkedInAt: now,
    },
  };
};

A simple change, but it makes our processing more explicit. If we prefer, we can still throw an exception if the guest was checked out. That can mean something is fishy, and we may still want to throw an exception in this case. Still, we should always discuss that with business. It’s fine to mix multiple strategies for various invariants.

Command Handling infrastructure

Such change, of course, has to be reflected in our infrastructure. In Event Sourcing the flow looks as follow:

Get the stream events and build the state from them.
Run the business logic based on the state and passed command.
Store new events in the same stream you’ve read.

Now, between 2 and 3, we should add an additional step checking if any changes have been made, so in our case, an empty array was returned. The pseudo-code for a generic command handling was returned could look as follows:

// 1. Aggregate the stream
const aggregationResult = await eventStore.aggregateStream(streamName, {
  evolve,
  initialState,
});

// Use the aggregate state
const state = aggregationResult.state;
const currentStreamVersion = aggregationResult.currentStreamVersion;

// 2. Run business logic
const result = handle(state);

const newEvents = Array.isArray(result) ? result : [result];

// 3. Check if any changes have been made
if (newEvents.length === 0) {
  return {
    newEvents: [],
    newState: state,
    nextExpectedStreamVersion: currentStreamVersion,
    createdNewStream: false,
  };
}

// 4. Append result to the stream
const appendResult = await eventStore.appendToStream(
  streamName,
  newEvents,
  {
     expectedStreamVersion: currentStreamVersion,
  },
);

// 5. Return result with updated state
return {
  ...appendResult,
  newEvents,
  newState: newEvents.reduce(evolve, state),
};

If you’re not doing Event Sourcing then it’s the same stuff; it could look as follows:

// 1. Aggregate the stream
const findResult = await repository.findById(recordId)_;

// Use the aggregate state
const state = findResult.state ?? initialState();
const currentStreamVersion = findResult.currentVersion;

// 2. Run business logic
const result = handle(state);

// 3. Check if any changes have been made
if (isIgnoredResult(result)) {
  return {
    newState: state,
    nextExpectedVersion: currentVersion,
    createdNewRecord: false,
  };
}

// 4. Append result to the stream
const storeResult = await repository.store(
  recordId,
  result,
  {
     expectedVersion: currentVersion,
  },
);

// 5. Return result with updated state
return {
  ...storeResult,
  newState: result,
};

Essentially, we’re loading the current state. If it doesn’t exist, we’re setting the default initial state. For Event Stream, not existing will mean an empty events array; for the classical approach, it can be null. Thanks to assigning explicit initial state. We’re making things explicit. Thanks to that, we can check whether the record exists in our business logic.

We also get the current version of the record and use it as the expected version when appending event(s)/performing the state update. We’re using Optimistic Concurrency here. We’re telling our storage engine to only accept the update if the stream/record version in the database is the same as the expected one. Otherwise, it should reject the update (usually by throwing an exception). Thanks to that, we’re ensuring that if another change happens in parallel, our update won’t be accepted. All mature storage engines support Optimistic Concurrency. If they don’t, then run away; they’re not mature!

Idempotency and Optimistic Concurrency

That’s cool stuff, but we already said that throwing exceptions is not a perfect solution. What’s more, the change happening in parallel doesn’t have to be conflicting. In our current business rules, the only conflicting change is when a guest is checked out. If we’re recording charges or payments, then the guest stay is still running, and we should safely succeed. How to handle that?

The solution for that is the retry policy. We could wrap our command handling in the retry that would check if concurrency error was thrown.

Let’s say that we have the following wrapper for the command handling:

const CommandHandler =
  <State, EventType extends Event>(
    evolve: (state: State, event: StreamEvent) => State;
    initialState: () => State;
  ) =>
  async (
    eventStore: EventStore,
    streamName: string,
    handle: (
      state: State,
    ) => StreamEvent[]
  ): Promise<CommandHandlerResult> => {
    // (...) Command handling code you saw above
  };

We can define it for our guest stay as:

const handle = CommandHandler({ evolve, initialState });

And use it in our endpoint for checking in guest as:

export const guestStayAccountsApi =
  (
    eventStore: EventStore,
  ): WebApiSetup =>
  router.post(
    '/guests/:guestId/stays/:roomId',
    on(async (request: CheckInRequest) => {
      // 1. Prepare command 
      const guestStayAccountId = toGuestStayAccountId(
        request.params.guestId, 
        request.params.roomId
      );

      const command: CheckIn = {
        type: 'CheckIn',
        data: {
          guestId: request.params.guestId,
           roomId: request.params.roomId,
         },
        metadata: { now: new Date() },
       };

      // 2. Run command handling logic
      await handle(eventStore, guestStayAccountId, (state) =>
        checkIn(command, state),
      );
        
      // 3. If no error was thrown, return success
      return Created({
        url: `/guests/${guestId}/stays/${roomId}/periods/${formatDateToUtcYYYYMMDD(now)}`,
      });
    }),
  );

With this approach, we have a handle method, which is an abstraction for command handling. Thanks to the repeatable pattern of handling business logic, we can make it generic, leaving business logic focused and explicit. As long it returns events, then we’re good.

Getting back to the retries. As our command handler is just a function, we could decorate it with retry code.

Let’s start by defining the general retry policy. I’m using the async-retry npm package. If you’re in C#, check Polly, and in Java, check Spring Retry. Each environment has a recommended package to handle that.

The only thing I added is a bit easier way to pass function to check if we should retry on a specfic error.

import retry from 'async-retry';

export type AsyncRetryOptions = retry.Options & {
  shouldRetryError?: (error: unknown) => boolean;
};

export const NoRetries: AsyncRetryOptions = { retries: 0 };

export const asyncRetry = async <T>(
  fn: () => Promise<T>,
  opts?: AsyncRetryOptions,
): Promise<T> => {
  if (opts === undefined || opts.retries === 0) return fn();

  return retry(
    async (bail) => {
      try {
        return await fn();
      } catch (error) {
        if (opts?.shouldRetryError && !opts.shouldRetryError(error)) {
          bail(error as Error);
        }
        throw error;
      }
    },
    opts ?? NoRetries,
  );
};

Let’s also define a check for concurrency error. It could look as:

const isExpectedVersionConflictError = (
  error: unknown,
): boolean =>
  error instanceof ExpectedVersionConflictError;

Depending on your storage engine this function will differ, for instance for PostgreSQL change it could be:

const isExpectedVersionConflictError = (
  error: unknown,
): boolean =>
  error instanceof Error && 'code' in error && error.code === '23505';

Having that, we can adjust our command handler to:

export const defaultRetryOptions: AsyncRetryOptions =
  {
    retries: 3,
    minTimeout: 100,
    factor: 1.5,
    shouldRetryError: isExpectedVersionConflictError,
  };

const CommandHandler =
  <State, EventType extends Event>(
    evolve: (state: State, event: StreamEvent) => State;
    initialState: () => State;
  ) =>
  async (
    eventStore: EventStore,
    streamName: string,
    handle: (
      state: State,
    ) => StreamEvent[],
    options?: { retry: AsyncRetryOptions }
  ): Promise<CommandHandlerResult> => 
    asyncRetry(() =>{
      // (...) Command handling code you saw before
    },
    options?.retry ?? defaultRetryOptions,
  );

No other change is needed; our logic can remain the same. Of course, we can make it more sophisticated or default to no retries and only make retries in specific cases, but I think you get the idea.

It’s essential to the business logic and the whole command handling flow. Without reading again, we won’t get the new state and the current stream version, and the update will fail.

Handling duplicates

Retries are nice, but how do we handle duplicates? Won’t retries generate new records? It may happen if we were always generating new random IDs for our guest stay. For instance:

import { randomUUID } from 'node:crypto';

const guestStayAccountId = randomUUID();

const command: CheckIn = {
  type: 'CheckIn',
  data: {
    guestId: request.params.guestId,
    roomId: request.params.roomId,
  },
  metadata: { now: new Date() },
};

await handle(eventStore, guestStayAccountId, (state) =>
  checkIn(command, state),
);

But that’s not what we did, we did:

const guestStayAccountId =  toGuestStayAccountId(
  request.params.guestId, 
  request.params.roomId
);

where toGuestStayAccountId creates a predictable id based on provided data:

const toGuestStayAccountId = (
  guestId: string,
  roomId: string,
) => `guest_stay_account-${guestId}:${roomId}`;

The stay can be represented by guest and room (it could also be reservation ID or whatever is predictable).

Thanks to that, no matter how many times we retry, we’ll always be accessing the same record; the combination of business logic, transactions, and optimistic concurrency will ensure that we won’t end up in the wrong state.

Idempotency handling in updates

We can also apply the same pattern to the update operation, let’s say we have the following code for recording charge:

function recordCharge (
  { data: { guestStayAccountId, chargeId, amount }, metadata }: RecordCharge,
  state: GuestStayAccount,
): ChargeRecorded {
  assertIsCheckedIn(state);

  return {
    type: 'ChargeRecorded',
    data: {
      chargeId,
      guestStayAccountId,
      amount: amount,
      recordedAt: metadata?.now ?? new Date(),
    },
  };
};

const assertIsCheckedIn = (state: GuestStayAccount): state is CheckedIn => {
  if (state.status === 'NotExisting')
    throw new IllegalStateError(`Guest account doesn't exist!`);

  if (state.status === 'CheckedOut')
    throw new IllegalStateError(`Guest account is already checked out);

  return true;
};

This function is not idempotent. Of course, we will ensure that we’re not recording charges for not existing or checked-out accounts, but this code won’t help us for not recording double charges. Doubled charges are not something we’d like to have in the financial module. Or anywhere.

To detect it, we’d need information about the recorded transaction ids. Then, we could check if the following transaction hasn’t been handled already.

Let’s change our state definition to include transaction ids:

type NotExisting = { status: 'NotExisting' };

type CheckedIn = { 
  status: 'CheckedIn'; 
  balance: number;
  // new property
  transactionIds: string[];
};

type CheckedOut = { status: 'CheckedOut' };

type GuestStayAccount = NotExisting | CheckedIn | CheckedOut;

const initialState = (): GuestStayAccount => ({
  status: 'NotExisting',
});

We also need to change our evolve function, which we’re using to define how we build our state from events.

function evolve (
  state: GuestStayAccount,
  { type, data: event }: GuestStayAccountEvent,
): GuestStayAccount => {
  switch (type) {
    case 'GuestCheckedIn': {
      return state.status === 'NotExisting'
        ? { status: 'CheckedIn', balance: 0 }
        : state;
    }
    case 'ChargeRecorded': {
      return state.status === 'CheckedIn'
        ? {
            ...state,
            balance: state.balance - event.amount, 
            // new line
            transactionIds: [ ...state.transactionIds, event.chargeId ],
          }
        : state;
    }
    case 'PaymentRecorded': {
      return state.status === 'CheckedIn'
        ? {
            ...state,
            balance: state.balance + event.amount,
            // new line
            transactionIds: [ ...state.transactionIds, event.paymentId ],
          }
        : state;
    }
    case 'GuestCheckedOut': {
      return state.status === 'CheckedIn' ? { status: 'CheckedOut' } : state;
    }
    case 'GuestCheckoutFailed': {
      return state;
    }
    default: {
      const _notExistingEventType: never = type;
      return state;
    }
  }
};

For the classical approach, you’d need to extend your table schema.

And now, the question? How can we get the charge id? The sender should generate it. Who’s “the sender”? For instance:

web page,
mobile application,
other module,
external payment gateway.

We’re good as long as the same charge has the same charge id in the command data. If we’re retrying operations from our web page, we should ensure that we’re not recording the new charge id on each retry.

The updated logic handling idempotency correctly can look as follows:

function recordCharge (
  { data: { guestStayAccountId, chargeId, amount }, metadata }: RecordCharge,
  state: GuestStayAccount,
): ChargeRecorded | [] {
  assertIsCheckedIn(state);

  // idempotence check
  if(state.transactionIds.includes(chargeId))
    return [];

  return {
    type: 'ChargeRecorded',
    data: {
      chargeId,
      guestStayAccountId,
      amount: amount,
      recordedAt: metadata?.now ?? new Date(),
    },
  };
};

Look Ma’ No Exceptions!

Ok, but what about those errors? Many people don’t like exceptions, and that’s for good reasons. Some languages even don’t allow us to return them. How would I model errors in general?

It depends, I see a few ways:

if they’re “just” errors and I’m in an environment like C# or Java, then I’d throw an exception and map it to the status. For functional environment or Go, Rust, I’d return the result. I’m not a big fan of Railway Oriented Programming in languages that don’t have the pipeline operator. Without it, it’s not ergonomic.
The same for events that are part of the asynchronous workflow.
if they represent idempotent handling, then return an empty array or explicitly ignore the object.
if they’re important to business, I’d model them as events and store them in the stream.

Let’s discuss the last option, and let’s say that knowing that the charge was rejected is important information for businesses, as they’d like to track those failure scenarios. They can be part of antifraud detection or similar processes.

Having that we could define additional event type:

type ChargeFailed = Event<
  'ChargeFailed',
  {
    chargeId: string;
    guestStayAccountId: string;
    reason: 'NotCheckedIn' | 'AlreadyCheckedOut';
    failedAt: Date;
  }
>;

type PaymentFailed = Event<
  'PaymentFailed',
  {
    paymentId: string;
    guestStayAccountId: string;
    reason: 'NotCheckedIn' | 'AlreadyCheckedOut';
    failedAt: Date;
  }
>;

// (...) other event types

export type GuestStayAccountEvent =
  | GuestCheckedIn
  | ChargeRecorded
  | ChargeFailed // <= new
  | PaymentRecorded
  | PaymentFailed // <= new
  | GuestCheckedOut
  | GuestCheckoutFailed;

Instead of throwing exception, we could return now an event:

function recordCharge (
  { data: { guestStayAccountId, chargeId, amount }, metadata }: RecordCharge,
  state: GuestStayAccount,
): ChargeRecorded | ChargeFailed | [] {
  // failure
  if (state.status !== 'CheckedIn') {
    return {
      type: 'ChargeFailed',
      data: {
        chargeId,
        guestStayAccountId,
        reason: state.status === 'NotExisting'?
          'NotCheckedIn' : 'AlreadyCheckedOut'
        failedAt: metadata?.now ?? new Date(),
      },
  }

  if(state.transactionIds.includes(chargeId))
    return [];

  return {
    type: 'ChargeRecorded',
    data: {
      chargeId,
      guestStayAccountId,
      amount: amount,
      recordedAt: metadata?.now ?? new Date(),
    },
  };
};

By default, this event will be stored in the stream. Still, even if we don’t want to store it in the stream, we can decide not to throw exceptions but model all edge cases as events. Then, we can decide on the application layer whether to store it or not. Maybe we would like to just log it.

We can orchestrate it like this:

  await handle(
    eventStore,  
    guestStayAccountId, 
    (state) => {
      const result = recordCharge(command, state);

      // check if it's failed event
      if(result !== [] && result.type === 'ChargeFailed') {
        console.log(result);
        // let's ignore it, we could also throw error here
        return [];
      }

      return result;
    }
);

Generic idempotency handling

I think business logic is the proper place to check the business rules, and not handling doubled charges is actually a business rule. Still, many people prefer to handle it generically. Let’s discuss how to do it.

We could store the handled command ids in the external store and run only business logic if it wasn’t already handled. The store could be defined as:

type IdempotencyKeyStore = {
  tryLock: (
    key: string,
    options?: { timeoutInMs?: number },
  ) => boolean | Promise<boolean>;
  accept: (key: string) => void | Promise<void>;
  release: (key: string) => void | Promise<void>;
};

The store has the following methods:

tryLock - that tries to lock the specified key for a certain period of time. Returns true if the lock was acquired and false otherwise.
accept - accepts the key lock after successful handling.
release - releases the lock in case of handling failure.

What’s most important is that those operations should be thread-safe and atomic. Without that, we may face race conditions. The safest safest option is to use a distributed lock, e.g., Redis or a relational database.

Having that, we can define our wrapper as:

type IdempotentOptions<T> = {
  store: IdempotencyKeyStore;
  key: string;
  defaultResult: () => T | Promise<T>;
  timeoutInMs?: number;
};

async function idempotent <T>(
  fn: () => Promise<T>,
  options: IdempotentOptions<T>,
): Promise<T> {
  const { key, timeoutInMs, defaultResult, store } = options;

  //Attempt to acquire the lock for the specified key
  const appended = await store.tryLock(key, { timeoutInMs });

  // If it was used, return the default result
  if (!appended) return await defaultResult();

  try {
    // run handler
    const result = await fn();

    // Mark operation as successful
    await store.accept(key);

    return result;
  } catch (error) {
    // Release key if it was used already
    await store.release(key);
    throw error;
  }
};

We’re passing the function to wrap together with options:

idempotency store,
idempotency key,
default result if the command was already handled,
optional lock timeout.

Having that, we:

Attempt to acquire the lock for the specified key.
If the attempt wasn’t successful return the default result.
If we acquired lock, then run the business logic.
If it fails, release the lock. We don’t want to lock the key, let someone fix the state condition, and retry later.
If it was successful, accept the lock and return the result.

The example lock store based on Redis could look as follows:

const defaultTryLockPeriod = 1; // 1 second
const defaultLockPeriod = 86400; // 1 day

const redisStore: IdempotencyKeyStore = {
  tryLock: async ((
    key: string,
    options?: { timeoutInMs?: number },
  ) => {
    const result = await redis.set(key, "in-progress", "NX", "EX", options?.timeoutInMs ?? defaultTryLockPeriod);
    return result === "OK";
  },
  accept: async (key: string) => {
    // Update the key to "completed" status with a longer TTL
    await redis.set(key, "completed", "XX", "EX", defaultLockPeriod); // 1 day TTL
  },
  release: (key: string) => 
    redis.del(key),
};

Having that, we could decorate our command handler with idempotent handling:

const CommandHandler =
  <State, EventType extends Event>(
    evolve: (state: State, event: StreamEvent) => State;
    initialState: () => State;
  ) =>
  async (
    idempotencyKeyStore: IdempotencyKeyStore,
    eventStore: EventStore,
    streamName: string,
    handle: (
      state: State,
    ) => StreamEvent[],
    options: { retry: AsyncRetryOptions, idempotencyKey: string }
  ): Promise<CommandHandlerResult> => 
    idempotent(() => {
      asyncRetry(() =>{
        // (...) Command handling code you saw before
        },
        options?.retry ?? defaultRetryOptions,
      ),
      {  
        idempotencyKeyStore: IdempotencyKeyStore;
        key: options.IdempotencyKey,
        defaultResult: async () => {
          const { state, currentStreamVersion } = await eventStore.aggregateStream(streamName, {
            evolve,
            initialState,
          });

          return {
            newEvents: [],
            newState: state,
            nextExpectedStreamVersion: currentStreamVersion,
            createdNewStream: false,
          };
        }
      }
    );

Nothing fancy besides getting the current state in case we already handled this command.

The endpoint will look as follows:

export const guestStayAccountsApi =
  (
    eventStore: EventStore,
    idempotencyKeyStore: IdempotencyKeyStore,
  ): WebApiSetup =>
  router.post(
    '/guests/:guestId/stays/:roomId/periods/:checkInDate/charges',
    on(async (request: RecordChargeRequest) => {
      const guestStayAccountId = parseGuestStayAccountId(request.params);

      const command: RecordCharge = {
        type: 'RecordCharge',
        data: {
          chargeId: request.body.chargeId,
          guestStayAccountId,
          amount: Number(request.body.amount),
        },
        metadata: { now: new Date() },
      };

      await handle(
        eventStore, 
        idempotencyKeyStore, 
        guestStayAccountId, 
        (state) => recordCharge(command, state),
        { idempotencyKey: command.chargeId }
      );

      return NoContent();
    }),
  );

And guess what? That’s also a way how external API providers are handling it:

Of course, it’s up to you how you compose; you could even make it more generic and part of your HTTP middleware based on the Idempotency-Key header. The choice is yours, but the logic will be the same.

TLDR

Idempotency isn’t just a buzzword—it’s a practical way to handle operations that should only happen once, no matter how many times they’re retried. Whether it’s checking in a guest, recording charges, or processing API calls, the key is ensuring consistency without breaking the flow. By designing systems that handle duplicates gracefully, use optimistic concurrency, or apply idempotency keys, we can avoid annoying edge cases and give users a better experience. It’s not about exceptions versus results—it’s about designing systems that actually work in the real world.

We went through the various ways to handle it. Both explicitly and generically. From building predictable state machines, handling it in business logic, to leveraging Optimistic Concurrency, retries and distributed locks.

As tempting as it is to use generic handling, as you saw above, idempotency handling is highly dependent on the business rules; those rules tend to change. If we use generic handling, we’re always adding additional overhead, even if most of our applications rarely have idempotency issues. That’s something to consider, as it can also increase costs in cloud environments.

Whether you’re processing financial transactions, integrating with external APIs, or handling user commands, the principles remain the same: make the system resilient, maintain consistency, and prioritize user experience.

Because, in the end, it’s not about whether you throw exceptions or return results—it’s about understanding the nuances of the business domain and designing solutions that respect those rules.

So, how are you making your command handling idempotent? If you haven’t thought about it yet, maybe now’s the time.

If you need to help in applying those practices, I’m here to help. Check my training page. A workshop is the most effective way to jump-start, I’m also doing consulting and mentoring, [email protected]">drop me an e-mail! and we’ll find the best way to help you and your team.

And guess what, all of those you saw in the article, is already available in Emmett! Come join us, we have cookies! Join our Discord channel and discuss further questions!

Cheers!

Oskar

Bootstrapping CRUD with Pongo

The leitmotif of this blog is the event-driven approach. I truly believe that it’s a way to keep our applications closer to business. By doing so, we can better reflect the business process in our system design and code. And that’s great, as it brings multiple benefits: easier evolution, resiliency, and better managed and traced workflows.

Still, sometimes you don’t need all of that.

Sometimes you just need a bag for data, or the CRUD as most of us prefer to call it.

CRUD comes from the common set of operations we perform on our data: Create, Read, Update, and Delete. It’s an implementation style suited for Content Management Systems. The responsibility is to store and manage data. Of course, we wrap that with basic validation, consistency rules, and authorisation. We run simple business logic or enrich data with information available only on the backend. You just put some data and retrieve it. What you put is what you get.

In CRUD, you don’t have specific behaviour. What’s more, from the system side, this data doesn’t have much business context. That’s why implementations are generic. It only has meaning for the user who puts this data inside and retrieves it. The user interprets it upon reading and makes decisions outside of the system based on it.

CRUD can also be a valid approach for proof of concepts. For instance, if we have a product idea, we bootstrap a basic application without an extensive set of business workflows, just basic data ingestion and visualisation, to ensure that there’s potential in this idea.

In general, there is nothing to be ashamed of when doing CRUD. It’s a valid implementation style but with limited use cases. If we choose it wisely, then it’s all fine.

We need also to remember that we should not set this decision in stone.

That’s why I like to match CRUD with CQRS. How come? Aren’t they contradicting? Not in my world!

CQRS stands for Command Query Responsibility Segregation. It’s a structural pattern. It tells us to slice our business application by the behaviour and then segregate them into two responsibilities:

Command handling - business logic that can change state but should not be returning business data,
Query handling - returns data but does not change the state (_“Asking a question should not change the answer”).

If those rules are fulfilled, our internal implementation can be CRUD, and we can apply CQRS principles. That’s essential for proof of concepts. We start with the simple, generic implementation and evolve it once new business requirements appear. As we already segregated our business functionality, we can adjust precisely the places that need to be changed. CRUD doesn’t have to be an unmaintainable amalgamate!

Still, no matter which way you choose, I believe that Pongo is a decent tool for CRUDs. My way or highway? Nah, I won’t tell you how to live!

Why Pongo?

It’s a Node.js tool, so it’s a lightweight and accessible environment with a big, vibrant community. You should not have issues with potential hiring,
It runs on PostgreSQL, so it’s easy operational-wise, and it’s easy to set up your hosting in cloud providers, on-premise or services like Neon, Supabase, Vercel, etc.
MongoDB-like API is easy to learn and is well-known to many of us.
The document approach and denormalised data help make it easier to set up your data. It’s well suited for the CRUD model, where you want to store and retrieve data in the same form as you put it,
Pongo will get you covered with built-in migrations, so there is no need to care a lot about the database schema.

All of that makes a good combination for fast bootstrapping.

For basics, you can check previous articles:

And documentation.

So pardon me, I won’t repeat myself, especially since you might have read them already. Let me show you some special sauce: Pongo command handling.

The typical flow in CRUD for updating records looks as follows:

Validate the incoming request.
Read the current state.
Do necessary validation based on it.
Run business logic.
Update the state.
Store it.

Similarly, for deletion:

Validate the incoming request.
You read the current state.
Do the necessary validation, checking if you can delete it.
Delete it.

For creation, it’s even simpler; you just:

Validate the incoming request.
Generate the new state based on request data.
Store it.

If we’d like to be sneaky, we could wrap it in a single flow that covers all of those cases.

Validate the incoming request.
Read the current state.
If a state exists and you want to update it, do the necessary validation based on it.
Run business logic. Returning new state, updated state, or null if you want to delete the state.
Depending on the result of the business logic:

Create if state didn’t exist and result state is not null,
Update if state existed and result state is different and not null,
Delete if state existed and result state is null,
Do nothing otherwise, and safely handle idempotency.

And guess what? That’s precisely what Pongo can do for you.

Let’s use our favourite Shopping Cart example. Types for it could look as follows:

interface ProductItem {
  productId: string;
  quantity: number;
}

type PricedProductItem = ProductItem & {
  unitPrice: number;
};

type ShoppingCart = {
  _id: string;
  clientId: string;
  productItems: PricedProductItem[];
  productItemsCount: number;
  totalAmount: number;
  status: 'Opened' | 'Confirmed';
  openedAt: Date;
  confirmedAt?: Date | undefined;
  cancelledAt?: Date | undefined;
};

We could define the basic set of operations:

adding a product item - that’s possible to not confirmed or non-existing shopping cart,
removing a product item - that’s possible when we have enough products already in the not confirmed shopping cart,
confirming non-empty shopping cart (we can confirm it twice, handling idempotence safely),
cancelling opened shopping cart (we can cancel it twice, handling idempotence safely).

The business logic could look as follows:

Adding product item:

const addProductItem = (
  command: {
    clientId: string;
    shoppingCartId: string;
    productItem: PricedProductItem;
    now: Date;
 },
  state: ShoppingCart | null,
): ShoppingCart => {
  if (state && state.status === 'Confirmed')
    throw new Error('Shopping Cart already closed');

  const { shoppingCartId, clientId, productItem, now } = command;

  const shoppingCart: ShoppingCart = state ?? {
    _id: shoppingCartId,
    clientId,
    openedAt: now,
    status: 'Opened',
    productItems: [],
    totalAmount: 0,
    productItemsCount: 0,
  };

  const currentProductItem = shoppingCart.productItems.find(
    (pi) =>
          pi.productId === productItem.productId &&
          pi.unitPrice === productItem.unitPrice,
    );

  if (currentProductItem !== undefined) {
    currentProductItem.quantity += productItem.quantity;
  } else {
    shoppingCart.productItems.push(productItem);
  }

  shoppingCart.totalAmount += productItem.unitPrice * productItem.quantity;
  shoppingCart.productItemsCount += productItem.quantity;

  return shoppingCart;
};

Removing product item:

const removeProductItem = (
  command: {
    productItem: PricedProductItem;
    now: Date;
 },
  state: ShoppingCart | null,
): ShoppingCart => {
  if (state === null || state.status !== 'Opened')
    throw new Error('Shopping Cart is not opened');

  const { productItem } = command;

  const currentProductItem = state.productItems.find(
    (pi) =>
          pi.productId === productItem.productId &&
          pi.unitPrice === productItem.unitPrice,
    );

  if (
    currentProductItem === undefined ||
    currentProductItem.quantity < productItem.quantity
  ) {
    throw new Error('Not enough products in shopping carts');
  }

  state.totalAmount -= productItem.unitPrice * productItem.quantity;
  state.productItemsCount -= productItem.quantity;

  return state;
};

Confirming

const confirm = (
  command: {
    now: Date;
 },
  state: ShoppingCart | null,
): ShoppingCart => {
  if (state === null) throw new Error('Shopping Cart is not opened');

  if (state.status === 'Confirmed') return state;

  if (state.productItems.length === 0)
    throw new Error('Shopping Cart is empty');

  const { now } = command;

  state.status = 'Confirmed';
  state.confirmedAt = now;

  return state;
};

Cancelling:

const cancel = (state: ShoppingCart | null): ShoppingCart | null => {
  if (state != null && state.status === 'Confirmed')
    throw new Error('Cannot cancel confirmed Shopping Cart');
  return null;
};

Then we can make a basic Pongo setup:

import { pongoClient } from "@event-driven-io/pongo";

const connectionString =
  "postgresql://dbuser:[email protected]:3211/mydb";

const pongo = pongoClient(connectionString);
const pongoDb = pongo.db();

const shoppingCarts = pongoDb.collection<ShoppingCart>("shoppingCarts");

And plug our code into some request processing pipeline (e.g. web api):

type AddProductItemRequest = Request<
  { clientId: string; shoppingCartId: string },
  unknown,
  { productId: string; quantity: number }
>;

router.post(
  '/clients/:clientId/shopping-carts/current/product-items',
  on(async (request: AddProductItemRequest) => {
    const command = {
      clientId: request.params.clientId,
      productItem: {
        productId: request.body.productId,
        quantity: request.body.quantity),
        unitPrice: request.body.unitPrice,
      },
    };
    
    const result = await shoppingCarts.handle((state) =>
      addProductItem(command, state),
    );

    return Ok(result.document);
  })
);

For all other endpoints the code will look the accordingly. Isn’t that nice?

I think that’s a simple and quick way to sping up a new CRUD system, or bootstrap a new Proof of Concepts.

Thoughts?

Cheers!

Oskar

Running a regular SQL on Pongo documents

Have you heard someone say: “We’ll use this tool because it requires a long onboarding and lots of memorisation?”

You could have seen the decision as such in hindsight, but that doesn’t happen too often intentionally. Or at least, I hope.

The Tools I built need to share a common goal: they must be accessible and enable advanced users to customise them to their needs. Users should be able to start quickly. Best if they could reuse the learnings in other areas. Users should get a learning ladder. That’s why I optimise the API and tooling for the newbies.

That’s why I wrote Small rant about Software Design, which is being triggered by yet another tool that thinks too soon about the advanced user. Too often, we forget that we won’t get advanced users if the newbies don’t pass the very first steps of the learning ladder.

That’s why, in Pongo, I’m trying to join two accessibilities: muscle memory and the Node.js community by reusing the MongoDB client API and PostgreSQL operation easiness and familiarity. I think that enables me to ramp up quickly and deliver business value by deploying the first version of your software to production.

To use Pongo, you just need to install it:

$ npm install @event-driven-io/pongo

Have a PostgreSQL instance working somewhere (e.g. with Docker).

Connect the client to the instance using database:

const connectionString =
  'postgresql://postgres:postgres@localhost:5432/postgres';cockroachdb

const pongo = pongoClient(connectionString);
const pongoDb = pongo.db();

Define your document type:

type History = { street: string };
type Address = {
  city: string;
  street?: string;
  zip?: string;
  history?: History[];
};

type User = {
  _id?: string;
  name: string;
  age: number;
  address?: Address;
  tags?: string[];
};

And boom, you can access the collection with all the typing benefits:

const users = pongoDb.collection<User>();

You can even get the typed client that will make that even smoother.

Then you can insert some docs:

const docs = [
  {
     name: 'Anita',
     age: 25,
     address: { city: 'Wonderland', street: 'Main St'},
  },
  {
     name: 'Roger',
     age: 30,
     address: { city: 'Wonderland', street: 'Elm St'},
  },
  {
     name: 'Cruella',
     age: 35,
     address: { city: 'Dreamland', street: 'Oak St'},
  },
];

await users.insertMany(docs);

Then you can filter them:

const docs = await users.find({
  address: { city: 'Wonderland', street: 'Elm St'},
});

Update them:

await users.updateOne(
  { name: 'Anita'},
  { $inc: { age: 1 } };,
);

And all that jazz. MongoDB API has its quirks, but I believe it’s flexible and familiar enough to be easy to use. But…

That may be my point of view, and other people may disagree. Some people are familiar enough with PostgreSQL JSONB syntax to just use SQL.

The other reason can be that Pongo doesn’t support some MongoDB syntax yet. I’m continuously working on Pongo to deliver decent compatibility. Still, it may take some time to be fully aligned. I might never reach full compatibility, as the API can have numerous permutations that I didn’t predict. I don’t want to block anyone waiting for the new release with the fix. I think that SQL is a decent fallback.

That’s why I added a few options for using SQL in the Pongo API.

SQL queries

If you want to query Pongo with SQL, you can use an SQL helper. It’ll handle the parameter formatting, allowing you to customise it. For instance:

import { plainString, SQL } from '@event-driven-io/dumbo';

const wonderland = plainString('Wonderland');

const docs = await users.find(
  SQL`data @> '{"address":{"city":"${wonderland}"}}'`
);

In Pongo documents are stored as regular table, the structure looks as follows:

CREATE TABLE IF NOT EXISTS users (
    _id           TEXT           PRIMARY KEY, 
    data          JSONB          NOT NULL, 
    metadata      JSONB          NOT NULL     DEFAULT '{}',
    _version      BIGINT         NOT NULL     DEFAULT 1,
    _partition    TEXT           NOT NULL     DEFAULT 'png_global',
    _archived     BOOLEAN        NOT NULL     DEFAULT FALSE,
    _created      TIMESTAMPTZ    NOT NULL     DEFAULT now(),
    _updated      TIMESTAMPTZ    NOT NULL     DEFAULT now()
)

As you see, we’re querying the nested content of the JSON kept in the data column. Of course, you can access any other column or use data from other tables.

Your SQL will be placed in the WHERE statement.

If you’re wondering what’s @event-driven-io/dumbo, then it’s a shared package between Pongo and Emmett.

SQL updates

In the same way, you can also do updates. So instead of using the object-oriented Mongo API, you can do the follows:

const wonderland = plainString('Wonderland');
const oz = plainString('Oz');

const docs = await users.updateMany(
  SQL`data @> '{"address":{"city":"${wonderland}"}}'`,
  SQL`jsonb_set(data || '{"address":{"city":"${oz}"}}'::jsonb, '{age}', to_jsonb(COALESCE((data->>'age')::NUMERIC, 0) + '1'), true)`
);

The first SQL will be placed in the WHERE part, and the next one will be put into the UPDATE SET pipeline.

This will do the same as:

const docs = await users.find(
  { address: { city: 'Wonderland' } },
  {
    $set: { address: { city: 'Oz' } },
    $inc: { age: 1 }
  }
);

My point is for the MongoDB syntax. But I’ll let you decide what looks simpler to you.

Do whatever you want with SQL

You can also have the freehand mode, where you can draw anything or actually place any SQL. There you have it!

You can access it from both the Pongo db and the collection. For instance to query only document id and address you can do the following

const result = await pongoDb.sql.query(
  SQL`SELECT _id, data->'address' as address from users`
);

It will return:

[
  {
    "_id": "01928fee-d18b-711b-89d7-830ba98585e8",
    "address": {
      "city": "Dreamland",
      "street": "Oak St"
    }
  },
  {
    "_id": "01928fee-d18b-711b-89d7-755d49f389f6",
    "address": {
      "city": "Oz"
    }
  },
  {
    "_id": "01928fee-d18b-711b-89d7-790f5373cd10",
    "address": {
      "city": "Oz"
    }
  }
]

As you can imagine, this means that you can also join with other tables, in general, whatever the SQL syntax allows you.

You can also do any other operation that changes the data like:

const result = await pongoDb.sql.command(`SQL
  UPDATE users 
  SET 
    data = jsonb_set(data || '{"address":{"city":"Oz"}}'::jsonb, '{age}', to_jsonb(COALESCE((data->>'age')::NUMERIC, 0) + '1'), true) || jsonb_build_object('_version', (_version + 1)::text),
    _version = _version + 1
WHERE 
    data @> '{"address":{"city": "Oz"}}'`);

I think that’s pretty neat and powerful.

So, if you’re afraid of using Pongo, you’ll get a showstopper in compatibility; hey, I have you covered!

It’s the same if your favourite language is SQL; you can do whatever you want. Underneath, Pongo uses node-postgres with connection pooling enabled by default, so the additional performance issue should not hit you.

See also the videos where I showed all of that live.

Thoughts? Yay or Nay?

If you have more questions, join our Discord server and let’s tackle that together!

Cheers!

Oskar

Pongo behind the scenes

If you want to make God laugh, tell him about your plans.

My plans were simple: recharge during summer, take a break in July, and then return to the regular writing cadence. As you may have noticed, the last part didn’t go entirely as expected.

Okay, I also had another plan to bootstrap two projects:

Emmett,
Pongo.

It’s going okay in terms of the ongoing work, okayish in terms of adoption, and not so well yet in making it sustainable, so earning money from it to pay for the effort.

I also wanted to do less speaking, as it takes time, energy, and travelling, but then I got a few invitations to conferences and webinars that I couldn’t reject.

And that is how we got to Pongo Internals. That’s what I showed last week during YugabytDB Community Hours

And earlier on FerretDB Document Database Community.

It was tiring, as I was doing between flood, AxonIQ Conference and having Tonsillitis. But I’ve made it.

It’s heartwarming that tools like FerretDB and YugabyteDB are open to collaboration. I’m all for that, so I hope you’ll see more of it.

I showcased their recent additions to Pongo like:

fresh new Pongo shell for accessing and manipulating data without setting up the project,
filtering and updating Pongo documents with custom SQL,
built-in optimistic concurrency without the typical MongoDB retries,
typed client,
migrations.

I also shared how it’s working internally, so here’s a bit about the PostgreSQL JSONB magic and other decisions I’ve made.

Check out if you want to see how sausages are made. And drop your thoughts afterwards!

I think the Pongo concept is unique and can streamline the development of new products. If you’re interested in using it or sponsoring some work, [email protected]">contact me. I’m happy to jump on the call with you, showcase what’s already possible, and discuss how to help your project!

Pongo is a community project, and I believe it is a decent place to start your journey in Open Source. If you’d like to do it, join our Discord server, and I’ll help you jumpstart your contribution.

Read more in:

Cheers!

Oskar

Pongo gets strongly-typed client, migrations, and command line tooling

When you think upfront and want to make things right, there’s an interesting feedback loop. Quite often, things start to click, often in a surprising way.

I recently wrote on Architecture Weekly about my performance investigations in Emmett and Pongo. One of the conclusions was that schema needs to be generated upfront. Initially, it was generated once on the first call. That reduced boilerplate and was good enough for many cases but not for serverless.

To generate the Emmett PostgreSQL schema, I also wanted to be able to generate it for Pongo documents that I use for read models.

Pongo documents are stored in collections, and collections are regular (well, almost) PostgreSQL tables. So, to know what to generate, I had to add some way to know what collections I’ll have. There’s no such API in vanilla Mongo, so I have to add it. And that’s fine, as I want to make Pongo a superset of Mongo.

My initial idea was to provide a list of collections with names. This list could later contain a JSON schema definition, database indexes, etc. I also had to add an option to provide the database list. Pongo (just like Mongo) allows different dbs to be used.

The naive version could look like this:

const schema = [{ name: 'postgres', collections: ['users', 'orders'] }];

const client = pongoClient(postgresConnectionString, {
  schema: { definition: schema },
});

That would give me the information I need to set up users and order tables for collections in the default Postgres database. I could call it a day, but…

But then I thought, well, wouldn’t it be nice to generate a strongly typed TypeScript client? Having schema makes that possible! I “just” have to use a sneaky feature like Proxy type. So I did, and bang, here we are with the new release!

Strongly Typed Client

The API needs to be a bit more advanced, but I think it’s still straightforward and explicit. You need to define schema like:

type User = {
  _id?: string;
  name: string;
  age: number;
  address?: Address;
  tags?: string[];
};

type Customer = {
  _id?: string;
  name: string;
  address?: Address;
};

const schema = pongoSchema.client({
  database: pongoSchema.db({
    users: pongoSchema.collection<User>('users'),
    customers: pongoSchema.collection<Customer>('customers'),
  }),
});

And pass it to the client, getting the typed version.

const typedClient = pongoClient(postgresConnectionString, {
  schema: { definition: schema },
});
// 👇 client have the same database as we defined above, and the collection
const users = typedClient.database.users;

const doc: User = {
  _id: randomUUUID(),
  name: 'Anita',
  age: 25,
};
const inserted = await users.insertOne(doc);

// 👇 yup, the collection is fully typed!
const pongoDoc = await users.findOne({
  name: 'Anita'
});

I think that’s much better developer experience, than the Mongo API that tells us always to do calls like:

const db  = client.db('postgres');

const users = db.collection<User>();

Of course, if you like it, you can still use it. It’s great to have more options!

Internally, it generates the collections upfront and assigns them to the typed properties. If you want to know how that works internally, reply to this article, and I can explain how sausages are made in the follow-up!

Pongo gets command line

And we’re getting back to the announced synergy between making things right. Having schema also enabled upfront schema generation and even migration. To make that accessible, I added command line tooling.

You can either install it globally through:

npm install -g @event-driven-io/pongo

And run it with:

pongo

or without installing it globally by using npx

npx @event-driven-io/pongo

Cool, but what do you get from it?

Sample configuration generation

You can generate the sample config by calling:

npx @event-driven-io/pongo config sample --generate --file ./src/pongoConfig.ts --collection users --collection orders

This command will create a config file in the selected location with predefined users and orders collections. It’ll look as follows:

import { pongoSchema } from '@event-driven-io/pongo';

type User = { name: string; description: string; date: Date }
type Order = { name: string; description: string; date: Date }

export default {
  schema: pongoSchema.client({
    database: pongoSchema.db({
      users: pongoSchema.collection<User>('users'),
      orders: pongoSchema.collection<Order>('orders'),
    }),
  }),
};

Or just print it with:

npx @event-driven-io/pongo config sample --print --collection users --collection customers

Then, you can use adjust the generated typing and import it to your application.

import { pongoClient } from '@event-driven-io/pongo';
import config from './pongo.config';

const pongo = pongoClient(connectionString, {
  schema: { definition: config.schema },
});

Performing Database Migrations

Having the existing configuration file and command-line tooling opens even more options. You not only get a strongly typed client but also can generate and perform migrations based on it!

You can do it with new command line tooling:

npx @event-driven-io/pongo migrate run --config ./dist/pongoConfig.js \
--connectionString postgresql://postgres:postgres@localhost:5432/postgres

It’ll automatically run the migrations based on the defined collections.

If you’re unsure and don’t trust it fully, you can also add the —-dryRun parameter. This will run the migration in the transaction and roll it back without making any changes.

You can also use migration CLI in your build pipelines. You might not want to pass the connection string there, as it’s not secured way. No worries, you can also set DB_CONNECTION_STRING environment variable and run it as

npx @event-driven-io/pongo migrate run --config ./dist/pongoConfig.js

You can also run it by providing a collections list:

npx @event-driven-io/pongo migrate run --collection users --collection customers \
--connectionString postgresql://postgres:postgres@localhost:5432/postgres

You can also just print migrations to see what schema structures will be generated by calling:

npx @event-driven-io/pongo migrate sql --print --collection users --collection customers

It will print:

CREATE TABLE IF NOT EXISTS migrations (
    id SERIAL PRIMARY KEY,
    name VARCHAR(255) NOT NULL UNIQUE,
    application VARCHAR(255) NOT NULL DEFAULT 'default',
    sql_hash VARCHAR(64) NOT NULL,
    timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
  );

CREATE TABLE IF NOT EXISTS users (
      _id           TEXT           PRIMARY KEY, 
      data          JSONB          NOT NULL, 
      metadata      JSONB          NOT NULL     DEFAULT '{}',
      _version      BIGINT         NOT NULL     DEFAULT 1,
      _partition    TEXT           NOT NULL     DEFAULT 'png_global',
      _archived     BOOLEAN        NOT NULL     DEFAULT FALSE,
      _created      TIMESTAMPTZ    NOT NULL     DEFAULT now(),
      _updated      TIMESTAMPTZ    NOT NULL     DEFAULT now()
  );

CREATE TABLE IF NOT EXISTS customers (
      _id           TEXT           PRIMARY KEY, 
      data          JSONB          NOT NULL, 
      metadata      JSONB          NOT NULL     DEFAULT '{}',
      _version      BIGINT         NOT NULL     DEFAULT 1,
      _partition    TEXT           NOT NULL     DEFAULT 'png_global',
      _archived     BOOLEAN        NOT NULL     DEFAULT FALSE,
      _created      TIMESTAMPTZ    NOT NULL     DEFAULT now(),
      _updated      TIMESTAMPTZ    NOT NULL     DEFAULT now()
  );

This first migrations table is essential, as it keeps all the migrations running so far. So, if you run the migration CLI once, it’ll only run the migrations once. Migrations are using internally Postgres Advisory Locks to ensure that no migrations are happening in parallel. Safety first!

In the future, you’ll also be able to provide your custom schema and data migrations through it!

You already got Schema Components abstraction. They define the database schema as a tree structure. They’re used for database collection, allowing migration through code. They’re exposed in the schema property. In the longer term, it’ll be possible to add your own, like indexes, migrations, etc.

Added possibility to disable generating Pongo schema upfront

And we’re getting to performance. It appears that running schema migrations automatically is an excellent developer experience but not ideal for regular deployment. Surprise!

Now, thanks to the schema and CLI tooling for migrations, you can run migrations manually (or through the build process) and can ignore the automated migration in the Pongo client and get the performance boost:

const typedClient = pongoClient(postgresConnectionString, {
  schema: { autoMigration: 'None', definition: schema },
});

This will disable any automated schema generation. As a result, your application will have fewer database calls, opened connections, and overhead!

TLDR

The need for those changes appeared unexpectedly. I had to improve the performance for non-pooled connections (e.g., in a serverless environment). I could have done a quick patch and called it a day, but I did a sanity check and rethought that a bit. That led to a bit more work but also surprising synergy and opening more options for the future.

I’m pretty happy about that.

I think that’ll also boost the developer experience even more!

Expect the follow up in Emmett.

What are your thoughts?

Read more about building Emmett and Pongo in:

Check also the sample and release notes

Cheers!

Oskar

My Architecture Drivers

I don’t feel like an authority or an expert. I prefer to think about myself as a practitioner. Our industry is filled with self-proclaimed experts; we need more doers.

For similar reasons, I was reluctant to call myself an “Architect” for a long time. Somehow, it became pejorative, but should it be? I changed my mind thanks to Jarek Pałka, who says that “we all are architects”. It’s a simple line but multidimensional. That means (to me) that we all are responsible for making our systems work as they should. It’s not about bragging but accountability.

Still, I never feel comfortable being asked about Software Architecture Drivers. How do you give someone a checklist for good software? Yet I had to answer this time, as Maciej Jędrzejewski asked me if I could write my thoughts to his new book. How could I reject the offer? Actually, I could, as it surprised me, but hey, maybe my few words will help others fight their impostor syndrome. Or it was just too flattering. Nevertheless, here they are!

It’s funny that we called our industry SOFTware, but it’s all about making HARD decisions. Usually, we make them when we’re the dumbest: we don’t know the business domain, we don’t know the user needs, and we are unsure of technology choices. Plus, even if we do, our changing environment is open to proving us wrong.

For me, architecture decisions are more a process than a set of specific rules. It’s a process of answering the following questions:

WHY? So, understanding the product vision and business model. Consider where the money flows: who the client and the user are. That’s an important fact, as we should care about all users but optimise for clients, especially those who bring money. In the end, our product should bring money.

WHAT? Understand what we actually need to build. Set a mental model of the business workflow. This is an excellent moment for collaborative tooling, brainstorming and modelling practices like Event Storming, Domain Storytelling, etc.

HOW? Think about the requirements and guarantees you need to have. Find architecture patterns and class of solutions that will fulfil your requirements. So, the type of databases, deployment type, integration patterns, not the specific technologies. Consider tools like C4 and other tools to structure your findings.

WITH. Select the tooling based on the outcome of the previous point. It has to fulfil requirements, but also non-functional like costs, match team experience, ease of use. Then rinse and repeat.

Architecture is not created in a vacuum. Talk and collaborate with business, users and your technical fellows.

Consider the team you (can) have. Most of the time, the best technology is the one that your team knows. We’re building new tools, but to be true, rarely sophisticated ones. Most of them are regular lines of business applications.

And hey, let me share the secret with you: your decisions will be wrong. Mine also. And that’s fine. We don’t need to be flawless; our system also doesn’t need to be. Expect the change; it’ll come.

So don’t be afraid to make decisions, but don’t rush yourself. Always consider alternative solutions. Record your decisions together with thrown away ideas. Provide the context and explain WHY, WHAT, HOW, WITH. Provide the assumed limits. Suggest how to evolve if, e.g. your system will be a huge success and becomes overwhelmed by traffic. Some problems are good to have. But don’t need to be solved immediately.

We should optimise not for maintainability but for removability. If our system is built so that we can relatively easily remove pieces from it, then we can drop bad ideas and move on to new ones. Also, by accident, we’re getting a system that’s easier to maintain.

What are your architecture drivers? Or better, what’s your process?

Check also Maciej’s book “Master Software Architecture Book”. I’m still in front of reading it till the end. But I like what I see so far; the outline and range of topics show the holistic, actionable vision of building software. And, most importantly, it’s not pompous.

I think getting more books/talks like that would be great. We need more books in the spirit of “this is my story, this is why I think it’s important and worked out for me. “. It’s good that it’s from a personal perspective, allowing us to compare or benchmark it to our vision and experience.

If you need even more architecture benchmarks, I have gathered some of my past articles on my general approach to architecture and software design:

And check Architecture Weekly where I’m showing my less-event-driven face every week!

Cheers!

Oskar

Using event metadata in event-driven projections

Some time ago, I wrote about the dangers that come from the I’ll just add one more field” attitude. Have you heard about the Broken Window Theory? Authors (James Q. Wilson and George L. Kelling) wrote:

Social psychologists and police officers tend to agree that if a window in a building is broken and is left unrepaired, all the rest of the windows will soon be broken. This is as true in nice neighborhoods as in rundown ones. Window-breaking does not necessarily occur on a large scale because some areas are inhabited by determined window-breakers whereas others are populated by window-lovers; rather, one un-repaired broken window is a signal that no one cares, and so breaking more windows costs nothing. (It has always been fun.)

Adam Tornhill tells us to treat our code as a crime scene, and to some degree, we should. In event modelling, adding new property can be such a broken window if we’re doing it without considering alternatives and tradeoffs. Yet, sometimes, it can be a decent option. When? Let’s discuss it; that’s what we’re here for!

Let’s use this blog’s favourite example: the shopping cart. Let’s say we defined the following events (in TypeScript, but treat it just as syntax):

export type ProductItemAddedToShoppingCart = Event<
  'ProductItemAddedToShoppingCart',
  {
    shoppingCartId: string;
    clientId: string;
    productItem: PricedProductItem;
    addedAt: Date;
  }
>;

export type ProductItemRemovedFromShoppingCart = Event<
  'ProductItemRemovedFromShoppingCart',
  {
    shoppingCartId: string;
    productItem: PricedProductItem;
    removedAt: Date;
  }
>;

export type ShoppingCartConfirmed = Event<
  'ShoppingCartConfirmed',
  {
    shoppingCartId: string;
    confirmedAt: Date;
  }
>;

export type ShoppingCartCancelled = Event<
  'ShoppingCartCancelled',
  {
    shoppingCartId: string;
    cancelledAt: Date;
  }
>;

export type ShoppingCartEvent =
  | ProductItemAddedToShoppingCart
  | ProductItemRemovedFromShoppingCart
  | ShoppingCartConfirmed
  | ShoppingCartCancelled;

export interface ProductItem {
  productId: string;
  quantity: number;
}

export type PricedProductItem = ProductItem & {
  unitPrice: number;
};

They show what can happen in our shopping cart’s lifetime. They’re as small as possible, but not smaller. They’re granular and focused on business but have some potential redundancy by all having shopping cart id. I want to keep events expressive and telling what has happened. I wrote in another article on the danger of slicing too much from events.

Of course, this is the grey area and a careful act of balancing where to put where. For instance, I put the client ID only in product items added to the shopping cart. I expect it to be always the first event in the stream, and I don’t want to repeat it. Why? Because the stream represents a shopping cart, keeping the shopping cart id can be seen as explicit information about the context; the client id is an additional reference to another stream. That’s why I’m not repeating the client id in other events. But maybe I should?

What if we’d like to have the read model that aggregates the general summary of client’s pending, confirmed and cancelled shopping carts? It could be defined as:

export type ClientShoppingSummary = {
  clientId: string;
  pending: PendingSummary | undefined;
  confirmed: ConfirmedSummary;
  cancelled: CancelledSummary;
};

export type ShoppingSummary = {
  productItemsCount: number;
  totalAmount: number;
};

export type PendingSummary = ShoppingSummary & {
  cartId: string;
};

export type ConfirmedSummary = ShoppingSummary & {
  cartsCount: number;
};

export type CancelledSummary = ShoppingSummary & {
  cartsCount: number;
};

It contains the pending shopping cart information (if there’s such a thing) plus the total number of confirmed and cancelled shopping carts, their total amounts, and total product item counts.

The id of our read model is equal to the client id. Every client will have a single summary.

To build this read model, we need to correlate events with respective records. We’ll be applying events sequentially. We need to know which record they need to update. If our read model id is equal to the client id, then best if we have the client id in events. But besides the Product Item Added event, we don’t.

Of course, we could reconsider adding it to all the events, but we already discussed that we would not necessarily like to. So what should we do?

We could query some other read model (e.g. shopping cart details) and load the client id, but then we’d have an even worse problem. Tying those two models together and decreasing scaling and isolation. That’s an important aspect, I wrote about it here.

Still, if we think that business-wise, data should always be there, then we could use event metadata. Besides the regular data, events can also have metadata, so common information that is typically used for some generic processing could also (if needed) be used for business processing. That’s always a steep hill, and you better be careful not to make metadata a “bag for random data.” This definitely can be a hidden trap. There are no hard rules here, but some good heuristics.

We can look at our endpoints and commands that initiate business logic, resulting in events. For instance, if we look at:

POST /clients/:clientId/shopping-carts/current/product-items

DELETE /clients/:clientId/shopping-carts/current/product-items

POST /clients/:clientId/shopping-carts/current/confirm

DELETE /clients/:clientId/shopping-carts/current

GET /clients/:clientId/shopping-carts/current

Then, we see that all of them are in the current shopping cart context and the specific client. That can lead to the conclusion that we already have this client context in our requests. Maybe it’s used for authorisation, maybe for tenancy reasons.

If that’s not visible in endpoints, we can check on our authorisation rules and middleware. They typically need some data based on the currently authenticated user.

Having that, we could consider making the client id a part of the shopping cart event metadata. This could look as follows:

export type ShoppingCartEventMetadata = { clientId: string };

export type ProductItemAddedToShoppingCart = Event<
  'ProductItemAddedToShoppingCart',
 {
    shoppingCartId: string;
    clientId: string;
    productItem: PricedProductItem;
    addedAt: Date;
 },
  ShoppingCartEventMetadata
>;

export type ProductItemRemovedFromShoppingCart = Event<
  'ProductItemRemovedFromShoppingCart',
 {
    shoppingCartId: string;
    productItem: PricedProductItem;
    removedAt: Date;
 },
  ShoppingCartEventMetadata
>;

export type ShoppingCartConfirmed = Event<
  'ShoppingCartConfirmed',
 {
    shoppingCartId: string;
    confirmedAt: Date;
 },
  ShoppingCartEventMetadata
>;

export type ShoppingCartCancelled = Event<
  'ShoppingCartCancelled',
 {
    shoppingCartId: string;
    cancelledAt: Date;
 },
  ShoppingCartEventMetadata
>;

If we define also such metadata for our commands, we can pass it as that:

 // Confirm Shopping Cart
router.post(
  '/clients/:clientId/shopping-carts/current/confirm',
  on(async (request: Request) => {
    const clientId = assertNotEmptyString(request.params.clientId);
    const shoppingCartId = getShoppingCartId(
      assertNotEmptyString(request.params.clientId),
    );

    const command: ConfirmShoppingCart = {
      type: 'ConfirmShoppingCart',
      data: { shoppingCartId },
      // 👇 See what we did here
      metadata: { clientId, now: getCurrentTime() },
    };

    await handle(eventStore, shoppingCartId, (state) =>
      confirm(command, state),
    );

    return NoContent();
  }),
);

// Business logic
export const confirm = (
  command: ConfirmShoppingCart,
  state: ShoppingCart,
): ShoppingCartConfirmed => {
  if (state.status !== 'Opened')
    throw new IllegalStateError('Shopping Cart is not opened');

  const totalQuantityOfAllProductItems = sum(state.productItems.values());

  if (totalQuantityOfAllProductItems <= 0)
    throw new IllegalStateError('Shopping Cart is empty');

  const {
    data: { shoppingCartId },
    metadata,
  } = command;

  return {
    type: 'ShoppingCartConfirmed',
    data: {
      shoppingCartId,
      confirmedAt: metadata?.now ?? new Date(),
    },
    // 👇 See what we did here
    metadata: { clientId: metadata!.clientId },  
  };
};

I’m using TypeScript, Express.js and Emmett, but I’m sure that you can translate that to your favourite language and tech stack.

As we now have the client ID in metadata, we could use it for the event to read model correlation. For instance, using Emmett+Pongo projections:

const clientShoppingSummaryCollectionName = 'ClientShoppingSummary';

export const clientShoppingSummaryProjection = pongoMultiStreamProjection({
  collectionName: clientShoppingSummaryCollectionName,
  // 👇 See what we did here
  getDocumentId: (event) => event.metadata.clientId,
  evolve,
  canHandle: [
    'ProductItemAddedToShoppingCart',
    'ProductItemRemovedFromShoppingCart',
    'ShoppingCartConfirmed',
    'ShoppingCartCancelled',
 ],
});

We’re saying that to find the document ID for each shopping cart event, you can use event.metadata.clientId.

Then, the projection definition can look as follows:

const evolve = (
  document: ClientShoppingSummary | null,
 { type, data: event, metadata }: ShoppingCartEvent,
): ClientShoppingSummary | null => {
  const summary: ClientShoppingSummary = document ?? {
    clientId: metadata!.clientId,
    pending: undefined,
    confirmed: initialSummary,
    cancelled: initialSummary,
 };

  switch (type) {
    case 'ProductItemAddedToShoppingCart':
      return {
        ...summary,
        pending: {
          cartId: event.shoppingCartId,
          ...withAdjustedTotals({
            summary: summary.pending,
            with: event.productItem,
            by: 'adding',
          }),
        },
    };
    case 'ProductItemRemovedFromShoppingCart':
      return {
        ...summary,
        pending: {
          cartId: event.shoppingCartId,
          ...withAdjustedTotals({
            summary: summary.pending,
            with: event.productItem,
            by: 'removing',
          }),
        },
    };
    case 'ShoppingCartConfirmed':
      return {
        ...summary,
        pending: undefined,
        confirmed: {
          cartsCount: summary.confirmed.cartsCount + 1,
          ...withAdjustedTotals({
            summary: summary.confirmed,
            with: summary.pending!,
            by: 'adding',
          }),
      },
    };
    case 'ShoppingCartCancelled':
      return {
        ...summary,
        pending: undefined,
        cancelled: {
          cartsCount: summary.confirmed.cartsCount + 1,
          ...withAdjustedTotals({
            summary: summary.confirmed,
            with: summary.pending!,
            by: 'adding',
          }),
        },
    };
    default:
      return summary;
    }
};

const initialSummary = {
  cartsCount: 0,
  productItemsCount: 0,
  totalAmount: 0,
};

const withAdjustedTotals = (options: {
  summary: ShoppingSummary | undefined;
  with: PricedProductItem | ShoppingSummary;
  by: 'adding' | 'removing';
}) => {
  const { summary: document, by } = options;

  const totalAmount =
    'totalAmount' in options.with
      ? options.with.totalAmount
      : options.with.unitPrice * options.with.quantity;
  const productItemsCount =
    'productItemsCount' in options.with
      ? options.with.productItemsCount
      : options.with.quantity;

  const plusOrMinus = by === 'adding' ? 1 : -1;

  return {
    ...document,
    totalAmount: (document?.totalAmount ?? 0) + totalAmount * plusOrMinus,
    productItemsCount:
      (document?.productItemsCount ?? 0) + productItemsCount * plusOrMinus,
 };
};

Is it the best option? It depends on the use case. I’d always treat it as a tradeoff and evaluate if there are other options (e.g. in-flight event transformations). Still, I hope this article gives you enough context for the tradeoff analysis and making educated decisions in your design.

Read also more in:

Cheers!

Oskar

Writing and testing event-driven projections with Emmett, Pongo and PostgreSQL

In the previous article, I told what happened when Emmett and Pongo walked into a bar. In other words, I announced that you can now do Event Sourcing in Node.js on top of PostgreSQL. You can use Emmett as an event store and Pongo, changing PostgreSQL into a document Mongo-like database. With all the strong consistency benefits and integration happening behind the scenes.

Let’s explore this in more detail today, focusing on the single-stream projection.

In event sourcing, a stream represents the history of a specific process or object, such as a shopping cart. After each business operation, a new event is appended to the end of the stream. For business logic, we’re getting all events (so facts), interpreting them, and building the current state in memory. That works well for business logic, as streams should be short, and this shouldn’t take long. You can read more about it in How to get the current entity state from events?

Yet, reading more than one record wouldn’t play well. For that, we need to materialise our data into read models. We can apply our events and store the result in the database table; then, we can apply filtering and get the set of results. Projections are functions that transform events into read models. Check more in Guide to Projections and Read Models in Event-Driven Architecture.

Some event stores, like EventStoreDB, separate the event storage and read model storage. That allows its creators to focus on making one thing right. Still, in many cases, it’s more than enough to just use PostgreSQL. That’s what the Emmett and Pongo combination does pretty well!

Single Stream Projections with Pongo

Let’s install Emmett PostgreSQL package:

$ npm add @event-driven-io/emmett-postgresql

Behind the scenes, it’ll also install Emmett and Pongo as peer dependencies, so you don’t need to install them separately.

Now we’ll define a simplified Shopping Cart flow, expressing it by the events that can happen:

import { type Event } from '@event-driven-io/emmett';

type ProductItemAdded = Event<
  'ProductItemAdded',
  { productItem: PricedProductItem; addedAt: Date }
>;

type ProductItemRemoved = Event<
  'ProductItemAdded',
  { productItem: PricedProductItemm; removedAt: Date }
>;

type PricedProductItem = {
  productId: string;
  quantity: number;
  unitPrice: number;
};

type DiscountApplied = Event<
  'DiscountApplied',
  { percent: number; couponId: string; discountedAt: Date }
>;

type ShoppingCartConfirmed = Event<
  'ShoppingCartConfirmed',
  { confirmedAt: Date }
>;

type ShoppingCartEvent = 
  | ProductItemAdded
  | ProductItemRemoved 
  | DiscountApplied
  | ShoppingCartConfirmed;

We’ll skip the business logic part; you can learn about it in detail in Emmett’s Getting Started Guide. Let’s go straight to thinking about read models.

Let’s say that we want to show the summary of the shopping cart, showing just the total items count and amount. This could be used, e.g., in the top menu bar, to give users quick feedback. It could be defined as:

type ShoppingCartSummary = {
  _id?: string;
  productItemsCount: number;
  totalAmount: number;
};

Now, let’s define how we’d like to apply those events:

const evolve = (
  document: ShoppingCartSummary | null,
  { type, data: event }: ProductItemAdded | ProductItemRemoved | DiscountApplied,
): ShoppingCartSummary => {
  document = document ?? { totalAmount: 0; productItemsCount: 0 }

  switch (type) {
    case 'ProductItemAdded': 
      return withAdjustedTotals({
        document,
        productItem: event.productItem,
        by: 'adding'
      });
    case 'ProductItemRemoved':
      return withAdjustedTotals({
        document,
        productItem: event.productItem,
        by: 'removing'
      });
    case 'DiscountApplied':
      return {
        ...document,
        totalAmount: (document.totalAmount * (100 - event.percent)) / 100,
      };
  }
};

const withAdjustedTotals = (
  options: { 
    document: ShoppingCartSummary; 
    productItem: PricedProductItem; 
    by: 'adding' | 'removing'
  }) => {
    const { document, productItem, by } = options;
    const plusOrMinus = by === 'adding' ? 1: -1;

    return {
      ...document,
      totalAmount:
        document.totalAmount +
        productItem.unitPrice * productItem.quantity * plusOrMinus,
      productItemsCount:
        document.productItemsCount + productItem.quantity * plusOrMinus,
    }; 
  }

As you see, the transformation may not need to handle all events. We don’t need to know the status (whether it’s confirmed or not); we just need information about totals.

The next step is to define our Pongo projection. We do it by:

import { pongoSingleStreamProjection } from '@event-driven-io/emmett-postgresql';

const collectionName = 'shopping_carts_summary';

const shoppingCartSummaryProjection = pongoSingleStreamProjection({
  canHandle: ['ProductItemAdded', 'ProductItemRemoved', 'DiscountApplied'],
  collectionName,
  evolve,
});

By that we’re handling the specified range of events, applying it using the evolve function and storing the result in the specified Pongo collection (so the PostgreSQL table with JSONB column for document data).

If you don’t like getting a null document in the evolve function, then you can also provide the initial state:

const shoppingCartSummaryProjection = pongoSingleStreamProjection({
  canHandle: ['ProductItemAdded', 'ProductItemRemoved', 'DiscountApplied'],
  collectionName,
  evolve,
  initialState: () => { totalAmount: 0; productItemsCount: 0 }
});

Then your evolve can skip the setup step and look as follows:

const evolve = (
  document: ShoppingCartSummary,
  { type, data: event }: ProductItemAdded | ProductItemRemoved | DiscountApplied,
): ShoppingCartSummary => {
  switch (type) {
    case 'ProductItemAdded': 
      return withAdjustedTotals({
        document,
        productItem: event.productItem,
        by: 'adding'
      });
    case 'ProductItemRemoved':
      return withAdjustedTotals({
        document,
        productItem: event.productItem,
        by: 'removing'
      });
    case 'DiscountApplied':
      return {
        ...document,
        totalAmount: (document.totalAmount * (100 - event.percent)) / 100,
      };
  }
};

Emmett will provide the initial state if the document with id equal to the stream name doesn’t exist.

We need to complete registration by passing it to event store options:

import { projection } from '@event-driven-io/emmett';
import { getPostgreSQLEventStore } from '@event-driven-io/emmett-postgresql';

const connectionString =
  "postgresql://dbuser:[email protected]:3211/mydb";

const eventStore = getPostgreSQLEventStore(connectionString, {
  projections: projections.inline([
    shoppingCartSummaryProjection,
  ]),
});

Inline registration means that projections will update your read models in the same transaction as appending events. So, either all was stored or nothing. Of course, you need to be careful with them, as they can slow your appends, but they’re really useful. Async projections will come in future releases.

Sounds cool; now we can append a few events through regular event store append events api and query results using Pongo:

import { getPostgreSQLEventStore } from '@event-driven-io/pongo';

const connectionString =
  "postgresql://dbuser:[email protected]:3211/mydb";

const pongo = pongoClient(connectionString);

const shoppingCartsSummary = pongo.db().collection('shopping_carts_summary');

const summary = await shoppingCartsSummary.findOne({ _id : 'shopping_cart-123' });

Cool, but manually querying data isn’t the best long-term solution. Wouldn’t it be possible to test it automatically? Yes, it would.

That’s why Emmett gives you a built-in way to test projections.

Testing projections

In the same way as testing other parts of your development flow (read more in Testing Event Sourcing, Emmett edition).

Before we start, let’s install PostgreSQL Test Containers. We’ll need them soon:

$ npm add @testcontainers/postgresql

As I described in How to test event-driven projections, projection tests should be tested against the real database. Both querying and update capabilities and serialisation can play tricks, so it is better to be certain that it really works. Tests that don’t give us such assurance are useless. And we don’t want them to be such.

Now, let’s start with the setup:

import { PostgreSQLProjectionSpec } from '@event-driven-io/emmett-postgresql';
import { 
  PostgreSqlContainer,
  StartedPostgreSqlContainer,
} from '@testcontainers/postgresql';
import { after, before, beforeEach, describe, it } from 'node:test';
import { v4 as uuid } from 'uuid';

void describe('Shopping carts summary', () => {
  let postgres: StartedPostgreSqlContainer;
  let connectionString: string;
  let given: PostgreSQLProjectionSpec<ProductItemAdded | ProductItemRemoved | DiscountApplied>;
  let shoppingCartId: string;

  before(async () => {
    postgres = await new PostgreSqlContainer().start();
    connectionString = postgres.getConnectionUri();

    given = PostgreSQLProjectionSpec.for({
      projection: shoppingCartShortInfoProjection,
      connectionString,
    });
  });

  beforeEach(() => (shoppingCartId = `shoppingCart-${uuid()}`));
});

We’re setting up the PostgreSQL test container and projection specification. We’ll use it to run our tests. The first one could look as follows:

import { expectPongoDocuments } from '@event-driven-io/emmett-postgresql';

void describe('Shopping carts summary', () => {
  // (...) test setup

  void it('first added product creates document', () =>
    given([])
      .when([
        {
          type: 'ProductItemAdded',
          data: {
            productItem: { unitPrice: 100, productId: 'shoes', quantity: 100 },
          },
          metadata: {
            streamName: shoppingCartId,
          },
        },
      ])
      .then(
        expectPongoDocuments
          .fromCollection<ShoppingCartSummary>('shopping_carts_summary')
          .withId(shoppingCartId)
          .toBeEqual({
            productItemsCount: 100,
            totalAmount: 10000,
            appliedDiscounts: [],
          }),
      ));
});

Tests are written in Behaviour-Driven Design style, using Given/When/Then:

Given existing stream of events,
When new events are added,
Then read model is updated.

We do it this way, as we expect read models to be ONLY updated through upcoming events.

Emmett gives you also out-of-the-box test assertion helpers to make testing Pongo easier.

You may have noticed that our Given is empty, so let’s fix it!

import {
  eventsInStream,
  newEventsInStream,
} from '@event-driven-io/emmett-postgresql';

void describe('Shopping carts summary', () => {
  // (...) test setup

  void it('applies discount for existing shopping cart with product', () => {
    const couponId = uuid();

    return given(
      eventsInStream(shoppingCartId, [
        {
          type: 'ProductItemAdded',
          data: {
            productItem: { unitPrice: 100, productId: 'shoes', quantity: 100 },
          },
        },
      ]),
    )
      .when(
        newEventsInStream(shoppingCartId, [
          {
            type: 'DiscountApplied',
            data: { percent: 10, couponId },
          },
        ]),
      )
      .then(
        expectPongoDocuments
          .fromCollection<ShoppingCartShortInfo>(
            shoppingCartShortInfoCollectionName,
          )
          .withId(shoppingCartId)
          .toBeEqual({
            productItemsCount: 100,
            totalAmount: 9000,
          }),
      );
  });
});

You probably noticed the next helpers: _eventsInStream_and newEventsInStream. They’re responsible for shortening the setup. Depending on your preferences, you can use the raw events setup, including manual metadata assignment, or a more explicit intention helper. My preference would be to use the helper, but it’s up to you to decide!

You could even write tests like this:

void describe('Shopping carts summary', () => {
  // (...) test setup

  void it('applies discount for existing shopping cart with product', () => 
    given(
        shoppingCartWithProductItem(shoppingCartId, {
          unitPrice: 100, 
          quantity: 100,
        })
      )
      .when(
        discountApplied(shoppingCartId, {
          percent: 10,
        }))
      .then(
        summaryUpdated(shoppingCartId, {
          productItemsCount: 100,
          totalAmount: 9000,
        })
      ));

The setup methods are defined below:

void describe('Shopping carts summary', () => {
  // (...) test

  const shoppingCartWithProductItem = (
      shoppingCartId: string, 
      productItem: Partial<PricedProductItem>,
  ) => 
    eventsInStream(shoppingCartId, [
      {
        type: 'ProductItemAdded',
        data: {
          productItem: { 
            unitPrice: 100, 
            productId: 'shoes', 
            quantity: 100,
            ...productItem
          },
        },
      },
    ]),

  const discountApplied = (
    shoppingCartId: string, 
    data: { discount?: number; couponId?: string },
  ) => 
    newEventsInStream(shoppingCartId, [
      {
        type: 'DiscountApplied',
        data: { percent: 10, couponId: uuid(), ...data },
      },
    ]),

  const summaryUpdated = (shoppingCartId: string, expected: ShoppingCartSummary) =>
    expectPongoDocuments
      .fromCollection<ShoppingCartShortInfo>(
        shoppingCartShortInfoCollectionName,
      )
      .withId(shoppingCartId)
      .toBeEqual(expected}),     
});

That approach is more focused on the business flow and allows us to reuse those test setups. Still, the outcome is the same.

Can read model data be only updated? Not only that, you can also delete it. Let’s imagine that the confirming cart should clear its read model, as we expect a new, empty shopping cart to be created when the new buying process starts.

To have that, we need to update our evolve function by adding ShopingCartConfirmed event:

const evolve = (
  document: ShoppingCartSummary,
  { type, data: event }: ShoppingCartEvent,
): ShoppingCartSummary | null => { // <= see here
  switch (type) {
    case 'ProductItemAdded': 
      return withAdjustedTotals({
        document,
        productItem: event.productItem,
        by: 'adding'
      });
    case 'ProductItemRemoved':
      return withAdjustedTotals({
        document,
        productItem: event.productItem,
        by: 'removing'
      });
    case 'DiscountApplied':
      return {
        ...document,
        totalAmount: (document.totalAmount * (100 - event.percent)) / 100,
      };

    case 'ShoppingCartConfirmed':
      return null; // <= and here
  }
};

We made the shopping cart confirmed event return null. That means that the document should be removed. Emmett is using the Pongo’s handle method internally:

const collection = pongo.db().collection<Document>(collectionName);

for (const event of events) {
  await collection.handle(getDocumentId(event), async (document) => {
    return await evolve(document, event);
  });
}

Pongo’s handle method loads the existing document and tries to insert, replace, or delete it depending on the function’s result. It’s a bit sneaky, but pretty useful, isn’t it?

The test checking will look as follows:

void describe('Shopping carts summary', () => {
  let given: PostgreSQLProjectionSpec<ShoppingCartEvent>;
  // (...) test setup

  void it('confirmed event removes read mode for shopping cart with applied discount', () => {
    const couponId = uuid();

    return given(
      eventsInStream(shoppingCartId, [
        {
          type: 'ProductItemAdded',
          data: {
            productItem: { unitPrice: 100, productId: 'shoes', quantity: 100 },
          },
        },
        {
          type: 'DiscountApplied',
          data: { percent: 10, couponId },
        },
      ]),
    )
      .when(
        newEventsInStream(shoppingCartId, [
          {
            type: 'ShoppingCartConfirmed',
            data: { confirmedAt: new Date() },
          },
        ]),
      )
      .then(
        expectPongoDocuments
          .fromCollection<ShoppingCartShortInfo>(
            shoppingCartShortInfoCollectionName,
          )
          .withId(shoppingCartId)          
          .notToExist(), // <= see this
      );
  });
});

The pattern looks the same, but the assertion is different.

I hope that this article shows you how powerful the combination of Emmett, Pongo, and PostgreSQL is. I also wanted to show you that I intend to give you certainty and trust in the software you’re building. Having built-in support for tests should help you with that.

Go ahead, play with it, check it, and drop me a line. Joining our Discord is an excellent way to do this.

Cheers!

Oskar

Event Sourcing on PostgreSQL in Node.js just became possible with Emmett

Last week, I announced Pongo - Mongo, but it was on PostgreSQL. So, the Node.js library allows using PostgreSQL as a document database.

Today, I have at least an equally big announcement: I released the PostgreSQL event store for Emmett. Boom!

What’s Emmett? It’s an Event Sourcing library. I announced it some time ago and have worked on it continuously for the last few months. It already supports EventStoreDB, and now it has our favourite PostgreSQL storage!

How to use it? Pretty simple. Start with installing npm package:

$ npm add @event-driven-io/emmett-postgresql

Then setup event store using connection string to PostgreSQL:

import { getPostgreSQLEventStore } from '@event-driven-io/emmett-postgresql';

const connectionString =
  "postgresql://dbuser:[email protected]:3211/mydb";

const eventStore = getPostgreSQLEventStore(connectionString);

Internally, it uses the node-postgres package with connection pooling. So you don’t need to do much more. Well, maybe besides gracefully closing it on the application closure:

await eventStore.close();

Cool, but what you can do with it? Check Emmett docs. The same you can do with EventStoreDB storage you can do with PostgreSQL!

Or you can actually do more with PostgreSQL, as…

PostgreSQL has inline projections support. What are inline projections? They’re functions updating your read models in the same transaction as appending events. So either all was stored or nothing. Of course, you need to be careful with them as they can slow your appends, but they’re really useful. Async projections will come in future releases.

Ok, but how to use it? Let’s say that you’d like to build a read model with a summary of your shopping cart:

type ShoppingCartShortInfo = {
  productItemsCount: number;
  totalAmount: number;
};

Transformation function could look like that:

const evolve = (
  document: ShoppingCartShortInfo | null,
  { type, data: event }: ProductItemAdded | DiscountApplied,
): ShoppingCartShortInfo => {
  document = document ?? { productItemsCount: 0, totalAmount: 0 };

  switch (type) {
    case 'ProductItemAdded':
      return {
        totalAmount:
          document.totalAmount +
          event.productItem.price * event.productItem.quantity,
        productItemsCount:
          document.productItemsCount + event.productItem.quantity,
      };
    case 'DiscountApplied':
      return {
        ...document,
        totalAmount: (document.totalAmount * (100 - event.percent)) / 100,
      };
  }
};

It’ll be run for each event of type ProductItemAdded and DiscountApplied that’s appended to the event store.

Let’s say that you’d like to use Pongo and store it as a document in PostgreSQL, then you can define projection as:

const shoppingCartShortInfoCollectionName = 'shoppingCartShortInfo';

const shoppingCartShortInfoProjection = pongoSingleStreamProjection({
  collectionName: shoppingCartShortInfoCollectionName,
  evolve,
  canHandle: ['ProductItemAdded', 'DiscountApplied'],
});

and register it through event store options:

import { projection } from '@event-driven-io/emmett';
import { getPostgreSQLEventStore } from '@event-driven-io/emmett-postgresql';

const connectionString =
  "postgresql://dbuser:[email protected]:3211/mydb";

const eventStore = getPostgreSQLEventStore(connectionString, {
  projections: projections.inline([
    shoppingCartShortInfoProjection,
    customProjection,
  ]),
});

We’re saying that we’d like to update the shoppingCartShortInfo collection using the evolve function for the following set of event types.

Internally, it’ll use the Pongo new feature: a handler that loads the existing document and tries to insert, replace or delete it depending on the result obtained from the function.

It look’s as follows:

const collection = pongo.db().collection<Document>(collectionName);

for (const event of events) {
  await collection.handle(getDocumentId(event), async (document) => {
    return await evolve(document, event);
  });
}

If you’re wondering what getDocumentId is, then for pongoSingleStreamProjection, it’ll automatically use the stream name as the document id.

Suppose you’d like to customise it, e.g. to match events from different streams. In that case, you can use the pongoMultiStreamProjection definition, which allows you to specify the document ID matcher for each event. For instance:

const shoppingCartShortInfoCollectionName = 'shoppingCartShortInfo';

const getDocumentId = ({type}:  ProductItemAdded | DiscountApplied): string => {
  switch(type)
  {
    case 'ProductItemAdded': 
      return event.metadata.streamName;
    case 'DiscountApplied': 
      return event.metadata.streamName;
  }
};

const shoppingCartShortInfoProjection = pongoSingleStreamProjection({
  collectionName: shoppingCartShortInfoCollectionName,
  evolve,
  canHandle: ['ProductItemAdded', 'DiscountApplied'],
  getDocumentId
});

You can also do a free-hand projection using pongoProjection that takes the following handler:

(pongo: PongoClient, events: ReadEvent<EventType>[]) => Promise<void>

Cool, isn’t it?

**Read more details in the follow up article Writing and testing event-driven projections with Emmett, Pongo and PostgreSQL

Of course, those are still experimental features; they need to be optimised, tested extensively, etc. But they work, which makes me happy.

As you see, I’m quite thrilled that I could deliver it, as this is a big milestone. This will enable many people to finally do Event Sourcing in Node.js using PostgreSQL and have basic building blocks.

All of that wouldn’t be possible with my recent changes to Pongo.

I managed to close the basic coverage of document manipulation methods. I added initial versions of what was initially missing: replaceOne, drop, rename, countDocuments, count, estimatedDocumentCount, findOneAndDelete, findOneAndReplace, findOneAndUpdate, etc.

Now, a bigger portion that were made as preparations to Emmett PostgreSQL projections you just learned about:

Added option to inject external connection pool and db client to Pongo collection as the first step for transaction handling. Now, you can create transactions outside and inject pool clients. It’s not yet fully the same as Mongo API; it’ll be delivered in follow-up PR.
Strengthened the schema and updated the id from UUID to text. Changed _id type to TEXT. In PostgreSQL, this should be almost equally the same indexable. Of course, it’ll take a bit more storage, but let’s live with that for now. Changing from uuid to text will allow more sophisticated key strategies. Most importantly, it’ll reuse the stream ID as a document ID for Emmett projections.

Also, thanks to the Franck Pachot contribution, we confirmed that Pongo is compatible not only with vanilla Postgres but also databases like Yugabyte!

It was just a week, but I’m extremely happy with how Pongo was taken by the community. I got to HackerNews front page and got over 900 GitHub stars!

Now it’s the time for Emmett! Synergy with Pongo should help with that.

What a week! It’s easy to forget that Pongo was released just 7 days ago!

Now I can go to vacations, next blog will be in August!

Cheers!

Oskar

Pongo - Mongo but on Postgres and with strong consistency benefits

Flexibility or Consistency? Why not have both? Wouldn’t it be great to have MongoDB flexible schema and PostgreSQL consistency?

MongoDB is a decent database, but it gives headaches with its eventual consistency handling. I wrote about it a few times in past:

Don’t get me wrong, eventual consistency is fine. We need to learn to live with that, still… Undeniably, having strong consistency guarantees, transactions, read your own writing is great.

On Friday, I decided to spend my working day on the small proof of concept that I called Pongo.

What’s Pongo?

It’s a MongoDB-compliant wrapper on top of Postgres.

You can setup it like that:

import { pongoClient } from "@event-driven-io/pongo";

const connectionString =
  "postgresql://dbuser:[email protected]:5432/yourdb";

const pongoClient = pongoClient(postgresConnectionString);
const pongoDb = pongoClient.db();

const users = pongoDb.collection<User>("users");

It will start internally with a PostgreSQL connection pool connected to your selected database.

Having that, you can then perform operations like:

const anita = { name: "Anita", age: 25 };

// Inserting
await pongoCollection.insertOne(roger);
await pongoCollection.insertOne(cruella);

const { insertedId } = await pongoCollection.insertOne(alice);
const anitaId = insertedId;

// Finding by Id
const anitaFromDb = await pongoCollection.findOne({ _id: anitaId });

// Updating
await users.updateOne({ _id: anitaId }, { $set: { age: 31 } });

// Deleting
await pongoCollection.deleteOne({ _id: cruella._id });

// Finding by Id
const anitaFromDb = await pongoCollection.findOne({ _id: anitaId });

// Finding more
const users = await pongoCollection.find({ age: { $lt: 40 } });

Internally, it’ll set up the collection as the PostgreSQL table with the key-value structure:

CREATE TABLE IF NOT EXISTS "YourCollectionName" (
    _id           TEXT           PRIMARY KEY,
    data          JSONB          NOT NULL,
    metadata      JSONB          NOT NULL     DEFAULT '{}',
    _version      BIGINT         NOT NULL     DEFAULT 1,
    _partition    TEXT           NOT NULL     DEFAULT 'png_global',
    _archived     BOOLEAN        NOT NULL     DEFAULT FALSE,
    _created      TIMESTAMPTZ    NOT NULL     DEFAULT now(),
    _updated      TIMESTAMPTZ    NOT NULL     DEFAULT now()
)

Essentially, it treats PostgreSQL as a key/value database. Sounds familiar? Yet, it’s a similar concept to Marten or, more correctly, to AWS DocumentDB (see here or there, they seem to be using Mongo syntactic sugar on top of AuroraDB with Postgres).

I explained in general strategy for migrating relational data to document-based that contrary to common belief, document data is structured but less rigidly, as in the relational approach. JSON has structure, but it is not enforced for each document. We can easily extend the schema for our documents, even for specific ones, by adding new fields. We should also not fail if the field we expect to exist, but doesn’t.

Handling semi-structured data in a relational database can be tricky, but PostgreSQL’s JSONB data type offers a practical solution. Unlike the plain text storage of the traditional JSON type, JSONB stores JSON data in a binary format. This simple change brings significant advantages in terms of performance and storage efficiency.

The binary format of JSONB means that data is pre-parsed, allowing faster read and write operations than text-based JSON. You don’t have to re-parse the data every time you query it, which saves processing time and improves overall performance. Additionally, JSONB supports advanced indexing options like GIN and GiST indexes, making searches within JSONB documents much quicker and more efficient.

Moreover, JSONB retains the flexibility of storing semi-structured data while allowing you to use PostgreSQL’s robust querying capabilities. You can perform complex queries, joins, and transactions with JSONB data, just as you can with regular relational data.

This flexibility, performance, and consistency combination makes PostgreSQL with JSONB a powerful tool. There are benchmarks showing that it can be even faster than MongoDB.

Still, the syntax is not the most pleasant (to say mildly). Just check the docs or see what Pongo does behind the scenes.

For instance, the MongoDB update syntax:

const pongoCollection = pongoDb.collection<User>("users");

await pongoCollection.updateOne(
  { _id: someId },
  { $push: { tags: "character" } }
);

will be translated to:

UPDATE "users"
SET data = jsonb_set(data, '{tags}', (COALESCE(data->'tags', '[]'::jsonb) || to_jsonb('character')))
WHERE _id = '137ef052-e41c-428b-b606-1c8070a47eda';

Or for query:

const result = await pongoCollection
  .find({ "address.history": { $elemMatch: { street: "Elm St" } } })
  .toArray();

will result in:

SELECT data
FROM "users"
WHERE jsonb_path_exists(
  data,
  '$.address.history[*] ? (@.street == "Elm St")'
);

I thought that it’d be much easier if you could reuse your muscle memory from working with Mongo and use familiar syntax to access the data. I even prepared the compliant shim:

import { MongoClient } from "@event-driven-io/pongo";
import { v4 as uuid } from "uuid";

type User = { name: string; age: number };

const connectionString =
  "postgresql://dbuser:[email protected]:3211/mydb";

const pongoClient = new MongoClient(postgresConnectionString);
const pongoDb = pongoClient.db();

const users = pongoDb.collection<User>("users");
const anita = { name: "Anita", age: 25 };

// Inserting
const { insertedId } = await pongoCollection.insertOne(alice);
const anitaId = insertedId;

// Updating
await users.updateOne({ _id: anitaId }, { $set: { age: 31 } });

// Deleting
await pongoCollection.deleteOne({ _id: cruella._id });

// Finding by Id
const anitaFromDb = await pongoCollection.findOne({ _id: anitaId });

// Finding more
const users = await pongoCollection.find({ age: { $lt: 40 } }).toArray();

And guess why? You can already use it by installing the package:

npm install @event-driven-io/pongo

Why did I create Pongo? There are two reasons.

First, I’ll need it for Emmett read models. I was a bit silent about updates to Emmett as I’m working on adding subscriptions and streaming capabilities. The ongoing Pull Request went a bit out of hand, and I was a bit worn out.

Thus, the second reason. Sometimes, you just need to have fun. We too often forget about it. That’s also why I came up with the name, a mixture of Mongo and Postgres and a reference to one of my favourite children’s movie characters.

Is it production-ready? You know the answer. What’s there works fine, but it’s far from having a fully compliant MongoDB API. And it might not have it fully, but Pareto principle works here. I also hope that I’ll get from you or others who decide to use its contribution or sponsoring to bring it to the expected level.

Still, it can already do the most needed operations, so you should be fine with trying it!

I’m planning to do a series of articles on the internals of building such a tool!

I’m curious about your thoughts on it. Yay or nay?

Cheers!

Oskar

Filtering EventStoreDB subscriptions by event types

Regular expressions are one of the classic examples of hate and hate relationships. Yes, it’s not a typo; hate and hate. Do you know anyone who loves or knows how to write moderately complex regex? And can they keep their skill for longer than two weeks without forgetting how to do it?

Maybe this whole wave of Large Language Models is about having someone who will show us how to write regexes. Maybe we hate regular expressions so much that we don’t care about hallucinations.

Still, undeniably, regular expressions are useful and powerful. Let me show you an example, but be careful; I warned you already!

We’ll use in this article knowledge from my two other articles:

Don’t want to dive deep into them? Fine, here’s the TLDR:

EventStoreDB is a database that has the durable append-only log as the physical structure where events are stored,
events are grouped into streams. Stream is a key/value pair where the key is the record id (e.g. order id), and the value is a sequence of events recorded for this record (e.g. OrderConfirmed, OrderPaymentRecorded, etc.)
Events have specific data and metadata, e.g. event type name; we can use it to deserialise events to a specific type. We can do manual type resolution, or automatic, based on conventions.
EventStoreDB has a functionality called Catch-up subscriptions, which allows you to subscribe to push notifications about new events. You can subscribe to all events or those coming from specific streams.

Today, we’ll focus on subscribing to all events and how to filter them by event type.

You can think about EventStoreDB subscription as some moving cursor. They start by pointing to the selected position (e.g., the start of the log) and then move sequentially event by event. The global log (called in EventStoreDB $all stream) represents the physical file storage. Thus, it’s fast as it doesn’t go to indexes. The safe default recommendation is to start from the global log if you want to get events from multiple streams. Typically, you do, as you may want to trigger some flows, build read models, or forward events to the messaging system.

By default, you’ll get all events, whether you need them or not, including internal events used for internal store processing. That’s a waste of network and resources. Not surprisingly, the mature database should provide a solution for that, and that’s what EventStoreDB do with server-side filtering. It will perform the filtering during moving cursor, skipping unwanted events without sending them to our application.

Out of the box, you can:

filter out system events,
filter by the stream name prefix,
filter by the event type prefix.

That covers default cases, especially if you have conventional-based naming. You can include module prefixes and stream category prefixes into the name and get a specific set of events, but what if you’d like to get events with specific types? What if those events don’t share a common prefix because they’re from different (sub)modules or stream categories? Then you have one more option.

You can filter out events using our inglorious regular expressions! Yes, I know how that sounds, but please stay with me.

The subscription with filter looks like that in C#:

var filterOptions = new SubscriptionFilterOptions(EventTypeFilter.RegularExpression("SOME REGEX"));

await using var subscription = client.SubscribeToAll(
        FromAll.Start,
        filterOptions: 
        cancellationToken: ct
);

Regular expressions allow us to provide the OR operator. Let’s say that we want to get InvoiceIssued and UserEmailProvided events. Regex for it could look as

^(InvoiceIssued|UserEmailProvided)$"

Not that hard, aye? Let’s try to generalise it then:

public static class EventFilters
{
    public static Regex OneOfEventTypesRegex(params string[] values) =>
        new("^(" + string.Join("|", values.Select(Regex.Escape)) + ")$");
}

Simple code, we’re just joining the strings with | separator, escaping the text if it includes some reserved regular expression characters like dot.

We can pass the array of strings and build regular expressions from them. We can call it as:

var filterRegex = EventFilters.OneOfEventTypesRegex("InvoiceIssued", "UserEmailProvided");

var filterOptions = new SubscriptionFilterOptions(EventTypeFilter.RegularExpression(filterRegex ));

We could also wrap that into a single method:

public static class EventFilters
{
    public static Regex OneOfEventTypesRegex(params string[] values) =>
        new("^(" + string.Join("|", values.Select(Regex.Escape)) + ")$");

    public static IEventFilter OneOfEventTypes(params string[] values) =>
        EventTypeFilter.RegularExpression(OneOfEventTypesRegex(values));
}

Now, if we have the event type mapper that’s mapping CLR type into the event type name (known from mentioned earlier article), we could also add:

public static class EventFilters
{
    // (...)
    public static Regex OneOfEventTypesRegex(params string[] values) =>
        new("^(" + string.Join("|", values.Select(Regex.Escape)) + ")$");

    public static Regex OneOfEventTypesRegex(EventTypeMapper eventTypeMapper, params Type[] eventTypes) =>
        OneOfEventTypesRegex(eventTypes.Select(eventTypeMapper.ToName).ToArray());

    public static Regex OneOfEventTypesRegex(params Type[] eventTypes) =>
        OneOfEventTypesRegex(EventTypeMapper.Instance, eventTypes);
    
    public static IEventFilter OneOfEventTypes(params string[] values) =>
        EventTypeFilter.RegularExpression(OneOfEventTypesRegex(values));

    public static IEventFilter OneOfEventTypes(EventTypeMapper eventTypeMapper, params Type[] eventTypes) =>
        EventTypeFilter.RegularExpression(OneOfEventTypesRegex(eventTypeMapper, eventTypes));

    public static IEventFilter OneOfEventTypes(params Type[] eventTypes) =>
        EventTypeFilter.RegularExpression(OneOfEventTypesRegex(eventTypes));
}

And use it as:

var eventTypeMapper = new EventTypeMapper();

eventTypeMapper.AddCustomMap<InvoiceIssued>("InvoiceIssued");
eventTypeMapper.AddCustomMap<UserEmailProvided>("UserEmailProvided");

var filterOptions = EventFilters.OneOfEventTypes(eventTypeMapper, typeof(ShoppingCartOpened));

Or conventional based

var filterOptions = EventFilters.OneOfEventTypes(typeof(InvoiceIssued), typeof(UserEmailProvided));

And that’s all, folks; it wasn’t as hard as we thought it would be!

Still, not to leave you with the impression that Regular Expressions are pleasant, let me finish with an additional example. If I’d like to filter out (so not get) system events AND my specific event called, e.g. CheckpointStored, then the filter would look like this:

public static class EventFilters
{
    public static readonly Regex ExcludeSystemAndCheckpointEventsRegex =
        new(@"^(?!\$)(?!" + Regex.Escape(typeof(CheckpointStored).FullName!) + "$).+");

And I’ll leave you with that, allowing you to think for a moment about how it works (or pasting it to ChatGPT asking for help). How would you generalise to filter out more than one event?

If you’d like to expand and align knowledge around EventStoreDB (or Event Sourcing in general), [email protected]">contact me! Besides blogging, I’m also doing consulting and trainings. Happy to help you and your team!

Cheers!

Oskar

How to automatically setup pgAdmin with a Docker database

Developer experience is a phrase repeated in multiple ways. In our industry, we finally realised how important it is to reduce the cognitive load. As our profession became mainstream, we realised that hacked mode doesn’t scale. As with lean manufacturing, we should cut waste. Waste can mean repetitive tasks that distract us from the work we do.

Of course, I’m not saying we should try to get “in the zone”. Or we should spend hours configuring Vim keybindings and learning all grep params by heart before we can code. No, I don’t believe in 10x developers. What I mean is cutting the annoying papercuts.

Today, I’d like to show you a few tricks on configuring the PostgreSQL local developer environment using Docker Compose.

Let’s start with the basic setup:

version: "3"
services:
    postgres:
        image: postgres:15.1-alpine
        container_name: postgres
        environment:
            - POSTGRES_DB=postgres
            - POSTGRES_USER=postgres
            - POSTGRES_PASSWORD=Password12!
        ports:
            - "5432:5432"

The most basic setup, besides adding two additional configurations through environment variables:

POSTGRES_DB - instructs PostgreSQL container to automatically create a default database with the provided name,
POSTGRES_USER - sets the default username.
POSTGRES_PASSWORD - sets the custom password for the PostgreSQL user.

We could skip those variables if we’d like to use the default one, but let’s make it a bit more spicy in preparing for what comes next.

If we run:

docker-compose up

Then, a new PostgreSQL container will be created and available on the localhost:5432 (as we also exposed this port to the host). We can now try to connect our application.

That’s nice, but what if we’d like to set up a basic database structure, such as a predefined set of tables, indexes, data, etc.? A Default Postgres image also helps here—or actually a Docker convention supported by it.

Most relational databases support a special docker-entrypoint-initdb.d folder. This folder is used to initialise the database automatically when the container is first created. You can put .sql or .sh scripts there, and Docker will automatically. This happens only the first time the container is started, not on subsequent restarts.

Let’s try that and add a basic script called 001-init.sql:

BEGIN;

-- structure setup

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    username VARCHAR(50) NOT NULL,
    email VARCHAR(100) NOT NULL
);

-- data setup

INSERT INTO users (username, email) 
VALUES ('user1', '[email protected]');

INSERT INTO users (username, email) 
VALUES ('user2', '[email protected]');

COMMIT;

It’s a simple script that shows that we can run the script transactionally (see BEGIN and COMMIT). We can also set up a database structure and insert some data.

We could split the script into two files: 001-init-structure.sql and 002-init-data.sql. They will be run in alphabetical order. As you can see, you could even put the sequence of the migration scripts exported from your application there.

Cool, but how do you put it inside the container? We can use volumes for that. Pardon the self-quote, but I wrote in my my other article that:

Volumes enable storing the container data. You can restart the container, and the data will remains. It also allows to mount/bind the host operating system files to the container. It can go both ways, you can send files to the container, but you can also see generated files from the container in the host storage.

How to do it? Simple as that:

version: "3"
services:
    postgres:
        image: postgres:15.1-alpine
        container_name: postgres
        environment:
            - POSTGRES_DB=postgres
            - POSTGRES_USER=postgres
            - POSTGRES_PASSWORD=Password12!
        ports:
            - "5432:5432"
	# VOLUMES CONFIG
        volumes:
            - ./docker/postgres:/docker-entrypoint-initdb.d

Where ./docker/postgres is a local folder path relative to the Docker Compose file. We can put our init files there, and they will be automatically copied to the Docker container during the build and then run on the first run of the Docker container instance. It’s pretty simple and helpful, isn’t it?

Ok, let’s add more spice and show the example of the shell script run on database initialisation. Why would we do it?

As you see, the existing PostgreSQL database container configuration is flexible. But each flexibility has its limits. What if we’d like to set up multiple databases instead of one? The existing setup won’t help. We need to do it on our own. As with any other database, PostgreSQL has its command line. It could be used to set up databases, run SQL scripts, etc.

We could add a new script called 000-create-multiple-postgresql-databases.sh and put there the following script:

#!/bin/bash

set -e
set -u

function create_database() {
	local database=$1
	echo "  Creating Database '$database' for '$POSTGRES_USER'"
	psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" <<-EOSQL
	    CREATE DATABASE $database;
	    GRANT ALL PRIVILEGES ON DATABASE $database TO $POSTGRES_USER;
EOSQL
}

if [ -n "$POSTGRES_MULTIPLE_DATABASES" ]; then
	echo "Multiple database creation requested: $POSTGRES_MULTIPLE_DATABASES"
	for db in $(echo $POSTGRES_MULTIPLE_DATABASES | tr ',' ' '); do
		create_database $db
	done
	echo "Multiple databases created"
fi

It checks if there’s a POSTGRES_MULTIPLE_DATABASES environment variable. If there is, it gets database names by splitting the value by a comma. Then, it runs the create_database function, which creates a new database and grants all permissions to the user provided through the POSTGRES_USER environment variable.

Now, if we put this file into the ./docker/postgres folder, it will be automatically run before our scripts. This will come true because we mapped this folder to the volume, and the scripts are run alphabetically.

We need to change the of POSTGRES_DB into POSTGRES_MULTIPLE_DATABASES:

version: "3"
services:
    postgres:
        image: postgres:15.1-alpine
        container_name: postgres
        environment:
            # UPDATED TO MULTIPLE DATABASES
            - POSTGRES_MULTIPLE_DATABASES="postgres,blogs,auth"
            - POSTGRES_USER=postgres
            - POSTGRES_PASSWORD=Password12!
        ports:
            - "5432:5432"
        volumes:
            - ./docker/postgres:/docker-entrypoint-initdb.d

Now, besides the default postgres database, two more will be created blogs and auth. You can be more creative here and use the shell scripts to customise even more database setup.

So we have a fully set-up database and can connect to it from our application, but wouldn’t it be good to have an IDE to view data?

PostgreSQL has a decent open-source web IDE called pgAdmin. It’s possible to use it as a Docker image. Let’s to it by extending our configuration!

version: "3"
services:
    postgres:
        image: postgres:15.1-alpine
        container_name: postgres
        environment:
            - POSTGRES_MULTIPLE_DATABASES="postgres,blogs,auth"
            - POSTGRES_USER=postgres
            - POSTGRES_PASSWORD=Password12!
        ports:
            - "5432:5432"
        volumes:
            - ./docker/postgres:/docker-entrypoint-initdb.d


    pgadmin:
        container_name: pgadmin_container
        image: dpage/pgadmin4
        environment:
            - PGADMIN_DEFAULT_EMAIL=${PGADMIN_DEFAULT_EMAIL:-[email protected]}
            - PGADMIN_DEFAULT_PASSWORD=${PGADMIN_DEFAULT_PASSWORD:-postgres}
            - PGADMIN_CONFIG_SERVER_MODE=False
            - PGADMIN_CONFIG_MASTER_PASSWORD_REQUIRED=False
        ports:
            - "${PGADMIN_PORT:-5050}:80"
        depends_on:
            - postgres

Let’s discuss those a bit cryptic environment variables setup.

PGADMIN_DEFAULT_EMAIL and PGADMIN_DEFAULT_PASSWORD - Sets the default credentials for the pgAdmin user. pgAdmin can also be hosted as a regular service (e.g. on a test environment) and have a more advanced user setup, but a single user for local development is more than enough.
PGADMIN_CONFIG_SERVER_MODE - determines whether pgAdmin runs in server mode (multi-user) or desktop mode (single-user). We’re setting it to false, so we won’t be prompted for login credentials. This is an annoying papercut we’re removing.
PGADMIN_CONFIG_MASTER_PASSWORD_REQUIRED - controls whether a master password is required to access saved server definitions and other sensitive information. By setting this to false, we skip the additional layer of password protection for server details in pgAdmin.

As you can see, we’re cutting security corners by just configuring the local development environment.

You may have noticed a weird syntax: ${PGADMIN_DEFAULT_EMAIL:[email protected]">[email protected]}. This setup means that if PGADMIN_DEFAULT_EMAIL is defined in the host environment, then its value is used. Otherwise, it will fall back to the default value (in our case [email protected]">[email protected]).

You could pass such variables in the shell:

export PGADMIN_DEFAULT_EMAIL=[email protected]
export PGADMIN_DEFAULT_PASSWORD=securepassword
export PGADMIN_PORT=6666
docker-compose up

Or define the .env file in the same folder as our docker-compose.yml file, and Docker will use it automatically.

[email protected]
PGADMIN_DEFAULT_PASSWORD=securepassword
PGADMIN_PORT=6666

It’s pretty useful for security and changing variables without modifying the main script. Check also Docker Compose Profiles, one the most useful and underrated features to learn more about running the same file with multiple variations.

Getting back to our Docker Compose configuration. If we start containers now, we’ll also get the pgAdmin running on http://localhost:5050. We’ll be automatically logged in, but…

…but we won’t see any database automatically. How come?!

pgAdmin doesn’t do any automatic discovery. We could setup the connection manually, but then we’d need to repeat it each time we clean up our volumes. And that happens often if we’d like to clean our test data in a fresh environment (you can do it by running docker compose down -v).

Let’s change that and set up our server list automatically. Let’s start by defining the servers.json file inside the new docker/pgAdmin folder. And put there:

{
    "Servers": {
        "1": {
            "Group": "Servers",
            "Name": "Docker",
            "Host": "postgres",
            "Port": 5432,
            "MaintenanceDB": "postgres",
            "Username": "postgres", "
            "Password": "Password12!",
            "SSLMode": "prefer",
            "Favorite": true
        }
    }
}

As you can see, we’re just configuring our database. We could define even more if we’d like to by adding “2”: { }” etc.

If you’re generating password randomly (e.g. in the CI/CD) then you can also define passfile and replace “Password”: “Password12!“, with “PassFile”: “/pgpass”,. Then you can keep the server setup intact, but just define the database password inside the passfile. We could put it in the docker/pgAdmin folder and put it inside:

postgres:5432:*:postgres:Password12!

Let’s go on this longer path to show the full setup. We need to adjust a bit our config:

version: "3"
services:
    postgres:
        image: postgres:15.1-alpine
        container_name: postgres
        environment:
            - POSTGRES_MULTIPLE_DATABASES="postgres,blogs,auth"
            - POSTGRES_USER=postgres
            - POSTGRES_PASSWORD=Password12!
        ports:
            - "5432:5432"
        volumes:
            - ./docker/postgres:/docker-entrypoint-initdb.d


    pgadmin:
        container_name: pgadmin_container
        image: dpage/pgadmin4
        environment:
            - PGADMIN_DEFAULT_EMAIL=${PGADMIN_DEFAULT_EMAIL:-[email protected]}
            - PGADMIN_DEFAULT_PASSWORD=${PGADMIN_DEFAULT_PASSWORD:-postgres}
            - PGADMIN_CONFIG_SERVER_MODE=False
            - PGADMIN_CONFIG_MASTER_PASSWORD_REQUIRED=False
        ports:
            - "${PGADMIN_PORT:-5050}:80"
        depends_on:
            - postgres
        user: root
        entrypoint: /bin/sh -c "chmod 600 /pgpass; /entrypoint.sh;"
        volumes:
            - ./docker/pgAdmin/pgpass:/pgpass
            - ./docker/pgAdmin/servers.json:/pgadmin4/servers.json

We defined that our user needs to be root, as we need to be able to set up the proper permissions to the passfile. We can do such a setup by changing the entrypoint:

/bin/sh -c "chmod 600 /pgpass; /entrypoint.sh;"

Entrypoint is used to specify the command executed when the container is started. Here, we’re saying that before running the regular start-up command, run an additional shell script to set proper pgpass permissions using the chmod function.

We’re also mapping volumes to place pgpass and servers.json into proper folders. As you see, we can also map specific files instead of just directories.

Now, if we run the:

docker compose up

We should get pgAdmin with a preconfigured database. Sweet!

Some can say: “Boy that’s a lot of plumbing!”. One can be correct here, but this is the setup you do once and don’t care about. You can put this code inside the project repository and call it a day.

Yet, there’s another option. If you’d prefer to keep it simpler for others to use and customise to your project conventions, you can build your own pgAdmin image embedding the setup.

You could add Dockerfile with the following config (e.g. inside docker/pgAdmin folder):

# Use the official pgAdmin image as the base
FROM dpage/pgadmin4

# Set environment variables
ENV PGADMIN_DEFAULT_EMAIL=${PGADMIN_DEFAULT_EMAIL:[email protected]}
ENV PGADMIN_DEFAULT_PASSWORD=${PGADMIN_DEFAULT_PASSWORD:-postgres}
ENV PGADMIN_CONFIG_SERVER_MODE=False
ENV PGADMIN_CONFIG_MASTER_PASSWORD_REQUIRED=False

# Copy custom configuration files
COPY pgpass /pgpass
COPY servers.json /pgadmin4/servers.json

# Ensure the pgpass file has the correct permissions
RUN chmod 600 /pgpass

# Use root user to allow necessary permissions (as in your original setup)
USER root

# Set the entrypoint to the original entrypoint script
ENTRYPOINT ["/entrypoint.sh"]

# Expose the default port (80 inside the container)
EXPOSE 80

Then you can build it publish it to your container repository, and use it as:

```yml
version: "3"
services:
    postgres:
        image: postgres:15.1-alpine
        container_name: postgres
        environment:
            - POSTGRES_MULTIPLE_DATABASES="postgres,blogs,auth"
            - POSTGRES_USER=postgres
            - POSTGRES_PASSWORD=Password12!
        ports:
            - "5432:5432"
        volumes:
            - ./docker/postgres:/docker-entrypoint-initdb.d

     pgadmin:
        container_name: pgadmin_container
        image: event_driven_io_pgadmin:latest
        ports:
            - "${PGADMIN_PORT:-5050}:80"
        depends_on:
            - postgres

Setting up NGINX load balancer for .NET WebApi

“Just put the load balancer in front of it, and call it a day”. But is it really that simple? Was it ever “just do XYZ?“. I was preparing a new workshop recently. I wanted to show how to load balance Marten Async Daemon - essentially, I wanted to expand the general explanation from my previous article on scaling out Marten. And of course, the 5-minute task of work appeared to be a bit longer.

As I’m writing this blog, not to forget what I learned, so here it is: guidance on how to configure load balancing of ASP.NET Web Api using Nginx and Docker Compose.

Let’s say we have an ASP.NET WebApi running as a Docker container. We’d like to run multiple instances of the same service to distribute the load evenly. The configuration made with the Docker Compose file could look as follows:

version: "3.8"
services:
    backend:
        build:
            dockerfile: Dockerfile
            context: .
            args:
                project_name: Helpdesk.Api
                run_codegen: true
        deploy:
            replicas: 3
        depends_on:
            postgres:
                condition: service_healthy
        restart: always

    postgres:
        image: postgres:15.1-alpine
        container_name: postgres
        healthcheck:
            test: ["CMD-SHELL", "pg_isready -U postgres"]
            interval: 5s
            timeout: 5s
            retries: 5
        environment:
            - POSTGRES_DB=postgres
            - POSTGRES_PASSWORD=Password12!
        ports:
            - "5432:5432"

It’s pretty standard for current applications. We have a WebApi service and related databases. In our case, that’s Postgres, as we’re running Marten as our storage library.

The most critical and non-standard setting the number of replicas:

deploy:
    replicas: 3

Docker Compose allows us to define using the following syntax declaratively how many instances of the defined service we’d like to run. If we now run:

docker compose up -d

And then

docker ps

We should see the four docker containers: 3 with WebAPI and one with PostgreSQL.

CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS                    PORTS                    NAMES
d9f8e7410a84   helpdeskapi-backend    "/bin/sh -c 'dotnet …"   14 minutes ago   Up 1 minute (healthy)                              helpdeskapi-backend-2
399d8643bccd   helpdeskapi-backend    "/bin/sh -c 'dotnet …"   14 minutes ago   Up 1 minute (healthy)                              helpdeskapi-backend-1
90492d1c32bd   helpdeskapi-backend    "/bin/sh -c 'dotnet …"   14 minutes ago   Up 1 minute (healthy)                              helpdeskapi-backend-3
54327a5a155f   postgres:15.1-alpine   "docker-entrypoint.s…"   14 minutes ago   Up 1 minute (healthy)   0.0.0.0:5432->5432/tcp     postgres

It’s essential to note here that we set the explicit name of the Postgres container in Docker Compose by setting container_name. That makes diagnostics easier, as we did above. Yet, we cannot do it for the WebApi service, as we’ll have more than one instance of the same service.

Now, all of our WebApi services will get different IP addresses but the same DNS name. It’ll be the name of the service, so backend. Of course, this DNS name will be only available if we’re inside the Docker internal networking (so between containers, but not in our local/host network). To make them accessible outside and load-balance the traffic, we need a dedicated service that’ll handle that.

We’ll use Nginx, which is one of the most popular tools. It can be used both as a Reverse Proxy and a Load Balancer.

Small reminder:

A reverse proxy is a server that sits between client devices and the backend servers. Its main roles are to forward client requests to appropriate backend servers and send the responses back to the clients.
Load balancing is the process of distributing incoming network traffic across multiple servers. This is crucial for maintaining performance and availability, especially for high-traffic applications.

We’ll start by extending our Docker Compose config with an additional service:

version: "3.8"
services:
    # This is what we added
    nginx:
        restart: always
        image: nginx:alpine
        ports:
            - 8080:80
        volumes:
            - ./nginx.conf:/etc/nginx/nginx.conf
        depends_on:
            - backend

    backend:
        build:
            dockerfile: Dockerfile
            context: .
            args:
                project_name: Helpdesk.Api
                run_codegen: true
        deploy:
            replicas: 3
        depends_on:
            postgres:
                condition: service_healthy
        restart: always

    postgres:
        image: postgres:15.1-alpine
        container_name: postgres
        healthcheck:
            test: ["CMD-SHELL", "pg_isready -U postgres"]
            interval: 5s
            timeout: 5s
            retries: 5
        environment:
            - POSTGRES_DB=postgres
            - POSTGRES_PASSWORD=Password12!
        ports:
            - "5432:5432"

We added a new container running our Nginx load balancer. We’re exposing its default 80 port to a different one on our host (e.g. 8080). It’s a good idea to do this, as many web servers are using port 80, and we’d like to avoid accidental conflicts. We also need to provide the configuration. We’re doing that by mapping the nginx.conf file from the same folder as our Docker Compose file into the configuration inside the Nginx container.

And yes, we need to configure the nginx.conf configuration file. For our ASP.NET service, it should look as follows:

worker_processes auto;

events {
    worker_connections 1024;
}

http {
    map $http_connection $connection_upgrade {
        "~*Upgrade" $http_connection;
        default keep-alive;
    }

    server {
        listen 80;
        location / {
            proxy_pass         http://backend:5248/;
            proxy_http_version 1.1;
            proxy_set_header   Upgrade $http_upgrade;
            proxy_set_header   Connection $connection_upgrade;
            proxy_set_header   Host $host;
            proxy_cache_bypass $http_upgrade;
            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header   X-Forwarded-Proto $scheme;
        }
    }
}

Configuration is short, but there are some things to unpack here.

We’re configuring the HTTP requests redirection. Nginx can also do more, such as TCP (e.g., PostgreSQL load balancing, but that’s not what we’re here for). That’s why we setup HTTP server configuration explicitly.
We’re load-balancing all requests by setting / as location. Nginx is a highly customisable and advanced tool. We could define more advanced rules and patterns, but that’ll be enough for our case.
We’re forwarding all those requests to our backend by defining:

proxy_pass         http://backend:5248/;

Nginx is listening on port 80. We could also put another port address here; the most important thing is to have it aligned with the port redirection in Docker Compose.

Plus, we have a few additional configurations needed:

map $http_connection $connection_upgrade { … } This block configures how Nginx handles connections, particularly differentiating between standard HTTP connections and upgraded WebSocket connections. ASP.NET Core applications, including those exposing APIs, might use WebSockets for real-time communication (e.g., SignalR). The map directive ensures that Nginx correctly upgrades HTTP connections to WebSockets when required, facilitating features like live updates in the UI or interactive API testing. For the same reasons, we also have below proxy_set_header Connection $connection_upgrade and proxy_cache_bypass $http_upgrade to handle WebSockets correctly (forward the upgraded headers and bypass cache, as it’s not applicable to WebSockets).
proxy_http_version 1.1 - Enforces HTTP/1.1 for proxying requests. HTTP/1.1 is necessary for features like keep-alive connections and WebSockets, which can be important for efficiently managing long-lived connections and real-time features in ASP.NET applications and for the Swagger UI’s interactive elements.
proxy_set_header Host $host - This header maintains the original Host header from the client request. ASP.NET applications must receive the original Host header for routing purposes and for the Swagger UI to correctly generate API endpoint URLs matching the client’s request host. For the same reasons we also set proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for and proxy_set_header X-Forwarded-Proto $scheme. The former adds the client’s IP address to the X-Forwarded-For header. The latter sets the protocol (HTTP or HTTPS) the original client uses. They’re critical to enforcing the correct security and generating the correct redirections used by Swagger UI.

We also use the default configs:

worker_processes auto; This directive specifies the number of worker processes Nginx should spawn. Setting it to auto lets Nginx automatically determine the optimal number of worker processes based on the available CPU cores. This improves the server’s performance and efficiency.
events { worker_connections 1024; } Defines the maximum number of simultaneous connections each worker process can handle. In this case, it’s set to 1024 connections per worker.

As you see, there are a lot of settings related to the proper WebSockets configuration and request URI redirections. Those are the trickiest parts, where “just use load balancer” becomes a typical “thank you for nothing” type of advice. To make that work fully, we also need to adjust our ASP.NET configuration. We need to add:

using Microsoft.AspNetCore.HttpOverrides;

var builder = WebApplication.CreateBuilder(args);

// Define the availability on all IPs with the defined port
builder.WebHost.UseUrls("http://*:5248");

// (...)  other configs

// Header forwarding to enable Swagger in Nginx
builder.Services.Configure<ForwardedHeadersOptions>(options =>
{
    options.ForwardedHeaders =
    ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
});

var app = builder.Build();

// (...) other configuration
app.UseSwagger()
   .UseSwaggerUI()
    // Header forwarding to enable Swagger in Nginx
   .UseForwardedHeaders();

I’ve spent a lot of time realising that besides launchSettings.json:

{
    "profiles": {
        "Helpdesk.Api": {
            "commandName": "Project",
            "dotnetRunMessages": true,
            "launchBrowser": true,
            "launchUrl": "swagger/index.html",
            "applicationUrl": "http://localhost:5248",
            "environmentVariables": {
                "ASPNETCORE_ENVIRONMENT": "Development"
            }
        }
    }
}

I also need to replicate the application URL explicitly in the ASP.NET configuration. I also need to specify the wildcard in the URL instead of the localhost or 127.0.0.1. They won’t work out of the box. See also the official guide in the ASP.NET documentation.

In the end, once you know it, then this looks not so hard and kinda makes sense, but yeah, once you know it. Before that, it’s never “just do XYZ”.

If you get to this place, then you may also like my other articles around Docker and Continuous Integration:

Also feel free to [email protected]">contact me! if you think that I could help your project. I’m open on doing consultancy and mentoring to help you speed up and streghten your systems.

Cheers!

Oskar

Combining the To-Do List and the Passage Of Time patterns for resilient business workflows

Managing processes is non-trivial. I have written about it in multiple posts and told you the horror story of the case that should have never happened. Business processes are usually the most critical part of the core functionality, so we need to ensure that we can diagnose them correctly. We also need to ensure that they won’t be stuck in the middle without being able to resume them. At war, love, and managing processes, all tricks are allowed.

Today, I’d like to show you how combining two useful patterns can help you in that:

Passage of Time described by Mathias Verraes
To-do List described by Yves Reynhout.

Let’s say we’re managing the Ordering process described in my other article. We must ensure that once the client confirms the shopping cart, it’ll be paid for and shipped. Even if we design a compensation flow to cancel the payment if the product is unavailable, etc., we might not predict all potential scenarios. There may be transient errors, bugs in the code, and other cases where our flow can get stuck.

In most similar flows, there’s always a deadline for how long the process can be ongoing. For instance, users may have 15 minutes to make a payment, or we may need to complete information about availability. We can also use this deadline for other cases, such as unpredictable ones. Thanks to that, if the order is not completed, it will be cancelled automatically. That’s not perfect, but if it happens rarely, then it’s usually good enough. The user can try to start the ordering process again, or we can take other compensating operations.

There are many ways to do it. Today, we’ll focus on combining the mentioned patterns, as it provides a straightforward way to handle our timeout strategy.

Our ordering process could be reflected by the following set of events:

public abstract record OrderEvent
{
    public record OrderInitiated(
        Guid OrderId,
        Guid ClientId,
        IReadOnlyList<PricedProductItem> ProductItems,
        decimal TotalPrice,
        DateTimeOffset InitiatedAt,
        DateTimeOffset TimeoutAfter
    ): OrderEvent;

    public record OrderPaymentRecorded(
        Guid OrderId,
        Guid PaymentId,
        IReadOnlyList<PricedProductItem> ProductItems,
        decimal Amount,
        DateTimeOffset PaymentRecordedAt
    ): OrderEvent;

    public record OrderCompleted(
        Guid OrderId,
        DateTimeOffset CompletedAt
    );

    public record OrderCancelled(
        Guid OrderId,
        Guid? PaymentId,
        OrderCancellationReason OrderCancellationReason,
        DateTimeOffset CancelledAt
    ): OrderEvent;

    private OrderEvent() { }
}

Based on them, we could define the read model informing us about pending orders:

public class PendingOrder(
    Guid Id,
    DateTimeOffset TimeoutAfter
);

And build it from events:

public class PendingOrdersProjection: SingleStreamProjection<PendingOrder>
{
    public PendingOrdersProjection()
    {
        DeleteEvent<OrderCompleted>();
        DeleteEvent<OrderCancelled>();
    }

    public static PendingOrder Create(OrderInitiated @event, PendingOrder details) =>
        new PendingOrder(@event.OrderId, @event.TimeoutAfter);
}

I’m using Marten projections here, but the syntax doesn’t matter much. More importantly, I’m adding new raw when Order is initiated and removing it upon order completion or cancellation. I’m also storing the information on the desired time-out. This time will be decided in the business logic that creates the OrderInitiated event.

The pending order read model is on a To-Do List. It keeps the information about the orders that must be completed or cancelled. I’m keeping a minimum set of information there. As completion will be handled by a different flow (positive scenario), I only need information about the potential cancellation.

Cool, but how do you handle the cancellation? The common approach is to use scheduled commands. If our tooling lets us do it (as many messaging frameworks do), we can say: “Schedule me a TimeoutOrder command in 15 minutes”. That’s a viable and fine approach, but it’s not perfect. Logically, we’re scheduling an action, that in most cases should not happen, as it’s an edge case. But we’re scheduling it always, for good reasons, but just in case. That’s also pretty vulnerable to any mishaps handling them, as e.g. if it’s muted or not handled correctly then error can be swallowed and not retried. Handling the scheduled command is like plot twist in movies, when killed villian appeared to be alive and striking back. It works, but definitely there are more elegant solutions.

The alternative is Passage of Time pattern. Mathias Verraes wrote about it as:

The mind switch is to think of the passage of time as just another Domain Event, exactly like all the other events. After all, if we define a Domain Event as a granular point in time where something happened that is relevant to the business, then certainly the next business day, month, or quarter, is extremely relevant.

In the new design, a cron or scheduler emits generic Passage of Time Events at regular intervals, such as a DayHasPassed {date} event at midnight, or a QuarterHasPassed {year, quarter}. All interested services listen for this event. They can react to it, by performing an action, by increasing a counter, or by querying some database and filter by date, to find items that have some work to be done.

That’s precisely what we could do in our case. If we had an event as:

public record MinuteHasPassed(
    DateTimeOffset Now,
    DateTimeOffset? PreviousTime
);

Then we could write the following handler:

public class HandleCancelOrder(
    IDocumentSession documentSession,
    TimeProvider timeProvider
):
    IEventHandler<TimeHasPassed>
{
    public async Task Handle(MinuteHasPassed @event, CancellationToken ct)
    {
        var orderIds = await documentSession.Query<PendingOrder>()
            .Where(o => o.TimeoutAfter <= @event.Now)
            .Select(o => o.Id)
            .ToListAsync(token: ct);

        var now = timeProvider.GetUtcNow();

        foreach (var orderId in orderIds)
        {
            await documentSession.GetAndUpdate(
                orderId,
                order => order.Cancel(OrderCancellationReason.TimedOut, now),
                ct: ct
            );
        }
    }
}

We first filter pending orders that should be timed out and get their IDs. Then, we try to cancel each one with the timeout reason (note: it could also be a dedicated method if handling is different from manual cancellation).

public class Order
{
    // (...)
    public OrderCancelled? Cancel(OrderCancellationReason cancellationReason, DateTimeOffset now)
    {
        if (OrderStatus.Closed.HasFlag(Status))
            return null;

        return new OrderCancelled(
            Id, 
            PaymentId,
            cancellationReason,
            now
        );
    }
}

As a result, we’ll store the OrderCancelled event that will trigger removing the pending order from our To-Do List item.

What if the MinuteHasPassed event handling will be delayed? Typically, that’s not a big deal, as timing out doesn’t need to happen at the precise second. It’s typically good enough to happen as soon as possible after a certain time.

What if one of MinuteHasPassed will be lost? Then, the next one will be published in another minute. As mentioned above, it’s not a big deal.

What if cancelling one of the pending orders fails? Not a big deal, as it will be caught again by the next passage of time cadence.

What if a race condition exists between our pending order list and our order write model? Of course, it may happen that someone managed to complete an order at the very last moment. We should silently fail to cancel it, and that’s what we’re doing in business logic by returning a null instead of the event. We should always use Optimistic Concurrency to ensure consistency. That’s also why we’re doing operations on the aggregate instead of just storing the OrderCancelled event. This should also help us if multiple instances handle the same To-Do List and compete for resources. In the worst case, the first succeeds, while the other fails silently.

Also, we can connect multiple To-Do Lists to the same passage of time events. As we’ll be continuously triggering those events, we can do it even with simple in-memory tooling.

All of that creates a simple but powerful and resilient combination.

The example using the Quartz.NET library can look like this: We need to register our jobs with specific schedules (in our case, minute, hour, day).

public static class QuartzExtensions
{
    public static IServiceCollection AddQuartzDefaults(this IServiceCollection services) =>
        services
            .AddQuartz(q => q
                .AddPassageOfTime(TimeUnit.Minute)
                .AddPassageOfTime(TimeUnit.Hour)
                .AddPassageOfTime(TimeUnit.Day)
            )
            .AddQuartzHostedService(q => q.WaitForJobsToComplete = true);

    public static IServiceCollectionQuartzConfigurator AddPassageOfTime(
        this IServiceCollectionQuartzConfigurator q,
        TimeUnit timeUnit
    )
    {
        var jobKey = new JobKey($"PassageOfTimeJob_{timeUnit}");
        q.AddJob<PassageOfTimeJob>(opts => opts.WithIdentity(jobKey));

        q.AddTrigger(opts => opts
            .ForJob(jobKey)
            .WithIdentity($"{jobKey}-trigger")
            .UsingJobData("timeUnit", timeUnit.ToString())
            .WithSimpleSchedule(x => x.WithInterval(timeUnit.ToTimeSpan()))
        );

        return q;
    }
}

Having that, we can we can define our job as:

public class PassageOfTimeJob(IEventBus eventBus, TimeProvider timeProvider): IJob
{
    public Task Execute(IJobExecutionContext context)
    {
        var timeUnit = context.MergedJobDataMap.GetString("timeUnit")!.ToTimeUnit();

        return eventBus.Publish(
            timeUnit.ToEvent(timeProvider.GetUtcNow(), context.PreviousFireTimeUtc),
            CancellationToken.None
        );
    }
}

EventBus is a simple in-memory messaging implementation that will trigger all registered handlers for our event.

As different jobs will need different cadences, we’d like to be able to subscribe to different time passages explicitly. Because of that, we’ll define a dedicated event types:

public abstract record TimeHasPassed(DateTimeOffset Now, DateTimeOffset? PreviousTime)
{
    public record MinuteHasPassed(DateTimeOffset Now, DateTimeOffset? PreviousTime): TimeHasPassed(Now, PreviousTime);

    public record HourHasPassed(DateTimeOffset Now, DateTimeOffset? PreviousTime): TimeHasPassed(Now, PreviousTime);

    public record DayHasPassed(DateTimeOffset Now, DateTimeOffset? PreviousTime): TimeHasPassed(Now, PreviousTime);
}

You could go wild and customise that more, but I’m sure you’re getting the point.

The final thing is to define our TimeUnit and the helper conversion methods:

public enum TimeUnit
{
    Minute,
    Hour,
    Day
}

public static class TimeUnitExtensions
{
    public static TimeUnit ToTimeUnit(this string timeUnitString) =>
        Enum.Parse<TimeUnit>(timeUnitString);

    public static TimeSpan ToTimeSpan(this TimeUnit timeUnit) =>
        timeUnit switch
        {
            TimeUnit.Minute => TimeSpan.FromMinutes(1),
            TimeUnit.Hour => TimeSpan.FromHours(1),
            TimeUnit.Day => TimeSpan.FromDays(1),
            _ => throw new ArgumentOutOfRangeException(nameof(timeUnit), $"Not expected time unit value: {timeUnit}")
        };

    public static TimeHasPassed ToEvent(this TimeUnit timeUnit, DateTimeOffset now, DateTimeOffset? previous) =>
        timeUnit switch
        {
            TimeUnit.Minute => new MinuteHasPassed(now, previous),
            TimeUnit.Hour => new HourHasPassed(now, previous),
            TimeUnit.Day => new DayHasPassed(now, previous),
            _ => throw new ArgumentOutOfRangeException(nameof(timeUnit), $"Not expected time unit value: {timeUnit}")
        };
}

Is it always a good solution? Only Siths deal in absolutes. This can swamp your database if you have many To-Do Lists with many records. Especially if you add the passage of time in seconds or milliseconds. That’s why I made the To-Do List data so minimal and ensured that items would be cleared right after the order was completed or cancelled. If you want to create a detailed view, don’t reuse it with the To-Do List; create a dedicated read model.

If you have many tasks that need to be scheduled quickly and tight performance requirements, scheduled messages may be a better option. But as always, don’t assume. Ask for metrics, measure, compare, and then make the final decision.

The combination of the To-Do List and the Passage of Time patterns is simple but powerful. In most systems, processes/checks need to happen every day, every hour, or every minute. This combination allows us to build a loosely coupled, scalable and resilient processing. Of course, we should not replace the classical process handling, but it should be a good fit for features like time-based deadlines or triggers.

If you liked this article and would like to expand that, I do also paid one-to-one mentoring sessions, consulting and group workshops. Don’t hesitate to [email protected]">contact me! if you’re interested!

Read also more in:

Cheers!

Oskar

Let's build the worst Event Sourcing system!

Everyone likes to talk about best practices. I went the other way around and gathered all the worst practices on how to build the worst Event Sourcing system. Was it easy?

It was not, as building the worst Event Sourcing system, we cannot cheat and just go with Event Streaming rebranding the name. What to do next?

Check the recording of my talk at NDCLondon 2024 where I told the story of the project Franz: the worst Event Sourcing system.

Jokes aside. Event Sourcing is perceived as a complex pattern that’s challenging to learn. In fact, it’s pretty simple, but the common misconception and the way it’s taught may lead to such a conclusion. By going through the worst ideas, I wanted to show the essence of Event Sourcing. I went through the massive set of internet articles and gathered the worst of them. Unfortunately, this wasn’t hard, as there were a lot of misconceptions spread around.

That was probably the hardest talk I gave so far from the presenting technique. I tried to be funny but not silly. The narrative has a few twists and is a bit inverted. But well, I think that not only attendees but also the speaker need to have some fun from time to time!

If you want to know how to do it and get a quick start, then check my NDC Oslo talk:

Or articles on this blog like Event Streaming is not Event Sourcing! where I tackled the most common misconception.

If you need more one-to-one help [email protected]">contact me!, I’m open for consultancy. Check also my workshops page for more end-to-end learning opportunity.

Cheers!

Oskar

Why you should batch message processing and how to do it with .NET AsyncEnumerable

AsyncEnumerable is a sneaky abstraction. It allows simplified and performant usage for iterating on pull-based and push-based sources.

“Pull-based and push-based sources” sound smart, but what are they?

Pull-based approach is when we’re querying a specific data source, such as, running a select statement on a relational database. We pull the data from it.

Push-based is when we’re getting notifications from an external source. It can be a messaging system, event store subscription, or also change notifications from the database. We don’t need to query for information; we’ll be notified.

How does AsyncEnumerable help in both approaches? For push-based, it’s pretty obvious. We can use a familiar foreach statement, and the AsyncEnumerable implementation will internally forward the notification and wait until the new one appears. For instance, if you use API for Postgres Logical Replication, you can do it as:

var (connectionString, slotName, publicationName) = options;
await using var conn = new LogicalReplicationConnection(connectionString);
await conn.Open(ct);

var slot = new PgOutputReplicationSlot(slotName);

await foreach (var message in conn.StartReplication(slot, new PgOutputReplicationOptions(publicationName, 1), ct))
{
    if (message is InsertMessage insertMessage)
    {
        yield return await InsertMessageHandler.Handle(insertMessage, ct);
    }

    conn.SetReplicationStatus(message.WalEnd);
    await conn.SendStatusUpdate(ct);
}

For a push-based approach, AsyncEnumerable (even if it’s not asynchronous) is also useful. Let’s say you’re using the EventStoreDB API to read events from the stream. Most of the time, streams will be short, as they should represent the history of a particular entity/process. Most of the entities in our system don’t have many operations happening. Still, sometimes we can have more events. Loading all of them in one batch (think: reading them with ToListAsync) won’t be efficient. It’ll increase the memory pressure if we load all of them at once. If we’re building the state from events, we don’t need to load all of them. It’s enough just to have a single one, apply it to the state, and get the next event.

That’s also a reason why EventStoreDB Api for reading streams is returning AsyncEnumerable to give you the option to load into memory just a subset of events, reducing memory pressure. Thanks to that, you can write the efficient state aggregation as:

public static async Task<T?> AggregateStream<T>(
    this EventStoreClient eventStore,
    Guid id,
    CancellationToken cancellationToken,
    ulong? fromVersion = null
) where T : class, IProjection
{
    var readResult = eventStore.ReadStreamAsync(
        Direction.Forwards,
        StreamNameMapper.ToStreamId<T>(id),
        fromVersion ?? StreamPosition.Start,
        cancellationToken: cancellationToken
    );

    if (await readResult.ReadState.ConfigureAwait(false) == ReadState.StreamNotFound)
        return null;

    var aggregate = (T)Activator.CreateInstance(typeof(T), true)!;

    await foreach (var @event in readResult)
    {
        var eventData = @event.Deserialize();

        aggregate.Apply(eventData!);
    }

    return aggregate;
}

That’s great, but having the API that always returns you a single event has also limitations. One of them is performance.

Let’s say we’re handling upcoming events from a messaging system or event store subscription and updating read models based on them. Trying to update them after each event will create a lot of network traffic between our system and the database. I explained in the other article how essential it is for a database like Elasticsearch.

The potential solution is to do batching. We could gather upcoming events into a collection and run updates in batches. But that’s also tricky, as events won’t come in the unified batch sizes. It may happen that you got 5 events, then 2 events one minute later, and 100 events 5 minutes later. The notification cadence will depend highly on the traffic that creates those events. If we set our batch to have 10 events, we’d need to wait 7 minutes to process them (until the batch is fully filled). That’s not great and, in most cases, not acceptable. It could only work if our traffic is extremely stable and continuous. We need to do better than that.

To make batching efficient, we should also define the deadline for the batches. We should say: “Either try to gather batch of size X or wait Y milliseconds to process it. Whetever happens first.”.

Unfortunately, AsyncEnumerable doesn’t provide us with the API for that. Let’s discuss how to fix it!

In .NET we have competing API for handling asynchronous processing: System.Threading.Channels. They’re great but a bit harder for some people to reason for. We’re no longer operative on a straightforward imperative programming model but making things reactive. (Read more on the usefulness of .NET channels in my other article). If we were using .NET channels, we could use the nice contrib package Open.ChannelExtensions and do things like:

var channel = Channel.CreateUnbounded<T>();

var batchingReader = channel.Reader
    .Batch(batchSize)
    .WithTimeout(deadline);

We’re creating a channel and reader that’d precisely do what we’d like to achieve with AsyncEnumerable, so batching by size or deadline. We could produce new data by writing to the channel as:

await channel.Writer.WriteAsync(@event, ct);

Cool! But we want to do this with AsyncEnumerable. Maybe we could wrap this code somehow? Yes, we can!

    public static IAsyncEnumerable<List<T>> Batch<T>(
        this IAsyncEnumerable<T> enumerable,
        int batchSize,
        TimeSpan deadline,
        CancellationToken ct
    ) =>
        enumerable
            .ToChannel(cancellationToken: ct)
            .Batch(batchSize)
            .WithTimeout(deadline)
            .AsAsyncEnumerable(cancellationToken: ct);

Yup, simple as that! Open.ChannelExtensions provides an API to forward AsyncEnumerable to the .NET channel and vice versa. Thanks to that, we’ve built a reusable method for use in any AsyncEnumerable. We could use it in the presented above logical replication, or for instance event store subscription:

var subscription = eventStoreClient
    .SubscribeToAll(FromAll.Start, cancellationToken: ct)
    .Batch(100, TimeSpan.FromMilliseconds(150), ct);

await foreach (var events in subscription)
{
    await HandleBatch(events, ct);
}

Of course, setting a proper batch size and deadline is not easy and is highly contextual to your use case. That’s something that should be fine-tuned or even made adaptive.

As you can see, it’s not that scary, and the code is simple. But all is simple if you’re aware of the issue and already know how to solve it. I hope that this code will help you to make your async code more efficient and performant.

If you need more help [email protected]">contact me!, I’m open for consultancy. Check also my workshops page for more end-to-end learning opportunity.

Cheers!

Oskar

Docker Compose Profiles, one the most useful and underrated features

Erik Shafer asked me on the Emmett Discord if I could provide a sample of how to run the WebApi application using Emmett. Of course, I said: sure will! I already had WebApi sample in the repository I also explained here How to build and push Docker image with GitHub actions?. Easy peasy, then, right?

Indeed, it wasn’t that hard. Of course, I had to fight with ESModules quirks, but I also expanded the scope a bit, as I also decided to use one of the most underrated Docker Compose features: Profiles.

What are Docker Compose Profiles?

They’re a way to logically group services inside your Docker Compose definition, allowing you to run a subset of services. My original Docker Compose definition contained the EventStoreDB startup, which I use in my Emmett samples as the real event store example.

version: '3.5'

services:
  eventstoredb:
    image: eventstore/eventstore:23.10.0-bookworm-slim
    container_name: eventstoredb
    environment:
      - EVENTSTORE_CLUSTER_SIZE=1
      - EVENTSTORE_RUN_PROJECTIONS=All
      - EVENTSTORE_START_STANDARD_PROJECTIONS=true
      - EVENTSTORE_EXT_TCP_PORT=1113
      - EVENTSTORE_HTTP_PORT=2113
      - EVENTSTORE_INSECURE=true
      - EVENTSTORE_ENABLE_EXTERNAL_TCP=true
      - EVENTSTORE_ENABLE_ATOM_PUB_OVER_HTTP=true
    ports:
      - '1113:1113'
      - '2113:2113'
    volumes:
      - type: volume
        source: eventstore-volume-data
        target: /var/lib/eventstore
      - type: volume
        source: eventstore-volume-logs
        target: /var/log/eventstore
    networks:
      - esdb_network

networks:
  esdb_network:
    driver: bridge

volumes:
  eventstore-volume-data:
  eventstore-volume-logs:

Nothing fancy here so far. You can just run it with:

docker compose up

It will start the database; then, you can run a sample application with

npm run start

And play with Emmett.

I wanted to keep the sample experience straightforward and use local development/debugging as the default. Docker image build and run would be optional (we could call it “Erik’s mode”!).

Now, profiles come in handy here, as they enable that, I just had to add:

version: '3.5'

services:
  app:
    build:
      dockerfile: Dockerfile
      context: .
    container_name: emmett_api
    profiles: [app]
    environment:
      - ESDB_CONNECTION_STRING=esdb://eventstoredb:2113?tls=false
    networks:
      - esdb_network
    ports:
      - '3000:3000'

  # (...) EventStoreDB Definition

The setup is pretty straightforward.

We’re stating which Docker file to use and where it is located.

    build:
      dockerfile: Dockerfile
      context: .

Used . means that the build context will be the folder as the location of the docker-compose file. dockerfile tells where the Docker file is located. In our case, it’s the same folder as the docker-compose file, and it’s named Dockerfile. That also opens more options. We could also define it as:

    build:
      dockerfile: src/app/Dockerfile
      context: .

That’d allow us to use some common dependencies outside the project folder, e.g. src/build. Essentially, we could expand the image-building process’s access to additional locations and allow project files to access parent folders to use, for instance, shared configurations or dependencies. Thanks go to Jakub Gutkowski for pointing that out!.

We ensure that we have a connection to EventStoreDB by placing it in the same network and passing the connection string as an environment variable.

    environment:
      - ESDB_CONNECTION_STRING=esdb://eventstoredb:2113?tls=false
    networks:
      - esdb_network

The new thing is the profile definition:

    profiles: [app]

Thanks to that, we’re saying that this service will only be used if we explicitly specify that in the command line. We can, for instance build the image by running

docker compose --profile app build

Or run both EventStoreDB and Emmett WebApi by calling:

docker compose --profile app up

And let’s stop here for a moment! Why both if I specified the app profile? Docker Compose will run, in this case, specified profile AND all services that don’t have a profile specified. That’s quite neat, as we can define the set of default services (e.g. databases, messaging systems, etc.) and add others as optional. Ah, and you can specify multiple profiles by, e.g.:

docker compose --profile backend --profile frontend up

You can also group multiple services into a single profile. Why would you do it? Let’s go to…

Docker Profiles advanced scenario

In my Event Sourcing .NET samples repository, I’m trying to cover multiple aspects, tools, ways to build Event Sourcing, CQRS and Event-Driven systems. I’m using:

Marten (so Postgres) and EventStoreDB as example event stores,
Postgres and Elasticsearch as read model stores,
Kafka used for integration between services,
UI tools like PgAdmin and Kafka UI to easier investigate sample data.

Multiple samples are using those services in various configurations.

I’m also using them in my Event Sourcing workshops, so I’d like to ensure the setup is smooth and we can focus on learning and not fighting with Docker.

Initially, I kept multiple Docker Compose files for:

default configuration with all services,
continuous integration pipeline configuration without UI components, as they’re not needed for tests. They’d just eat resources and make pipeline runs longer. They also don’t have Kafka, as I’m just testing inner modules functionalities,
sample web API Docker Image build (similar to the one explained above),
only containing Postgres-related configurations,
accordingly, only with EventStoreDB,
etc.

I’m sure that you can relate that to your projects. Now, how can Docker Compose Profiles help us with that? It could definitely help us merge multiple configurations into one and easier manage updating versions, etc.

Let’s see the config I ended up with and then explain the reasoning. I’ll trim the detailed service configuration; you can check the whole file here.

version: "3"
services:
    #######################################################
    #  Postgres
    #######################################################
    postgres:
        profiles: [ postgres, postgres-all, all, all-no-ui, ci ]
        image: postgres:15.1-alpine
	      # (...) rest of the config

    pgadmin:
        profiles: [ postgres-ui, postgres-all, all ]
        image: dpage/pgadmin4
	      # (...) rest of the config

    #######################################################
    #  EventStoreDB
    #######################################################
    eventstore.db:
        image: eventstore/eventstore:23.10.0-bookworm-slim
        profiles: [ eventstoredb, eventstoredb-all, all, all-no-ui, ci ]
	      # (...) rest of the config

    #######################################################
    #  Elastic Search
    #######################################################
    elasticsearch:
        image: docker.elastic.co/elasticsearch/elasticsearch:8.13.2
        profiles: [ elastic, elastic-all, all, all-no-ui, ci ]
	      # (...) rest of the config

    kibana:
        image: docker.elastic.co/kibana/kibana:8.13.2
        profiles: [ elastic-ui, elastic-all, all ]
	      # (...) rest of the config

    #######################################################
    #  Kafka
    #######################################################
    kafka:
        image: confluentinc/confluent-local:7.6.1
        profiles: [kafka, kafka-all, all, all-no-ui]
	      # (...) rest of the config

    init-kafka:
        image: confluentinc/confluent-local:7.6.1
        profiles: [ kafka, kafka-all, all, all-no-ui ]
        command: "#shell script to setup Kafka topics"
	      # (...) rest of the config
        
    schema_registry:
        image: confluentinc/cp-schema-registry:7.6.1
        profiles: [ kafka-ui, kafka-all, all ]        
	      # (...) rest of the config

    kafka_topics_ui:
        image: provectuslabs/kafka-ui:latest
        profiles: [ kafka-ui, kafka-all, all ]
        depends_on:
            - kafka
	      # (...) rest of the config

    #######################################################
    #  Open Telemetry
    #######################################################
    jaeger:
        image: jaegertracing/all-in-one:latest
        profiles: [ otel, otel-all, all ]
	      # (...) rest of the config

    #######################################################
    #  Test Backend Service
    #######################################################
    backend:
        build:
            dockerfile: Dockerfile
            context: .
        profiles: [build]
	    # (...) rest of the config

## (...) Network and Volumes config

As you see, we have a few general profiles:

postgres
elastic
kafka
eventstoredb
otel
build

They group the needed tooling containers.

Each of them has the additional profiles with prefixes:

{profile}-all (e.g. postgres-all) - will start all needed tooling containers plus supportive like ui,
{profile}-all-no-ui - will start just the needed tooling without UI components. There’s no {profile}-all-ui, as starting UI without actual components doesn’t make sense.

I also defined additional profiles:

all - that’ll run all components,
ci - only components needed for the CI pipeline (so no UI and Kafka).

So by default, if I don’t mind my RAM being eaten by all containers, I’d run:

docker compose --profile all up

If I’d like to run the Marten sample with Elasticsearch read models, I could just run:

docker compose --profile postgres --profile elastic up

In the CI, I can run:

docker compose --profile ci up

It’s important to find balance and conventions for profile names. If you have too many of them, it’ll be challenging for people to memorise all of them. That’s why grouping them and adding standard conventions can be helpful. We should always consider intended usage and make it accessible. I could potentially provide profiles for dedicated samples.

Read more in the official Docker Compose Profiles guide.

See also the Pull Requests where I introduced explained changes to:

If you get to this place, then you may also like my other articles around Docker and Continuous Integration:

Cheers!

Oskar

How to write a left-fold streams collector in Java

Last week, we covered the latest improvements to Java 22 around pattern matching and records. They enable explicit business logic modelling, making it concise and guarded by the compiler. As usual, I put it into the context of Event Sourcing.

We ended it with two functions. First one evolveing the state based on the event:

public sealed interface ShoppingCart {
  // (...)

  static ShoppingCart evolve(ShoppingCart state, Event event) {
    return switch (when(state, event)) {
      case When(Initial _, Opened _) ->
        new Pending(ProductItems.empty());

      case When(
        Pending(var productItems),
        ProductItemAdded(_, var productItem, _)
      ) -> new Pending(productItems.add(productItem));

      case When(
        Pending(var productItems),
        ProductItemRemoved(_, var productItem, _)
      ) -> new Pending(productItems.remove(productItem));

      case When(Pending _, Confirmed _),
           When(Pending _, Canceled _) -> new Closed();

      default -> state;
    };
  }

The other taking an array of events into the final state:

public sealed interface ShoppingCart {
  record Initial() implements ShoppingCart {
  }

  record Pending(ProductItems ProductItems) implements ShoppingCart {
  }

  record Closed() implements ShoppingCart {

  }

  static  ShoppingCart getShoppingCart(Event[] events) {
    ShoppingCart state = new Initial();

    for (var event : events) {
      state = evolve(state, event);
    }
    return state;
  }
  // (...)
}

Essentially, we’re:

Reading the list of events.
Creating the initial state.
We’re evolving the new state by applying the events to the current state for each event.
We repeat that for each event, getting the final state.

This process is called Left Fold/Aggregation/Reduce (depending on who phrase it).

That’s fine, but some hip kids don’t like to create such functions for each state. Cool kids of Java like streams. At least, that’s what I heard. Wanna be like them?

Wouldn’t it be nice if we could do something like:

public sealed interface ShoppingCartService {
  // (...)
  static ShoppingCart getShoppingCart(EventStore eventStore, UUID shoppingCartId) {
    var events = eventStore.readStream(ShoppingCart.Event.class, shoppingCartId);

    return events.stream()
      .collect(foldLeft(ShoppingCart.Initial::new, ShoppingCart::evolve));
  }
}

That’s possible but not as easy as it seems. There is no left-fold collector that’d aggregate values on top of the element (state) respecting the order. You can’t use standard Java Stream collectors like Collectors.toList() or Collectors.toMap() for this. Streams API is built to enable both parallel and sequential processing, and that’s great for reactive processing but not great for our case, where we don’t need to parallelize that but just iterate the aggregating state. Knowing that we can simplify the processing, but to do that, we still need to go custom.

Let’s try, then and build a custom stream collector! Being a cool kid is not only about perks; it also requires some work upfront!

Let’s see first the whole implementation and tackle its pieces one by one:

public class FoldLeft<Entity, Event> implements Collector<Event, AtomicReference<Entity>, Entity> {
  private final Supplier<Entity> getInitial;
  private final BiFunction<Entity, Event, Entity> evolve;

  public FoldLeft(Supplier<Entity> getInitial, BiFunction<Entity, Event, Entity> evolve) {
    this.getInitial = getInitial;
    this.evolve = evolve;
  }

  public static <Entity, Event> FoldLeft<Entity, Event> foldLeft(
    Supplier<Entity> getInitial,
    BiFunction<Entity, Event, Entity> evolve
  ) {
    return new FoldLeft<>(getInitial, evolve);
  }


  @Override
  public Supplier<AtomicReference<Entity>> supplier() {
    return () -> new AtomicReference<>(getInitial.get());
  }

  @Override
  public BiConsumer<AtomicReference<Entity>, Event> accumulator() {
    return (wrapper, event) -> wrapper.set(evolve.apply(wrapper.get(), event));
  }

  @Override
  public BinaryOperator<AtomicReference<Entity>> combiner() {
    return (left, right) -> {
      left.set(right.get());
      return left;
    };
  }

  @Override
  public Function<AtomicReference<Entity>, Entity> finisher() {
    return AtomicReference::get;
  }

  @Override
  public Set<Characteristics> characteristics() {
    return new HashSet<>();
  }
}

It’s not that much code, but it still can be a bit confusing if that’s the first time we’re implementing a custom collector. That’s fine, as we probably should not be doing that too often!

The custom collector, you need the following ingredients:

Supplier: Provides an initial value for the accumulation (an empty shopping cart in your case).
Accumulator: Applies an event to the current state of the shopping cart, evolving the state based on the event type.
Combiner: Merges two states, which is crucial for parallel processing. In our case, it’s just a simple value replacement. We’re saying that the evolved state is now the current accumulated one.
Finisher: Extracts the final state from the accumulator’s container.
Characteristics: Provide hints to the implementation about the collector’s characteristics, such as whether it is CONCURRENT or UNORDERED. In our case, we return an empty set, implying none of these characteristics are explicitly claimed.

We’re starting by stating that we’re defining the custom collector by implementing Collector<Event, AtomicReference, Entity>

public class FoldLeft<Entity, Event> implements Collector<Event, AtomicReference<Entity>, Entity> {
  // (...)
}

It means that our stream will contain values of Event type and we’re expecting instance of Entity as a result. We’ll be accumulating values to AtomicReference, ensuring that updates to the state are thread-safe, which is crucial if your stream processing might be parallelised.

Then, we’re providing functions that will return the initial state and the way to evolve it based on events. I added a static method to make usage less verbose.

private final Supplier<Entity> getInitial;
private final BiFunction<Entity, Event, Entity> evolve;

public FoldLeft(Supplier<Entity> getInitial, BiFunction<Entity, Event, Entity> evolve) {
  this.getInitial = getInitial;
  this.evolve = evolve;
}

public static <Entity, Event> FoldLeft<Entity, Event> foldLeft(
  Supplier<Entity> getInitial,
  BiFunction<Entity, Event, Entity> evolve
) {
  return new FoldLeft<>(getInitial, evolve);
}

Then, we’re overriding all the methods. Supplier with the initial state.

@Override
public Supplier<AtomicReference<Entity>> supplier() {
  return () -> new AtomicReference<>(getInitial.get());
}

Accumulator with a function how to evolve state:

@Override
public BiConsumer<AtomicReference<Entity>, Event> accumulator() {
  return (wrapper, event) -> wrapper.set(evolve.apply(wrapper.get(), event));
}

Combiner that’ll set the new (evolved) state value:

@Override
public BinaryOperator<AtomicReference<Entity>> combiner() {
  return (left, right) -> {
    left.set(right.get());
    return left;
  };
}

Finisher returns the final value, taking the value from our atomic reference wrapper.

@Override
public Function<AtomicReference<Entity>, Entity> finisher() {
  return AtomicReference::get;
}

And boom, we created our own reusable left-fold collector, which we can use elsewhere for different entity types or even not for Event Sourcing!

Cheers!

Oskar

This is not your uncle's Java! Modelling with Java 22 records pattern matching in practice

I like learning new things. It stimulates my creativity, helps me gain diverse perspectives, and helps me be humble. When you’re a notorious debutant, you learn to appreciate small stuff and simplicity, the power of ignorance. It shows that if you’re down the rabbit hole, then this works both ways. Not many people could go the same way, but it also takes time to get out of that and embrace the outside world.

Still, I not only like to scratch the surface but also learn idiomatic ways and check how far I can go. Some time ago, I got back to doing more Java, and it was a good move. This is not only because it appears that many of my clients after I went solo, are from JVM land but also because the language is changing rapidly. And those changes are made quickly and with a proper dose of consideration. I like reading JDK Enhancements Proposals. Even if you don’t like Java, they’re written in a really accessible way, so it’s worth reading them to learn more about decision-making and see how languages and environments are evolving.

This is not your uncle’s Java anymore!

If you’re a frequent reader of my blog, you may have noticed that I use the Shopping Cart example frequently. And that’s intentional; I think it’s worth not doing it all at once. If I’m learning new technical aspects, I wouldn’t like to crunch the business domain simultaneously. That also works the other way; it’s safer to use a boring tech stack if I’m diving into a new domain. That reduces cognitive load.

In my opinion, one of the best ways to learn a new technology or language is to have some sort of Kata. So, the topic we discussed many times is that we don’t need to think about mechanics. For me, Kata is a shopping cart. It’s straightforward enough, so I don’t need to think too much about the domain, but it shows the common cases that can happen during implementation (consistency, concurrency, nested data, integration with other flows). It allows me to dive a bit deeper if I’d like to, e.g. to analyse integrations with other components, high traffic during Black Friday etc.

Still, getting back to not-your-uncle Java. Let’s discuss the latest records and pattern-matching enhancements added in Java 22. As mentioned, we’ll use my event-sourced shopping cart Kata. Please check How to effectively compose your business logic if you want to learn more about the domain.

Let’s start this time with the end result and then explain what actually happened.

public class ShoppingCartDecider {
  public static ShoppingCart.Event decide(Command command, ShoppingCart state) {
    return switch (on(state, command)) {
      case On(Initial _, Open(var id, var clientId, var now)) ->
        new Opened(id, clientId, now);

      case On(
        Pending _,
        AddProductItem(var id, var productItem, var now)
      ) -> new ProductItemAdded(id, productItem, now);

      case On(
        Pending(var productItems),
        RemoveProductItem(var id, var productItem, var now)
      ) -> {
        if (!productItems.hasEnough(productItem))
          throw new IllegalStateException("Not enough product items to remove");

        yield new ProductItemRemoved(id, productItem, now);
      }

      case On(Pending _, Confirm(var id, var now)) ->
        new Confirmed(id, now);

      case On(Pending _, Cancel(var id, var now)) ->
        new Canceled(id, now);

      default -> throw new IllegalStateException(
        String.format("Cannot %s on %s", command.getClass().getName(), state.getClass().getName())
      );
    };
  }
}

This method is responsible for making business decisions on our shopping cart. We can:

open a new shopping cart (if it wasn’t already opened),
add product item to not closed (so not confirmed or cancelled) shopping cart,
remove products if we added them already,
confirm or cancel the pending cart.

Our function takes the current state and the command representing our intention. Both of them are immutable objects defined as union types.

Shopping Cart can be represented by the following states:

public sealed interface ShoppingCart {
  record Initial() implements ShoppingCart {
  }

  record Pending(ProductItems ProductItems) implements ShoppingCart {
  }

  record Closed() implements ShoppingCart {
  }
}

I’m using sealed interface here, which allows me to say that only those three states can represent Shopping Carts. What’s more, that’s also an enabler for advanced pattern matching. Notice that states contain only information that’s needed for the business logic. To make it more focused, we’re outsourcing the logic to ProductItems value object, which is defined as:

public class ProductItems {
  Map<String, Integer> values;

  private ProductItems(Map<String, Integer> values) {
    this.values = values;
  }

  public static ProductItems empty() {
    return new ProductItems(new HashMap<>());
  }

  public ProductItems add(PricedProductItem productItem) {
    var newValues = new HashMap<>(values);

    newValues.compute(key((productItem)), (_, currentQuantity) ->
      Optional.ofNullable(currentQuantity).orElse(0) + productItem.quantity
    );

    return new ProductItems(newValues);
  }

  public ProductItems remove(PricedProductItem productItem) {
    var newValues = new HashMap<>(values);

    newValues.compute(key((productItem)), (_, currentQuantity) ->
      Optional.ofNullable(currentQuantity).orElse(0) - productItem.quantity
    );

    return new ProductItems(newValues);
  }

  public boolean hasEnough(PricedProductItem productItem) {
    var currentQuantity = values.getOrDefault(key(productItem), 0);

    return currentQuantity >= productItem.quantity();
  }


  private static String key(PricedProductItem pricedProductItem) {
    return String.format("%s_%s", pricedProductItem.productId, pricedProductItem.unitPrice());
  }

  public record PricedProductItem(
    UUID productId,
    int quantity,
    double unitPrice
  ) {
  }
}

It’s also an immutable object but modelled as a class not to expose internal information. Product Item is represented by its product id and unit price. We can have the same product at different prices (e.g., some have a discount applied). To simplify business logic, we don’t need to maintain a list of objects; it’s fine to use a map where the key is the identifier built from product id unit price, and value quantity. We allow negative quantity to simplify processing, as we’ll check that in the business logic through the exposed hasEnough method.

Speaking about the business logic, we can define the API as the following set of commands:

public class ShoppingCartDecider {
  public sealed interface Command {
    record Open(
      UUID shoppingCartId,
      UUID clientId,
      OffsetDateTime now
    ) implements Command {
    }

    record AddProductItem(
      UUID shoppingCartId,
      PricedProductItem productItem,
      OffsetDateTime now
    ) implements Command {
    }

    record RemoveProductItem(
      UUID shoppingCartId,
      PricedProductItem productItem,
      OffsetDateTime now
    ) implements Command {
    }

    record Confirm(
      UUID shoppingCartId,
      OffsetDateTime now
    ) implements Command {
    }

    record Cancel(
      UUID shoppingCartId,
      OffsetDateTime now
    ) implements Command {
    }
  }
}

I define the Command type without prefix, as having it nested gives enough information about the context. That’s also why I don’t use suffixes like CancelShoppingCart but just name it Cancel.

In Event Sourcing outcome of the business logic is event or set of events, let’s define them. We can do it accordingly to commands, and put them inside Shopping Cart class, as they’re strictly related to shopping cart lifetime.

public sealed interface ShoppingCart {
  // (...)

  sealed interface Event {
    record Opened(
      UUID shoppingCartId,
      UUID clientId,
      OffsetDateTime openedAt
    ) implements Event {
    }

    record ProductItemAdded(
      UUID shoppingCartId,
      ProductItems.PricedProductItem productItem,
      OffsetDateTime addedAt
    ) implements Event {
    }

    record ProductItemRemoved(
      UUID shoppingCartId,
      ProductItems.PricedProductItem productItem,
      OffsetDateTime removedAt
    ) implements Event {
    }

    record Confirmed(
      UUID shoppingCartId,
      OffsetDateTime confirmedAt
    ) implements Event {
    }

    record Canceled(
      UUID shoppingCartId,
      OffsetDateTime canceledAt
    ) implements Event {
    }
  }
}

Each event is appended to the stream. An event stream is a history of the record. It keeps all results (facts) of what has happened. It can look as follow:

Id: "shopping_cart-1294f9"

Events:
1. Opened
2. ProductItemAdded
3. ProductItemRemoved
4. ProductItemAdded
5. Confirmed

When we want to run the next decision, we need to build the current state from events. We read all of them and apply them one after another, evolving the state into its final form. You can do it the following way:

ShoppingCart getShoppingCart(Event[] events) {
  ShoppingCart state = new Initial();
    
  for (var event : events) {
    state = evolve(state, event);
  }
  return state;
}

What would the evolve function look like? Similarly to the decide presented first:

public sealed interface ShoppingCart {
  // (...)

  static ShoppingCart evolve(ShoppingCart state, Event event) {
    return switch (when(state, event)) {
      case When(Initial _, Opened _) ->
        new Pending(ProductItems.empty());

      case When(
        Pending(var productItems),
        ProductItemAdded(_, var productItem, _)
      ) -> new Pending(productItems.add(productItem));

      case When(
        Pending(var productItems),
        ProductItemRemoved(_, var productItem, _)
      ) -> new Pending(productItems.remove(productItem));

      case When(Pending _, Confirmed _),
           When(Pending _, Canceled _) -> new Closed();

      default -> state;
    };
  }

We’re defining the expected state transitions/evolutions; we just return the state for other cases. Why am I not throwing an exception here? Read more in Should you throw an exception when rebuilding the state from events?. Check also the follow-up article showing how to do make it generic using a custom streams collector.

And let’s stop here and explain a few stuff.

What’s actually this code?

switch (when(state, event)) {
   // (...)
}

In Java 22, you can do pattern matching on records, yet the switch can take just a single object; here, we’d like to get the permutations of the potential state and events evolving them. Java doesn’t have tuples like e.g. C# or TypeScript. You need to be explicit and define dedicated types. From what I heard recently from Brian Goetz, that’s intentional. See more:

So, let’s define some explicit records to use them in switch!

public class FunctionalTools {
  public static <State, Event> When<State, Event> when(State state, Event event){
    return new When<>(state, event);
  }

  public static <State, Event> On<State, Event> on(State state, Event event){
    return new On<>(state, event);
  }

  public record When<State, Event>(State state, Event event) {
  }

  public record On<State, Command>(State state, Command command) {
  }
}

As you see, those records are just wrappers for the values. They’re explicit and named to be a bit more readable in the intended usage. I also added some helper factory functions to reduce the noise of generic record setup in the end usage.

Now, we can use the fabulous feature which is static import, e.g.

import static FunctionalTools.When;
import static FunctionalTools.when;

Et voilà! Now, we can use it in the switch statement like:

switch (when(state, event)) {
  // (...)
  case When(
    Pending(var productItems),
    ProductItemAdded(_, var productItem, _)
  ) -> new Pending(productItems.add(productItem));
}

This means we’re creating a When record using the when function from state and event. Then we can destructure it and say that having When record with the first property of type Pending (in other words, pending shopping cart) and the second parameter of type ProductItemAdded (applying this event on the pending state), we should run the following code. In this case, return the new Pending shopping cart with the added product item.

What’s more, this type of check is exhaustive, so if we don’t provide a default case, the compiler will tell you that you didn’t provide all permutations!

As you see, a bit of explicit code and intentional design can create a straightforward, declarative definition of our code. Much shorter than the imperative one. Of course we should be careful not to make it cryptic. The intention is to have explicit business logic, not sneaky, smart code.

Let’s show the final code again. Decider, for running business logic:

public class ShoppingCartDecider {
  public sealed interface Command {
    record Open(
      UUID shoppingCartId,
      UUID clientId,
      OffsetDateTime now
    ) implements Command {
    }

    record AddProductItem(
      UUID shoppingCartId,
      PricedProductItem productItem,
      OffsetDateTime now
    ) implements Command {
    }

    record RemoveProductItem(
      UUID shoppingCartId,
      PricedProductItem productItem,
      OffsetDateTime now
    ) implements Command {
    }

    record Confirm(
      UUID shoppingCartId,
      OffsetDateTime now
    ) implements Command {
    }

    record Cancel(
      UUID shoppingCartId,
      OffsetDateTime now
    ) implements Command {
    }
  }

  public static ShoppingCart.Event decide(Command command, ShoppingCart state) {
    return switch (on(state, command)) {
      case On(Initial _, Open(var id, var clientId, var now)) ->
        new Opened(id, clientId, now);

      case On(
        Pending _,
        AddProductItem(var id, var productItem, var now)
      ) -> new ProductItemAdded(id, productItem, now);

      case On(
        Pending(var productItems),
        RemoveProductItem(var id, var productItem, var now)
      ) -> {
        if (!productItems.hasEnough(productItem))
          throw new IllegalStateException("Not enough product items to remove");

        yield new ProductItemRemoved(id, productItem, now);
      }

      case On(Pending _, Confirm(var id, var now)) ->
        new Confirmed(id, now);

      case On(Pending _, Cancel(var id, var now)) ->
        new Canceled(id, now);

      default -> throw new IllegalStateException(
        String.format("Cannot %s on %s", command.getClass().getName(), state.getClass().getName())
      );
    };
  }
}

And shopping cart defining our state and its evolution:

public sealed interface ShoppingCart {
  record Initial() implements ShoppingCart {
  }

  record Pending(ProductItems ProductItems) implements ShoppingCart {
  }

  record Closed() implements ShoppingCart {

  }

  sealed interface Event {
    record Opened(
      UUID shoppingCartId,
      UUID clientId,
      OffsetDateTime openedAt
    ) implements Event {
    }

    record ProductItemAdded(
      UUID shoppingCartId,
      ProductItems.PricedProductItem productItem,
      OffsetDateTime addedAt
    ) implements Event {
    }

    record ProductItemRemoved(
      UUID shoppingCartId,
      ProductItems.PricedProductItem productItem,
      OffsetDateTime removedAt
    ) implements Event {
    }

    record Confirmed(
      UUID shoppingCartId,
      OffsetDateTime confirmedAt
    ) implements Event {
    }

    record Canceled(
      UUID shoppingCartId,
      OffsetDateTime canceledAt
    ) implements Event {
    }
  }

  static  ShoppingCart getShoppingCart(Event[] events) {
    ShoppingCart state = new Initial();

    for (var event : events) {
      state = evolve(state, event);
    }
    return state;
  }

  static ShoppingCart evolve(ShoppingCart state, Event event) {
    return switch (when(state, event)) {
      case When(Initial _, Opened _) ->
        new Pending(ProductItems.empty());

      case When(
        Pending(var productItems),
        ProductItemAdded(_, var productItem, _)
      ) -> new Pending(productItems.add(productItem));

      case When(
        Pending(var productItems),
        ProductItemRemoved(_, var productItem, _)
      ) -> new Pending(productItems.remove(productItem));

      case When(Pending _, Confirmed _),
           When(Pending _, Canceled _) -> new Closed();

      default -> state;
    };
  }
}

I encourage you to play with those features. They can be super useful not only for Event Sourcing but also for domain modelling in general.

One of the possible options is to go through my recently updated Event Sourcing self-paced kit. See more in my repository. Or maybe doing a full workshop with me.

Yay or nay?

Cheers!

Oskar

How to configure a custom Test Container on the EventStoreDB example

Testcontainers became a popular way of setting up dependencies for integration testing. They’re not ideal, as configuring them to run your tests efficiently is not as trivial as it’s glossed; I explained that in A simple way to configure integration tests pipeline. Still, undeniably, it can speed up the initial ramp-up phase and, if used wisely, can be a decent way to handle common testing dependencies.

Testcontainers already provide the default configurations for tools like PostgreSQL, Kafka, MongoDB etc. Yet, sometimes you need to use something a bit less mainstream, for instance, EventStoreDB as your event store. How to do it?

Luckily, test Containers provide a built-in way of handling such cases, providing a generic abstraction for that: GenericContainer. We can extend it to provide our test container. Let’s see how to do it using Typescript.

We need to start by adding two classes, one representing a container image configuration and the other representing a started container instance.

import {
  AbstractStartedContainer,
  GenericContainer,
  type StartedTestContainer,
} from 'testcontainers';

export const EVENTSTOREDB_IMAGE_NAME = 'eventstore/eventstore';
export const EVENTSTOREDB_IMAGE_TAG = '23.10.1-bookworm-slim';

export class EventStoreDBContainer extends GenericContainer {
  constructor(
    image = `${EVENTSTOREDB_IMAGE_NAME}:${EVENTSTOREDB_IMAGE_TAG}`
  ) {
    super(image);
    };
  }

  async start(): Promise<StartedEventStoreDBContainer> {
    return new StartedEventStoreDBContainer(await super.start());
  }
}

export class StartedEventStoreDBContainer extends AbstractStartedContainer {
  constructor(container: StartedTestContainer) {
    super(container);
  }

  getConnectionString(): string {
    return `esdb://${this.getHost()}:${this.getMappedPort(2113)}?tls=false`;
  }
}

And that’s the bare minimum that we need to have. The container definition takes the default LTS EventStoreDB image, allowing us to provide the other one. We’re also exposing the EventStoreDB connection string. It’s important as Testcontainers run containers on random ports to help in test isolation.

We could set it up as:

const container: StartedEventStoreDBContainer = await new EventStoreDBContainer().start();

const eventStore = EventStoreDBClient.connectionString(this.getConnectionString())

That’s some aid, but well, if Testcontainers are to reduce the cognitive load by providing a default container setup, then we could do better than that.

We could set up default recommended options for testing. What could they be? For instance, EventStoreDB is secure by default. You should run the setup to configure certificates, etc. It’s also recommended that a cluster with at least three nodes on production be configured to get proper resiliency. All of that is, of course, important, and running tests against the production setup is a must at some point (read more in I tested it on production, and I’m not ashamed of it). Still, for most of the integration tests touching common features (like appending events and (subscribing to notifications about them](/en/persistent_vs_catch_up_eventstoredb_subscriptions_in_action/)), running an insecure single cluster should be more than enough. We could even disable file storage and use an in-memory setup to enable the reusing of the container between multiple tests. All of those are performance optimisations and tradeoffs, but we need to be sure that our test suite is not only giving proper results but also running fast. Tests on real production databases can be covered by synthetic tests.

Ok, then, how to do it? Let’s define some default options like default EventStoreDB ports to expose:

export const EVENTSTOREDB_PORT = 2113;
export const EVENTSTOREDB_TCP_PORT = 1113;
export const EVENTSTOREDB_TCP_PORTS = [
  EVENTSTOREDB_TCP_PORT,
  EVENTSTOREDB_PORT,
];

Then, defining the default image setup.

export const EVENTSTOREDB_IMAGE_NAME = 'eventstore/eventstore';
export const EVENTSTOREDB_IMAGE_TAG = '23.10.1-bookworm-slim';
export const EVENTSTOREDB_ARM64_IMAGE_TAG = '23.10.1-alpha-arm64v8';

export const EVENTSTOREDB_DEFAULT_IMAGE = `${EVENTSTOREDB_IMAGE_NAME}:${process.arch !== 'arm64' ? EVENTSTOREDB_IMAGE_TAG : EVENTSTOREDB_ARM64_IMAGE_TAG}`;

And yes, we’re also giving help to the Mac M1 users, as the ARM64 image is not included in the default tag. It’s still in alpha, so why not automate it?

Now, let’s define the explained earlier options:

export type EventStoreDBContainerOptions = {
  disableProjections?: boolean;
  isSecure?: boolean;
  useFileStorage?: boolean;
  withoutReuse?: boolean;
};

export const defaultEventStoreDBContainerOptions: EventStoreDBContainerOptions =
  {
    disableProjections: false,
    isSecure: false,
    useFileStorage: false,
    withoutReuse: false,
  };

As you see, we’re making them customisable, as we’d like people to enable the full setup. It’s also a good practice to have all default options named in a way so that the default can be assumed as false. That reduces the cognitive load.

Having that, we can update our base container definition:

import {
 AbstractStartedContainer,
 GenericContainer,
} from 'testcontainers';
import type { Environment } from 'testcontainers/build/types';

export class EventStoreDBContainer extends GenericContainer {
  private readonly tcpPorts = EVENTSTOREDB_TCP_PORTS;

  constructor(
    image = EVENTSTOREDB_DEFAULT_IMAGE,
    options: EventStoreDBContainerOptions = defaultEventStoreDBContainerOptions,
  ) {
    super(image);

    const environment: Environment = {
      ...(!options.disableProjections
        ? {
            EVENTSTORE_RUN_PROJECTIONS: 'ALL',
          }
        : {}),
      ...(!options.isSecure
        ? {
            EVENTSTORE_INSECURE: 'true',
          }
        : {}),
      ...(options.useFileStorage
        ? {
            EVENTSTORE_MEM_DB: 'false',
            EVENTSTORE_DB: '/data/integration-tests',
          }
        : {}),
      EVENTSTORE_CLUSTER_SIZE: '1',
      EVENTSTORE_START_STANDARD_PROJECTIONS: 'true',
      EVENTSTORE_EXT_TCP_PORT: `${EVENTSTOREDB_TCP_PORT}`,
      EVENTSTORE_HTTP_PORT: `${EVENTSTOREDB_PORT}`,
      EVENTSTORE_ENABLE_EXTERNAL_TCP: 'true',
      EVENTSTORE_ENABLE_ATOM_PUB_OVER_HTTP: 'true',
    };

    this.withEnvironment(environment).withExposedPorts(...this.tcpPorts);

    if (!options.withoutReuse) this.withReuse();
  }

  async start(): Promise<StartedEventStoreDBContainer> {
    return new StartedEventStoreDBContainer(await super.start());
  }
}

As you can see, we’re setting up the default Environment settings, exposing ports, and marking our container as reusable by default (by calling this.withReuse()). Thanks to that, Testcontainers will not start a new container if a Testcontainers-managed container with the same configuration is already running. That cuts the cold-start time. Yet, you need to ensure that the test data you’re adding won’t collide with other tests, so you better be careful.

You can still override the environment settings by calling withEnvironment method.

Having that, let’s also extend the started container by adding getClient method that’ll return already setup EventStoreDB client for your container.

import { EventStoreDBClient } from '@eventstore/db-client';
import {
  AbstractStartedContainer,
  type StartedTestContainer,
} from 'testcontainers';

export class StartedEventStoreDBContainer extends AbstractStartedContainer {
  constructor(container: StartedTestContainer) {
    super(container);
  }

  getConnectionString(): string {
    return `esdb://${this.getHost()}:${this.getMappedPort(2113)}?tls=false`;
  }

  getClient(): EventStoreDBClient {
    return EventStoreDBClient.connectionString(this.getConnectionString());
  }
}

You can test if all is working fine through the test:

import { jsonEvent } from '@eventstore/db-client';
import assert from 'node:assert/strict';
import { randomUUID } from 'node:crypto';
import { after, beforeEach, describe, it } from 'node:test';
import {
  EventStoreDBContainer,
  StartedEventStoreDBContainer,
} from './eventStoreDBContainer';

void describe('EventStoreDBContainer', () => {
  let container: StartedEventStoreDBContainer;

  beforeEach(async () => {
    container = await new EventStoreDBContainer().start();
  });

  void it('should connect to EventStoreDB and append new event', async () => {
    const client = container.getClient();

    const result = await client.appendToStream(
      `test-${randomUUID()}`,
      jsonEvent({ type: 'test-event', data: { test: 'test' } }),
    );

    assert.ok(result.success);
  });

  after(async () => {
    await container.stop();
  });
});

You can use the same pattern for any other tool with container image. Also you can define custom images in the similar way, if you’re using C#, Java, Go, or other supported by Testcontainers developer environment.

Still, if you’re using Node.js, and you’d like to use EventStoreDB Testcontainer, then hey, you can just install Emmett Testcontainers package:

npm add @event-driven-io/@event-driven-io/emmett-testcontainers

And use it as:

import {
  EventStoreDBContainer,
  type StartedEventStoreDBContainer,
} from '@event-driven-io/emmett-testcontainers';

let esdbContainer: StartedEventStoreDBContainer;
void describe('ShoppingCart E2E', () => {
  // Set up a container before all tests
  before(async () => {
    esdbContainer = await new EventStoreDBContainer().start();
  });

  // Stop container once we finished testing
  after(() => {
    return esdbContainer.stop();
  });
  // (...) Tests will go here
});

Read also more in Testing Event Sourcing, Emmett edition.

Check also my other articles on DevOps and Continuous Integration:

Cheers!

Oskar

Mocking the native Node.js Test Runner

Last week, we discussed an overused but applicable pattern: in-memory bus. This time, we’ll continue with the leitmotif and talk about mocking. No, I won’t mock you; I will mock TypeScript code.

Node.js in version 18th added its own native Test Runner. Why did they do it? Jest looks like abandonware even after transferring to OpenJs Foundation, issues are not getting closed, and it’s slow. Vitest is a decent new alternative.

Native Test Runner is also young but nicely integrated directly into Node.js and maintained by the core team. It’s lightweight, thanks to Thiago Valentim’s suggestion and contribution. I started using it in Emmett. So far, so good; it’s really fast (each test file spawns as a Node.js subprocess); sometimes configuring it with TypeScript adds a bit of a headache, but which tool doesn’t?

That’s how we’re reaching mocking. What’s mocking? Nice, try, but I won’t go into that battle. Telling what’s right is a slippery slope, as you may get various answers from different people and arguments on the nitty gritty of the definition. Let me just quote Martin Fowler:

“(Mocks are) objects pre-programmed with expectations which form a specification of the calls they are expected to receive. (…) mocks insist upon behavior verification. The other doubles can, and usually do, use state verification.”

Fowler shows the following example:

class OrderInteractionTester...

  public void testOrderSendsMailIfUnfilled() {
    Order order = new Order(TALISKER, 51);
    Mock warehouse = mock(Warehouse.class);
    Mock mailer = mock(MailService.class);
    order.setMailer((MailService) mailer.proxy());

    mailer.expects(once()).method("send");
    warehouse.expects(once()).method("hasInventory")
      .withAnyArguments()
      .will(returnValue(false));

    order.fill((Warehouse) warehouse.proxy());
  }
}

That’s also the reason why I don’t like Mocks; I disagree that testing if the method is called, by definition, focuses on behaviour. Too often, it just focuses on the mechanics and the way we implement our code instead of the real behaviour we’d like to test. Focusing on behaviour is really important to me, and I wrote about here. Still, let’s not go too far down this road, if you have more time, I encourage you to read a great take from James Shore about testing without mocks.

Still, it’s undeniably useful to sometimes mock the implementation of the real object or verify whether we actually called it. One way or another. Thanks to that, we can test our code or application without calling external services, doing a lot of IO operations, etc. Or ensure that dependency was called.

Also, it’s not always easy to see the results of our behaviour. Let’s take event publishing from the last article. We have the following interface:

interface EventsPublisher {
  publish<EventType extends Event = Event>(event: EventType): Promise<void>;
}

It’s fire and forget; the event is forwarded to the handlers. We could write a stub that’d be doing a simple implementation, but hey, we’d repeat much of what we did, as our implementation is already in-memory. We could, of course, implement a stub for the handler and ensure that it was called. That wouldn’t be bad, but we could also just mock it.

Let’s say that we have an Anti-Corruption Layer for payment gateway. We use it to reduce the scope of change in our system’s external API. When we make a payment, we also want to notify others that it was successfully made. The dummy implementation for that could look as follows:

interface ExternalPaymentGateway {
  makePayment(amount: number): Promise<void>;
}

type PaymentMade = Event<'PaymentMade', { amount: number; paidAt: Date }>;

const PaymentGatewayAcl = (
  externalApi: ExternalPaymentGateway,
  eventPublisher: EventsPublisher,
) => {
  return {
    makePayment: async (amount: number): Promise<void> => {
      await externalApi.makePayment(amount);

      await eventPublisher.publish<PaymentMade>({
        type: 'PaymentMade',
        data: { amount, paidAt: new Date() },
      });
    },
  };
};

Our ACL has two dependencies: an external payment API gateway and an Events Publisher.

We also have a random factor, so the paid date is generated based on the current time. This makes our tests potentially non-deterministic, as from outside, we won’t know the exact value. We could pass the function that’d return us a new date and then mock this dependency, but we can also do it differently. We’ll get to that in a bit.

Let’s try to set up our tests:

import assert from 'node:assert';
import { describe, it, mock } from 'node:test';
import { type Event, type EventsPublisher, getInMemoryMessageBus } from '@event-driven-io/emmett';

void describe('Payment Gateway Acl', () => {
  const eventsPublisher = getInMemoryMessageBus();
  const externalPaymentGatewayStub: ExternalPaymentGateway = {
    makePayment: () => Promise.resolve(),
  };  
  const sut = PaymentGatewayAcl(externalPaymentGatewayStub, eventsPublisher);

  const now = new Date();
  mock.timers.enable({ apis: ['Date'], now });

  void it('publishes event when payment was made',  () => {
     console.log('We'll get here in a bit!');
     assert.fail();
  });

  void it('does NOT publish event when payment failed', () => {
     console.log('We'll get here in a bit!');
     assert.fail();
});

The syntax is similar to the one you may know from Jest, Jasmine, and other testing frameworks. We’re starting by setting up our dependencies and system under test: PaymentGatewayAcl. We’re using the in-memory bus as it is and providing a stub of the external payment gateway with a do-nothing-and-always-succeed implementation.

Node.js Native Test Runner provides basic abstractions for mocking. It is also neat that it allows mocking dates and timers like setTimeout.

import { mock } from 'node:test';

const now = new Date();
mock.timers.enable({ apis: ['Date'], now });

That means the Node.js test runtime will always inject the provided date instead of generating a new one, making our test non-deterministic. That’s the benefit of having a native test runner instead of a third-party solution. Read more options in documentation.

Let’s now fill the first test:

void describe('Payment Gateway Acl', () => {
    // (...) setup

  void it('publishes event when payment was made', async (test) => {
    // Given
    const publishEvent = test.mock.method(eventsPublisher, 'publish');

    const amount = Math.random() * 100;

    // When
    await sut.makePayment(amount);

    // Then
    const expectedEvent: PaymentMade = {
      type: 'PaymentMade',
      data: {
        amount,
        paidAt: now,
      },
    };

    verifyThat(publishEvent).calledWith(expectedEvent);
  });
});

We’re telling the test runner to watch for our event publisher publish method.

const publishEvent = test.mock.method(eventsPublisher, 'publish');

We’ll need it to verify if the method was called with the expected parameters in the last line. We’re also using the test.mock from the test parameter, as this will ensure mocks will be set up only in the scope of the test. It is essential to have test isolation. We wouldn’t like to have calls in other tests interfere with our verification:

verifyThat(publishEvent).calledWith(expectedEvent);

And here we’re getting to the native test runner’s childhood issues. You can mock only a specific method, not the whole object. The API is pretty raw, and not all types are exposed out of the box.

The verifyThat helper you saw was made by me. Let’s discuss how to provide such a simple wrapper.

We need to define types for Mocked Function:

type AnyFunction = (...args: any[]) => any;

type Call = {
  arguments: unknown[];
  result: unknown;
  target: unknown;
  this: unknown;
};

type MockedFunction<T extends AnyFunction> = T & {
  mock?: { calls: Call[] };
};

We started by defining AnyFunction, which (as the name suggests) represents functions with a set of arguments, essentially any functions. Defining it like that is essential; you’ll see it in a moment.

Then, we’re adding typing for mocked function calls. It represents collections of calls that were made. Each call may have different arguments and results.

Now we can define the verifyThat wrapper as:

export const verifyThat = <T extends AnyFunction>(fn: MockedFunction<T>) => {
  return {
    calledWith: (...args: Parameters<T>) => {
      assert.ok(
        fn.mock?.calls.length !== undefined &&
          fn.mock.calls.length >= 1 &&
          fn.mock.calls.some((call) => deepEquals(call.arguments, args)),
      );
    },
    calledTimes: (times: number) => {
      assert.equal(fn.mock?.calls?.length, times);
    },
    notCalled: () => {
      assert.equal(fn?.mock?.calls?.length, 0);
    },
    called: () => {
      assert.ok(
        fn.mock?.calls.length !== undefined && fn.mock.calls.length > 0,
      );
    },
  };
};

const deepEquals = (left: unknown, right: unknown): boolean => {
  try {
    assert.deepEqual(left, right);
    return true;
  } catch {
    return false;
  }
};

The verifyThat is a builder function that returns an object with assertions for the passed function.

The most interesting one is calledWith. It uses Parameters to build into TypeScript. It allows you to construct the array of arguments for the passed function. Moreover, it also respects the parameter types, even if they’re different! Thanks to that, the compiler won’t let you provide an incomplete number of parameters or use the wrong types. Sweet!

I also added some examples of the other assertions you can pass there. I’m also using built-in assertions provided by Node.js natively. Let’s use one of them in the negative scenario test:

void describe('Payment Gateway Acl', () => {
    // (...) setup

  void it('does NOT publish event when payment failed', async (test) => {
    const publishEvent = test.mock.method(eventsPublisher, 'publish');
    const makePayment = test.mock.method(
      externalPaymentGatewayStub,
      'makePayment',
    );
    makePayment.mock.mockImplementation(() =>
      Promise.reject('You shall not pass!'),
    );

    const amount = Math.random() * 100;

    try {
      await sut.makePayment(amount);
      assert.fail('Expecting error!');
    } catch {
      verifyThat(publishEvent).notCalled();
    }
  });
});

Native test runner also allows the replacement of the existing implementation. We’re doing that to simulate the failure of the payment gateway.

makePayment.mock.mockImplementation(() =>
  Promise.reject('You shall not pass!'),
);

This will make our dependency fail; we can verify in the try/catch statement if our event publisher wasn’t indeed called by using humble wrapper:

verifyThat(publishEvent).notCalled();

And that’s it. I think I’ll also include this set of helpers for Emmett, so you can also benefit from it, but in any case, feel free to steal this code and have fun!

Yet, beware not to abuse mocks, and try to shape your code so you don’t need too much of them.

Cheers!

Oskar

How to build an in-memory Message Bus in TypeScript

I’m writing this article on Friday, and it’s about time to have some fun. As this is a programming blog, let’s have some fun coding. Let’s do an exercise designing a type-safe in-memory message bus in TypeScript!

What’s a message bus? Let’s start from the beginning.

There are two ways of communicating: direct and indirect. For direct communication, we ask another component to perform a specific action and want to know if that happened. For indirect communication, we notify others that something has happened and let them decide what to do. In a nutshell, direct is represented by command, and indirect by event. Read more in What’s the difference between a command and an event?.

Typically, direct communication is assumed to be blocking and indirect non-blocking, but that’s a common practice, not a rule. Both types of communication can be blocking or non-blocking. In the real world, we may ask other people to do something and wait until they finish or assume that they will reply to us when they have done it. For indirect communication, even though we’re not interested in what will happen after we broadcast news, we’d like to know whether all interested parties took action.

This may sound mind-boggling, but it’s not if we focus on modelling our communication as it’s happening in real business processes. I explained that with examples in How TypeScript can help in modelling business workflows.

What can we do with our messages, then? The bare minimum is:

sending: expecting it to be handled, but not returning a reason,
publishing: broadcasting information to all subscribers and waiting for them to process it.
scheduling: assuming that this message will be published later asynchronously, for instance, using Outbox Pattern.

There is more to that; Gregor Hohpe curated most of the Enterprise Integration Patterns in his book and website. There are over 65 of them, so let’s start simple and expand in further articles.

Let’s start with defining Command and Event types:

type Event<
  EventType extends string = string,
  EventData extends Record<string, unknown> = Record<string, unknown>,
  EventMetaData extends Record<string, unknown> = Record<string, unknown>,
> = Flavour<
  Readonly<{
    type: EventType;
    data: Readonly<EventData>;
    metaData?: Readonly<EventMetaData>;
  }>,
  'Event'
>;

type Command<
  CommandType extends string = string,
  CommandData extends Record<string, unknown> = Record<string, unknown>,
  CommandMetaData extends Record<string, unknown> = Record<string, unknown>,
> = Flavour<
  Readonly<{
    type: CommandType;
    data: Readonly<CommandData>;
    metaData?: Readonly<CommandMetaData>;
  }>,
  'Command'
>;

They both have the type representing their name, data-specific information this message gathers, and optional metadata. As the structure is the same, they’re both flavoured to let the TypeScript compiler distinguish the differences between them. Read more details on that in other article, in which I explained the details of this definition.

Having them, let’s define how our sending and publishing definition will look like:

interface EventsPublisher {
  publish<EventType extends Event = Event>(event: EventType): Promise<void>;
}

interface CommandSender {
  send<CommandType extends Command = Command>(
    command: CommandType,
  ): Promise<void>;
}

We’re making the intention specific by limiting publishing for events, as they are indirect broadcasts, and sending to allow only commands, as they represent direct communication to a single handler.

The schedule will look accordingly, but allow to schedule both types.

type ScheduleOptions = { afterInMs: number } | { at: Date };

interface MessageScheduler<CommandOrEvent extends Command | Event> {
  schedule<MessageType extends CommandOrEvent>(
    message: MessageType,
    when?: ScheduleOptions,
  ): Promise<void>;
}

Here, we assume that it’ll be forwarded in a separate process, and we’re not expecting to get any response either for command or from the event. If we want to reply to the status of the command, then we should do it by publishing the a follow up event with a new fact and handing it back. We also allow passing additional options informing when the message should be scheduled.

You may wonder why we aren’t putting all that into a single interface, and the reason is that we want to be precise about the intention. Typically, you’re either handling events or commands, and it’s better to have the option to use a narrowed interface that allows you to do only certain things. If we have the granular definitions, then we can also compose them into other interfaces, getting an all-in-one box. Here’s how you do it:

interface EventBus extends EventsPublisher, MessageScheduler<Event> {}

interface CommandBus extends CommandSender, MessageScheduler<Command> {}

interface MessageBus extends CommandBus, EventBus {
  schedule<MessageType extends Command | Event>(
    message: MessageType,
    when?: ScheduleOptions,
  ): Promise<void>;
}

There you have it!

Now that we know how to send, publish, and schedule messages, how do we handle them? We’ll need processors that allow registering functions with a specific handling. Let’s define them!

type EventHandler<EventType extends Event = Event> = (
  event: EventType,
) => Promise<void> | void;

interface EventProcessor {
  subscribe<EventType extends Event>(
    eventHandler: EventHandler<EventType>,
    ...eventTypes: EventTypeOf<EventType>[]
  ): void;
}

type CommandHandler<CommandType extends Command = Command> = (
  command: CommandType,
) => Promise<void> | void;

interface CommandProcessor {
  handle<CommandType extends Command>(
    commandHandler: CommandHandler<CommandType>,
    ...commandTypes: CommandTypeOf<CommandType>[]
  ): void;
}

We allow registering a single handler for the event and command handlers, as we may want to unify the handling of the commands or events grouped into Union Types. For instance having:

export type GuestStayAccountEvent =
  | GuestCheckedIn
  | ChargeRecorded
  | PaymentRecorded
  | GuestCheckedOut
  | GuestCheckoutFailed;

const handleGuestStay = (event: GuestStayAccountEvent) => {
  switch (event.type) {
    case 'GuestCheckedIn':
      return onCheckedIn(event.data);
    case 'ChargeRecorded':
      return onChargeRecorded(event.data);
    case 'GuestCheckedOut':
      return onGuestCheckedOut(event.data);
    case 'GuestCheckoutFailed':
      return onGuestCheckoutFailed(event.data);
  }
};

We can register it with:

eventProcessor.subscribe(
  handleGuestStay,
  'GuestCheckedIn',
  'ChargeRecorded',
  'GuestCheckedOut',
  'GuestCheckoutFailed',
);

And accordingly with command handling.

You may notice that I used two new types: EventTypeOf and CommandTypeOf. They’re used to having a strongly typed way of handling message types. They’re defined as:

type CommandTypeOf<T extends Command> = T['type'];

type EventTypeOf<T extends Event> = T['type'];

It is a TypeScript trick that tells the compiler of the expected range of the message types. If we try to provide other values, then the compiler will show an error. Sweet!

It would be even sweeter if we didn’t have to provide those string values but take them directly from the message type definition. Unfortunately, coding in TypeScript, we cannot have only good things. Those type annotations are only available at compile time. On runtime, there’s no trace of that. That’s the “beauty” of the dynamic runtime in JavaScript.

Cool, the last step before going finally to implementation is to implement a processor for scheduled messages:

type ScheduledMessage = {
  message: Event | Command;
  options?: ScheduleOptions;
};

interface ScheduledMessageProcessor {
  dequeue(): ScheduledMessage[];
}

We’re exposing method that will dequeue the pending messages. We can take them and store them in Outbox or forward to messaging queue.

Now, we have our requirements, let’s proceed with the implementation. We’ll tackle that one by one, let’s start with the overall definition of the message bus setup:

type MessageHandler = EventHandler | CommandHandler;

type MessageProcessor = EventProcessor | CommandProcessor;

const getInMemoryMessageBus = ():
  & MessageBus
  & MessageProcessor
  & ScheduledMessageProcessor => {
  const allHandlers = new Map<string, MessageHandler[]>();
  let pendingMessages: ScheduledMessage[] = [];

  return { 
    // (...) here will go the interfaces methods definition
  };
};

We’ll take benefit of the Structural Typing and provide the one implementation to rule them all, as our command and event handler definitions are the same from the JavaScript runtime perspective. Only at compile time they differ, later on, they just take the message and run the message handling.

We’ll also group all handlers in the same map that takes the message type and arrays of functions to process it. Event handlers can have multiple ones, for commands it’ll be always a single handler, as we won’t allow to register more than one.

We’ll also have the unified pending messages collection, this will also guarantee the ordering on the producer side, which is quite important for handling workflows and building projections when subscribing to events.

Let’s now show it one by one starting with registering event handler:

return {
 // (...)
 subscribe<EventType extends Event>(
    eventHandler: EventHandler<EventType>,
    ...eventTypes: EventTypeOf<EventType>[]
  ): void {
    for (const eventType of eventTypes) {
      if (!allHandlers.has(eventType)) allHandlers.set(eventType, []);

      allHandlers.set(eventType, [
        ...(allHandlers.get(eventType) ?? []),
        eventHandler as MessageHandler,
      ]);
    }
  },
};

For each event type, we’re just appending the handler to already existing set. A single event type can have multiple handlers (as event notification is a broadcast).

Registering command handler looks accordingly, but there’s a slight change:

return {
  handle: <CommandType extends Command>(
    commandHandler: CommandHandler<CommandType>,
    ...commandTypes: CommandTypeOf<CommandType>[]
  ): void => {
    const alreadyRegistered = [...allHandlers.keys()].filter((registered) =>
      commandTypes.includes(registered),
    );

    if (alreadyRegistered.length > 0)
       throw new Error(
        `Cannot register handler for commands ${alreadyRegistered.join(', ')} as they're already registered!`,
      );
    for (const commandType of commandTypes) {
      allHandlers.set(commandType, [commandHandler as MessageHandler]);
    }
  },
};

As mentioned, we’re checking if there’s no handler already registered for the specific command type. If there’s then we’re throwing an error, otherwise just adding it to the handlers registration.

Let’s now show how to publish an event:

return {
  // (...)
  publish: async <EventType extends Event = Event>(
    event: EventType,
  ): Promise<void> => {
    const handlers = allHandlers.get(event.type) ?? [];

    for (const handler of handlers) {
      const eventHandler = handler as EventHandler<EventType>;

      await eventHandler(event);
    }
  },
};

We’re just calling each registered handler sequentially. We could also use Promise.all and allow handling them in parallel, or allow both type of handling depending on how the event handler was registered.

Sending command will look accordingly, but again we’re adding a validation.

return {
  // (...)
  send: async <CommandType extends Command = Command>(
    command: CommandType,
  ): Promise<void> => {
    const handlers = allHandlers.get(command.type);

    if (handlers === undefined || handlers.length === 0)
      throw new EmmettError(
        `No handler registered for command ${command.type}!`,
      );

    const commandHandler = handlers[0] as CommandHandler<CommandType>;

      await commandHandler(command);
  },
};

We’re doublechecking if there’s an actual handler for command, because if it’s not then there’s no point for sending it, remember, it’s a direct communication, expecting command to be handled precisely once. We don’t need to check if there are more handlers, as we’re not allowing to register such.

Scheduling is even simpler as we just need to put message into the pending items collection:

return {
  // (...)
  schedule: <MessageType extends Command | Event>(
    message: MessageType,
    when?: ScheduleOptions,
  ): void => {
    pendingMessages = [...pendingMessages, { message, options: when }];
  },
};

We’ll dequeue it later on while processing those scheduled message. We’ll do it simply as that:

return {
  // (...)
  dequeue: (): ScheduledMessage[] => {
    const pending = pendingMessages;
    pendingMessages = [];
    return pending;
  },
};

The good thing is that we can assign such object to any of the interfaces we defined above. E.g.

const eventPublisher: EventPublisher = getInMemoryMessageBus();

const commandBus: CommandBus = getInMemoryMessageBus();

It’ll respect the proper types. It’s a good starting point, we can separate handling when they’ll become too different. Our interfaces are already simple enough that they should be stable, allowing us to expand the implementation once we need it (e.g. adding stuff like retry policy, telemetry, middlewares, etc.).

Do you want to use it in your project? No worries, I got you covered, Emmett already supports that!

Just run

npm install @event-driven-io/emmett

And you can use all those types out of the box, e.g.

import { getInMemoryMessageBus } from '@event-driven-io/emmett';

const messageBus = getInMemoryMessageBus();

And you can use all those types out of the box, e.g.

import { getInMemoryMessageBus } from '@event-driven-io/emmett';

const messageBus = getInMemoryMessageBus();

Then use it as for event subscription:

 messageBus.subscribe(
  handleGuestStay,
  'GuestCheckedIn',
  'ChargeRecorded',
  'GuestCheckedOut',
  'GuestCheckoutFailed',
);

and for event publishing:

const event:GuestCheckedOut = {
  type: 'GuestCheckedOut',
  data: {
    guestStayAccountId: 'r9293';
    checkedOutAt: new Date();
  }
>;

await messageBus.publish(event);

Last, but not least, I’d like to end this article with a disclaimer.

Command bus can be overkill in cases where we have only a single entry point (e.g. API endpoint). In such cases, I suggest to have just explicit application code in the endpoint.

For instance with Emmett it’d look like that:

export const shoppingCartApi =
  (
    eventStore: EventStore,
    // (...) other dependencies
    getUnitPrice: (productId: string) => Promise<number>,
  ): WebApiSetup =>
  (router: Router): void => {
  router.post(
    '/clients/:clientId/shopping-carts/current/product-items',
    async (request: AddProductItemRequest, response: Response) => {
      // 1. Translate request params to the command
      const shoppingCartId = getShoppingCartId(
        assertNotEmptyString(request.params.clientId),
      );
      const productId = assertNotEmptyString(request.body.productId);

      const command: AddProductItemToShoppingCart = {
        type: 'AddProductItemToShoppingCart',
        data: {
          shoppingCartId,
          productItem: {
            productId,
            quantity: assertPositiveNumber(request.body.quantity),
            unitPrice: await getUnitPrice(productId),
          },
        },
      };

      // 2. Handle command
      await handle(eventStore, shoppingCartId, (state) =>
        addProductItem(command, state),
      );

      // 3. Send response status
      response.sendStatus(204);
    },
  );
}

That gives a proper developer experience, rather than just calling ev. As you understand your dependencies, you can go to the definition of business logic. A command bus requires much more jumping around the codebase as you hide all dependencies and handling details.

export const shoppingCartApi =
  (
    commandSender: CommandSender,
    getUnitPrice: (productId: string) => Promise<number>,
  ): WebApiSetup =>
  (router: Router): void => {
  router.post(
    '/clients/:clientId/shopping-carts/current/product-items',
    async (request: AddProductItemRequest, response: Response) => {
      // 1. Translate request params to the command
      const shoppingCartId = getShoppingCartId(
        assertNotEmptyString(request.params.clientId),
      );
      const productId = assertNotEmptyString(request.body.productId);

      const command: AddProductItemToShoppingCart = {
        type: 'AddProductItemToShoppingCart',
        data: {
          shoppingCartId,
          productItem: {
            productId,
            quantity: assertPositiveNumber(request.body.quantity),
            unitPrice: await getUnitPrice(productId),
          },
        },
      };

      // 2. Handle command
      await commandSender.send(command, state));

      // 3. Send response status
      response.sendStatus(204);
    },
  );
}

If I have more than one entry point, e.g., event (as it always can/should have more than one recipient) or command that may come from messaging tooling and API, then the command bus is useful, as it allows me to build common middleware.

The downside of the message bus is that it can create another level of indirection, making it hard to understand where the handler is, what the impact of the change is, etc.

Also, it doesn’t give proper delivery guarantees unless we wrap the whole processing in transactions, which can cause deadlocks and other types of complexity.

That’s why I’m not using it everywhere, but where I need it. I evolved from the message bus all the things. But it still can be useful, especially for event publishing, where, by nature, you don’t want to know how it’ll be handled as an event producer.

Cheers!

Oskar

Event modelling anti-patterns explained

Have you heard about Passive Aggressive Events or CRUD sourcing? Or maybe about the Clickbait event?

If you don’t, you better check the talk I gave at Kafka Summit 2024. Knowing only best practices is one side of the coin. Knowing what NOT to do can be even more important.

The talk also summarised my current article series about anti-patterns in event modelling. Here’s the full list:

Check also more general considerations:

During the session, I explained the specifics of event modelling, starting with bad practices and knowing why and how to avoid them. I told the story about the project that aimed to modernise legacy software into the event-driven world. In theory, artificial, but in practice, none of the examples were made up. Either I made those mistakes on my own, or I saw them in my projects or helped to fix them for my clients.

I tried to make it both entertaining and educational, bitter and sweet. It is not easy when you’re not a native speaker. There’s a thin line between being funny and being silly.

The thin line is also between bad and good practices. And this thin line is: context.

If you’d like to avoid those mistakes, don’t hesitate to [email protected]">contact me!. I’m here to help. Check my training page. A workshop is the most effective way to jump-start.

You can check recommendations on my Linked.in profile to see how other people liked working with me.

Watch also my other talks:

Cheers!

Oskar

I'm no longer Marten maintainer

Folks, I’d like to inform you that I’m no longer a Marten maintainer. As you know, sometimes in the project’s lifetime, there’s a moment when either you need to settle things or make hard decisions. Unfortunately, the latter occurred here.

I’m not happy that this is ending not as I hoped. Marten was a big part of my life: over 6 years as a contributor and over 5 years as a co-maintainer. Still, I’d like to thank Jeremy Miller and Babu Annamalai for our past collaboration. I think that we managed to build a unique tool. I’d also like to say big kudos to the Marten community. I always said that’s one of the big reasons that motivated me on this path.

For various reasons, I already wasn’t active in the final push for the v7 version, which is a significant milestone. I think that shows that the team is motivated to keep it going.

I’m not leaving the Event Sourcing and Event-Driven space. I’m planning to continue what I was doing, so if you need any help from me, feel free to reach out to me.

The closest goals are the workshops on Techorama and DDD Europe. Then, I’ll think about what’s next and how much I need to adjust my course.

This blog is also a chronicle of my journey, I will finish it with a few stats and dates.

As mentioned, I was Marten’s:

User for over 7 years. I’m not sure of the exact date, but I started my EventSourcing .NET samples to learn Event Sourcing and Marten. First commit was on 28th January 2017, so it had to be a bit earlier than that.
Contributor for over 6 years. I opened my first pull request to Marten (and in general to OSS ever) on the 6th of August 2017. We were missing asynchronous apply methods in projections, so I thought, “let’s try to do this OSS-thing”. It was merged on the 1st September 2017. Funnily enough, eventually, we didn’t use this change in our project as we were using it to load additional data while updating read models, which is not a great practice in general. But, well, we all need to start somewhere.
Maintainer for over 5 years. It happened officially on Jeremy’s blog on 27th September 2018. I was already active on our Gitter, so I am sending more Pull Requests. Jeremy invited Babu and me to join the core team, and we agreed.

During that time, I’ve made:

146 Pull Requests,
592 Commits,
214724 Additions,
149,399 Removals

Plus also work on Weasel, spinoff for schema management:

23 Pull Requests,
90 Commits,
12051 Additions,
10010 Removals.

Being 2nd most active contributor (both in this period and in general).

If we exclude the last three months when I wasn’t very active code-wise, then even on 1st. Of course, those are just code output stats; they don’t show the quality or importance of contribution, just activity. Ultimately, they’re just fun/dumb statistics for a chronicle like this.

We’ve built the Discord community for 1235 members (at the time of writing).

We joined as one of the first .NET Foundation in March 2020. And we left it as one of the first in February 2022.

I’m not happy about the ending, but I don’t regret the journey.

Something is ending, something is starting. Off we go to the next chapter.

Cheers

Oskar

Testing Event Sourcing, Emmett edition

I’ve been going pretty down the rabbit hole in the last few years. What am I searching for? A way to deliver better software. And that’s, of course, a “thank you for nothing” type of answer. And also a joke, I’ll tell you what I mean by better. I mean software close to the business, fulfilling the requirements with expected quality and allowing fast iterations to get a quicker feedback loop.

To do that, we need to use proper design/modelling techniques and a tool that enables building synergy. Some say your code should scream, but I don’t like shouting. That’s enough if the code is explicit and sincere. Yet, too often, being explicit means a lot of manual work and boilerplate. That’s not superb developer experience; I wrote more about that in Stacking the bricks in the software development process. And that’s the biggest source of beef around coding styles and architecture. Different people see the tipping point between boilerplate and generic in other places. Where do I see it? That’s a longer take, but let’s focus on one specific aspect today: quality and testing.

I wrote already here a few times about testing:

Today, I’d like to show you the next iteration and my current state of the art, which resulted from that journey.

I’ll show you how the Emmett way to testing can help with your developer experience!

Before we go, let’s install the Emmett npm package. We’ll use it in next steps:

npm add @event-driven-io/emmett

Domain

Let me use a similar Shopping Cart domain I explained in Straightforward Event Sourcing with TypeScript and NodeJS. The business logic looks as follows:

import {
  EmmettError,
  IllegalStateError,
  sum,
} from '@event-driven-io/emmett';

export const addProductItem = (
  command: AddProductItemToShoppingCart,
  state: ShoppingCart,
): ProductItemAddedToShoppingCart => {
  if (state.status === 'Closed')
    throw new IllegalStateError('Shopping Cart already closed');

  const {
    data: { shoppingCartId, productItem },
    metadata,
  } = command;

  return {
    type: 'ProductItemAddedToShoppingCart',
    data: {
      shoppingCartId,
      productItem,
      addedAt: metadata?.now ?? new Date(),
    },
  };
};

export const removeProductItem = (
  command: RemoveProductItemFromShoppingCart,
  state: ShoppingCart,
): ProductItemRemovedFromShoppingCart => {
  if (state.status !== 'Opened')
    throw new IllegalStateError('Shopping Cart is not opened');

  const {
    data: { shoppingCartId, productItem },
    metadata,
  } = command;

  const currentQuantity = state.productItems.get(productItem.productId) ?? 0;

  if (currentQuantity < productItem.quantity)
    throw new IllegalStateError('Not enough products');

  return {
    type: 'ProductItemRemovedFromShoppingCart',
    data: {
      shoppingCartId,
      productItem,
      removedAt: metadata?.now ?? new Date(),
    },
  };
};

export const confirm = (
  command: ConfirmShoppingCart,
  state: ShoppingCart,
): ShoppingCartConfirmed => {
  if (state.status !== 'Opened')
    throw new IllegalStateError('Shopping Cart is not opened');

  const totalQuantityOfAllProductItems = sum(state.productItems.values());

  if (totalQuantityOfAllProductItems <= 0)
    throw new IllegalStateError('Shopping Cart is empty');

  const {
    data: { shoppingCartId },
    metadata,
  } = command;

  return {
    type: 'ShoppingCartConfirmed',
    data: {
      shoppingCartId,
      confirmedAt: metadata?.now ?? new Date(),
    },
  };
};

export const cancel = (
  command: CancelShoppingCart,
  state: ShoppingCart,
): ShoppingCartCancelled => {
  if (state.status !== 'Opened')
    throw new IllegalStateError('Shopping Cart is not opened');

  const {
    data: { shoppingCartId },
    metadata,
  } = command;

  return {
    type: 'ShoppingCartCancelled',
    data: {
      shoppingCartId,
      canceledAt: metadata?.now ?? new Date(),
    },
  };
};

export const decide = (command: ShoppingCartCommand, state: ShoppingCart) => {
  const { type } = command;

  switch (type) {
    case 'AddProductItemToShoppingCart':
      return addProductItem(command, state);
    case 'RemoveProductItemFromShoppingCart':
      return removeProductItem(command, state);
    case 'ConfirmShoppingCart':
      return confirm(command, state);
    case 'CancelShoppingCart':
      return cancel(command, state);
    default: {
      const _notExistingCommandType: never = type;
      throw new EmmettError(`Unknown command type`);
    }
  }
};

As you can see, it’s a simple set of functions that evaluate business rules (based on the input command and current state) and return an event as a result.

Okay, but how do I test it? Wouldn’t it be good if your library provided you with the recommended (but not enforced) method out of the box?

If you think the answer is yes, you should like what you’ll see next.

One of the biggest benefits of having repeatable patterns is testing, which Emmett helps to do out of the box. For Event Sourcing, the testing pattern looks like this:

GIVEN set of events recorded for the entity,
WHEN we run the command on the state built from events,
THEN we’re getting new event(s) as a result of business logic. Or the exception is thrown.

Let’s see how you can apply this pattern in Emmett.

import { DeciderSpecification } from '@event-driven-io/emmett';

const given = DeciderSpecification.for({
  decide,
  evolve,
  initialState: getInitialState,
});

describe('ShoppingCart', () => {
  describe('When empty', () => {
    it('should add product item', () => {
      given([])
        .when({
          type: 'AddProductItemToShoppingCart',
          data: {
            shoppingCartId,
            productItem,
          },
          metadata: { now },
        })
        .then([
          {
            type: 'ProductItemAddedToShoppingCart',
            data: {
              shoppingCartId,
              productItem,
              addedAt: now,
            },
          },
        ]);
    });
  });

  describe('When opened', () => {
    it('should confirm', () => {
      given({
        type: 'ProductItemAddedToShoppingCart',
        data: {
          shoppingCartId,
          productItem,
          addedAt: oldTime,
        },
      })
        .when({
          type: 'ConfirmShoppingCart',
          data: {
            shoppingCartId,
          },
          metadata: { now },
        })
        .then([
          {
            type: 'ShoppingCartConfirmed',
            data: {
              shoppingCartId,
              confirmedAt: now,
            },
          },
        ]);
    });
  });

  describe('When confirmed', () => {
    it('should not add products', () => {
      given([
        {
          type: 'ProductItemAddedToShoppingCart',
          data: {
            shoppingCartId,
            productItem,
            addedAt: oldTime,
          },
        },
        {
          type: 'ShoppingCartConfirmed',
          data: { shoppingCartId, confirmedAt: oldTime },
        },
      ])
        .when({
          type: 'AddProductItemToShoppingCart',
          data: {
            shoppingCartId,
            productItem,
          },
          metadata: { now },
        })
        .thenThrows(
          (error: Error) => error.message === 'Shopping Cart already closed',
        );
    });
  });

  const getRandomProduct = (): PricedProductItem => {
    return {
      productId: randomUUID(),
      unitPrice: Math.random() * 10,
      quantity: Math.random() * 10,
    };
  };
  const oldTime = new Date();
  const now = new Date();
  const shoppingCartId = randomUUID();

  const productItem = getRandomProduct();
});

We start by setting up a specification for our tests by passing:

decide - a function that groups our command handling into a single method,
evolve - a function that is used to build the current state from events,
getInitialState - the initial, empty state on which we apply events to get the current one.

DeciderSpecification helps you follow the Given/When/Then pattern. It builds the state internally based on the events passed in the Given part. It’ll run the business logic provided in the When part and then test the result events or an error.

It’s as simple as that! I know that using simple is dangerous, but I really think it’s harder to make it more straightforward. Or at least I haven’t found a better way yet.

WebApi

Let’s not stop here, as Emmett also helps you with integration and end-to-end testing. But before we go, let’s show you quickly how you could define Express.js WebApi with a little help from Emmett.

We need to install the Emmett Express.js npm package.

npm add @event-driven-io/emmett-expressjs

Having that the WebApi setup would look like:

import { type WebApiSetup } from '@event-driven-io/emmett-expressjs';
import { Router } from 'express';
import { evolve, getInitialState } from '../shoppingCart';

// Let's setup the command handler, we'll use it in endpoints
const handle = CommandHandler(evolve, getInitialState);

export const shoppingCartApi =
  (
    // external dependencies
    eventStore: EventStore,
    getUnitPrice: (productId: string) => Promise<number>,
  ): WebApiSetup =>
  (router: Router): void => {
    // We'll setup routes here
  };

The API definition is a function that takes external dependencies and returns the Web API setup. We’re also setting upCommand Handler that will orchestrate loading and storing events in the event store (see more in Emmett docs).

We can use it in the full setup like:

import { getInMemoryEventStore } from '@event-driven-io/emmett';
import { getApplication, startAPI } from '@event-driven-io/emmett-expressjs';
import type { Application } from 'express';
import type { Server } from 'http';

const eventStore = getInMemoryEventStore();

const shoppingCarts = shoppingCartApi(
  eventStore,
  getUnitPrice,
  () => new Date(),
);

const application: Application = getApplication({
  apis: [shoppingCarts],
});

const server: Server = startAPI(application);

We’ll use in-memory event store for now. Yes, it’s also provided out of the box!

Let’s not keep the setup empty for too long and define our first endpoint!

router.post(
  '/clients/:clientId/shopping-carts/current/product-items',
  on(async (request: AddProductItemRequest) => {
    // 1. Translate request params to the command
    const shoppingCartId = getShoppingCartId(
      assertNotEmptyString(request.params.clientId),
    );
    const productId = assertNotEmptyString(request.body.productId);

    const command: AddProductItemToShoppingCart = {
      type: 'AddProductItemToShoppingCart',
      data: {
        shoppingCartId,
        productItem: {
          productId,
          quantity: assertPositiveNumber(request.body.quantity),
          unitPrice: await getUnitPrice(productId),
        },
      },
    };

    // 2. Handle command
    await handle(eventStore, shoppingCartId, (state) =>
      addProductItem(command, state),
    );

    // 3. Return response status
    return NoContent();
  }),
);

Again, simple stuff. We:

Translate and request params to the command.
Run command handler on top of business logic.
Return the proper HTTP response.

Of course, we could make it even crisper and automagically do the request mapping, more conventional-based status resolution, decorators, and fire-command-and-forget, but we won’t. Why?

I prefer composability over magical glue. I believe that a healthy amount of copy-paste won’t harm you. I target removability and segregation of the code and making things explicit that should be explicit.

Still, I won’t tell you how to live! If you want to add more, feel free to do it. I want to give you basic building blocks and recommendations so you can build on top of that!

Let’s skip other endpoints to keep the focus on tests. They’ll look accordingly to the above one. You can find a complete definition in Emmett docs.

Integration Testing

Cool, we now have API up and running. We even tested our domain logic with unit tests. That’s great, but as you know, a lot can happen in the meantime. The request mapping or validation may fail; middlewares (like auth one) can say no. It’d be great to test it.

There are many different shapes of Testing: Pyramids, Honeycombs, Thropies etc. All of them share the goal of having them reliable and fast. However, agreeing on where the compromise is and where to put the biggest effort is, of course, challenging to agree.

No matter what your preference is, Emmett has got you covered.

Let’s say that you’re a fan of Hexagonal/Ports & Adapters Architecture and you’d like to test the whole flow being able to replace dependencies (adapters) with in-memory implementations to have your tests running in-memory. Such approaches have tradeoffs. The pros are that they run smoothly, allow a fast feedback loop, and run tests continuously. The downside is that they don’t validate all integration scenarios with real tools. Don’t worry, we’ll cover that later!

I heard that one picture could speak more than a thousand words, so let’s look at this one:

The picture shows the boundaries between the business logic and our application layer.

Our application layer is thin; it’s a vertical slice to which the entry point (port) is the WebApi endpoint. Inside it, we can do additional stuff like getting product prices and mapping requests with additional data added to the command. We handle commands in the business logic that return event(s). We’re storing them in the event store. This looks like that in the already known adding product to shopping cart code:

const handle = CommandHandler(evolve, getInitialState);

type AddProductItemRequest = Request<
  Partial<{ clientId: string; shoppingCartId: string }>,
  unknown,
  Partial<{ productId: number; quantity: number }>
>;

////////////////////////////////////////////////////
// Web Api
////////////////////////////////////////////////////

export const shoppingCartApi =
  (
    // external dependencies
    eventStore: EventStore,
    getUnitPrice: (productId: string) => Promise<number>,
  ): WebApiSetup =>
  (router: Router): void => {
    ////////////////////////////////////////////////////
    // Endpoint
    ////////////////////////////////////////////////////
    router.post(
      '/clients/:clientId/shopping-carts/current/product-items',
      on(async (request: AddProductItemRequest) => {
        const shoppingCartId = getShoppingCartId(
          assertNotEmptyString(request.params.clientId),
        );
        const productId = assertNotEmptyString(request.body.productId);

        const command: AddProductItemToShoppingCart = {
          type: 'AddProductItemToShoppingCart',
          data: {
            shoppingCartId,
            productItem: {
              productId,
              quantity: assertPositiveNumber(request.body.quantity),
              unitPrice: await getUnitPrice(productId),
            },
          },
        };

        await handle(eventStore, shoppingCartId, (state) =>
          addProductItem(command, state),
        );

        return NoContent();
      }),
    );
    // (...) other endpoints
  };

////////////////////////////////////////////////////
// Business Logic
////////////////////////////////////////////////////

export type AddProductItemToShoppingCart = Command<
  'AddProductItemToShoppingCart',
  {
    shoppingCartId: string;
    productItem: PricedProductItem;
  }
>;

export const addProductItem = (
  command: AddProductItemToShoppingCart,
  state: ShoppingCart,
): ProductItemAddedToShoppingCart => {
  if (state.status === 'Closed')
    throw new IllegalStateError('Shopping Cart already closed');

  const {
    data: { shoppingCartId, productItem },
    metadata,
  } = command;

  return {
    type: 'ProductItemAddedToShoppingCart',
    data: {
      shoppingCartId,
      productItem,
      addedAt: metadata?.now ?? new Date(),
    },
  };
};

Our slice has 3 ports that one can plug in:

WebApi endpoint, where the user can send a request (which will be translated to a command).
getUnitPrice function that gets the product price. Depending on our design, it may represent a call to an external service (e.g. with HTTP), calling a database (e.g. read model) or just running some computation. We’re also using it as an input to the command.
Event store, from which we load events and store new facts.

We already made those dependencies explicit, allowing us to replace the real ones in the tests. Also, as we know that we’re doing Event Sourcing then, why not take advantage of that and write our tests in the following style:

GIVEN set of events recorded for the entity,
WHEN we run the web API request,
THEN we’re getting new event(s) as a result of business logic. Or the error status is returned with Problem Details.

Let’s start with defining our WebApi specification. We’re using ApiSpecification class from @event-driven-io/emmett-expressjs.

import {
  getInMemoryEventStore,
  type EventStore,
} from '@event-driven-io/emmett';
import {
  ApiSpecification,
  getApplication,
} from '@event-driven-io/emmett-expressjs';

const unitPrice = 100;
const now = new Date();

const given = ApiSpecification.for<ShoppingCartEvent>(
  (): EventStore => getInMemoryEventStore(),
  (eventStore: EventStore) =>
    getApplication({
      apis: [
        shoppingCartApi(
          eventStore,
          () => Promise.resolve(unitPrice),
          () => now,
        ),
      ],
    }),
);

We’re also using the same getApplication known from the previous steps. The only difference is that we replaced real dependencies with the in-memory ones. ApiSpecification uses internally SuperTest package. It allows straightforward testing of Express.js, e.g. it ensures that server starts in a random port and provides the helpers for building test requests, which we’ll use in our tests.

Having it, we can define our tests as:

import {
  existingStream,
  expectNewEvents,
  expectResponse,
} from '@event-driven-io/emmett-expressjs';

describe('When opened with product item', () => {
  it('should confirm', () => {
    return given(
      existingStream(shoppingCartId, [
        {
          type: 'ProductItemAddedToShoppingCart',
          data: {
            shoppingCartId,
            productItem,
            addedAt: oldTime,
          },
        },
      ]),
    )
      .when((request) =>
        request.post(`/clients/${clientId}/shopping-carts/current/confirm`),
      )
      .then([
        expectResponse(204),
        expectNewEvents(shoppingCartId, [
          {
            type: 'ShoppingCartConfirmed',
            data: {
              shoppingCartId,
              confirmedAt: now,
            },
          },
        ]),
      ]);
  });
});

The test follows the similar Given/When/Then pattern as our unit tests but uses HTTP request to trigger the command handling and uses additional helpers to set up and verify data:

exisitingStream - allows you to specify the stream id and events existing in the stream. You can set more than one stream if your command handling logic requires that,
expectResponse - verifies the HTTP response with expected criteria like status code. You can also check the response body, headers, etc. For expected errors you can use expectError accordingly.
expectEvents - ensures that new events are appended to the specific streams in the event store.

Complete tests will look like this:

import {
  getInMemoryEventStore,
  type EventStore,
} from '@event-driven-io/emmett';
import {
  ApiSpecification,
  existingStream,
  expectError,
  expectNewEvents,
  expectResponse,
  getApplication,
} from '@event-driven-io/emmett-expressjs';
import { randomUUID } from 'node:crypto';

describe('ShoppingCart', () => {
  let clientId: string;
  let shoppingCartId: string;

  beforeEach(() => {
    clientId = randomUUID();
    shoppingCartId = `shopping_cart:${clientId}:current`;
  });

  describe('When empty', () => {
    it('should add product item', () => {
      return given()
        .when((request) =>
          request
            .post(`/clients/${clientId}/shopping-carts/current/product-items`)
            .send(productItem),
        )
        .then([
          expectNewEvents(shoppingCartId, [
            {
              type: 'ProductItemAddedToShoppingCart',
              data: {
                shoppingCartId,
                productItem,
                addedAt: now,
              },
            },
          ]),
        ]);
    });
  });

  describe('When opened with product item', () => {
    it('should confirm', () => {
      return given(
        existingStream(shoppingCartId, [
          {
            type: 'ProductItemAddedToShoppingCart',
            data: {
              shoppingCartId,
              productItem,
              addedAt: oldTime,
            },
          },
        ]),
      )
        .when((request) =>
          request.post(`/clients/${clientId}/shopping-carts/current/confirm`),
        )
        .then([
          expectResponse(204),
          expectNewEvents(shoppingCartId, [
            {
              type: 'ShoppingCartConfirmed',
              data: {
                shoppingCartId,
                confirmedAt: now,
              },
            },
          ]),
        ]);
    });
  });

  describe('When confirmed', () => {
    it('should not add products', () => {
      return given(
        existingStream(shoppingCartId, [
          {
            type: 'ProductItemAddedToShoppingCart',
            data: {
              shoppingCartId,
              productItem,
              addedAt: oldTime,
            },
          },
          {
            type: 'ShoppingCartConfirmed',
            data: { shoppingCartId, confirmedAt: oldTime },
          },
        ]),
      )
        .when((request) =>
          request
            .post(`/clients/${clientId}/shopping-carts/current/product-items`)
            .send(productItem),
        )
        .then(
          expectError(403, {
            detail: 'Shopping Cart already closed',
            status: 403,
            title: 'Forbidden',
            type: 'about:blank',
          }),
        );
    });
  });

  const oldTime = new Date();
  const now = new Date();
  const unitPrice = Math.random() * 10;

  const given = ApiSpecification.for<ShoppingCartEvent>(
    (): EventStore => getInMemoryEventStore(),
    (eventStore: EventStore) =>
      getApplication({
        apis: [
          shoppingCartApi(
            eventStore,
            () => Promise.resolve(unitPrice),
            () => now,
          ),
        ],
      }),
  );

  const getRandomProduct = (): PricedProductItem => {
    return {
      productId: randomUUID(),
      unitPrice,
      quantity: Math.random() * 10,
    };
  };

  const productItem = getRandomProduct();
});

You can use those tests as complementary to the business logic (e.g., testing the most important scenarios), or you may even replace unit tests with them. As they’re in memory, they’re fast enough to be run continuously.

You can also replace the in-memory store with the real one (e.g. EventStoreDB) and test your module in isolation from other modules. The choice is yours!

Again, in Emmett, we don’t want to force you to anything but give you options and the recommended safe path.

I encourage you to watch Martin Thwaites’ talk “Building Operable Software with TDD (but not the way you think)”. It nicely explains why I can now shift the balance from unit testing to integration testing.

End-to-End Testing

You may say:

Oskar, those tests are cool, but I’d like to either test business logic with unit tests or run my end-to-end tests treating the WebApi as a black box.

And I answer: sure, why not! I also give you help with that.

Let’s start by adding some flavour and use EventStoreDB this time. We need to install two more packages. One for adding implementation of the EventStoreDB event store:

npm add @event-driven-io/@event-driven-io/emmett-esdb

Now, we need to switch the in-memory implementation to EventStoreDB in WebApi setup. Updated will look as follows:

import { getEventStoreDBEventStore } from '@event-driven-io/emmett-esdb';
import { EventStoreDBClient } from '@eventstore/db-client';

const eventStoreDBClient = EventStoreDBClient.connectionString(
  `esdb://localhost:2113?tls=false`,
);
const eventStore = getEventStoreDBEventStore(eventStoreDBClient);

const shoppingCarts = shoppingCartApi(
  eventStore,
  getUnitPrice,
  () => new Date(),
);

It’s as simple as that; we’re injecting just a different implementation.

As EventStoreDB is a real database, we need to set it up for our tests. The simplest option is to use a Docker container. You can do it in multiple ways, but the fastest can be using TestContainers. The library allows us to easily set up containers for our tests. It automatically randomise ports, helps in teardown etc. (read more here, to see if it’s always a best option)

Emmett provides the package with additional test containers like the one for EventStoreDB. You need to install:

npm add @event-driven-io/@event-driven-io/emmett-testcontainers

:::

Having that, we can set our test container with:

import {
  EventStoreDBContainer,
  type StartedEventStoreDBContainer,
} from '@event-driven-io/emmett-testcontainers';

let esdbContainer: StartedEventStoreDBContainer;
describe('ShoppingCart E2E', () => {
  // Set up a container before all tests
  before(async () => {
    esdbContainer = await new EventStoreDBContainer().start();
  });

  // Stop container once we finished testing
  after(() => {
    return esdbContainer.stop();
  });
  // (...) Tests will go here
});

And create our test specification using the ApiE2ESpecification type:

import { type EventStore } from '@event-driven-io/emmett';
import { getEventStoreDBEventStore } from '@event-driven-io/emmett-esdb';
import {
  ApiE2ESpecification,
  getApplication,
} from '@event-driven-io/emmett-expressjs';

const given = ApiE2ESpecification.for(
  (): EventStore => getEventStoreDBEventStore(esdbContainer.getClient()),
  (eventStore: EventStore) =>
    getApplication({
      apis: [
        shoppingCartApi(
          eventStore,
          () => Promise.resolve(unitPrice),
          () => now,
        ),
      ],
    }),
);

The test will look accordingly to the integration tests, with the distinction that we also use HTTP requests for the setup. We’re also checking only responses; we treat the WebApi as a black box.

import { getEventStoreDBEventStore } from '@event-driven-io/emmett-esdb';
import { expectResponse } from '@event-driven-io/emmett-expressjs';
import type { StartedEventStoreDBContainer } from '@event-driven-io/emmett-testcontainers';

describe('When opened with product item', () => {
  it('should confirm', () => {
    return given((request) =>
      request
        .post(`/clients/${clientId}/shopping-carts/current/product-items`)
        .send(productItem),
    )
      .when((request) =>
        request.post(`/clients/${clientId}/shopping-carts/current/confirm`),
      )
      .then([expectResponse(204)]);
  });
});

Complete tests will look like this:

import { type EventStore } from '@event-driven-io/emmett';
import { getEventStoreDBEventStore } from '@event-driven-io/emmett-esdb';
import {
  ApiE2ESpecification,
  expectResponse,
  getApplication,
} from '@event-driven-io/emmett-expressjs';
import {
  EventStoreDBContainer,
  StartedEventStoreDBContainer,
} from '@event-driven-io/emmett-testcontainers';
import { randomUUID } from 'node:crypto';

describe('ShoppingCart E2E', () => {
  const unitPrice = 100;
  let clientId: string;
  let shoppingCartId: string;
  let esdbContainer: StartedEventStoreDBContainer;
  let given: ApiE2ESpecification;

  before(async () => {
    esdbContainer = await new EventStoreDBContainer().start();

    given = ApiE2ESpecification.for(
      (): EventStore => getEventStoreDBEventStore(esdbContainer.getClient()),
      (eventStore: EventStore) =>
        getApplication({
          apis: [
            shoppingCartApi(
              eventStore,
              () => Promise.resolve(unitPrice),
              () => now,
            ),
          ],
        }),
    );
  });

  beforeEach(() => {
    clientId = randomUUID();
    shoppingCartId = `shopping_cart:${clientId}:current`;
  });

  after(() => {
    return esdbContainer.stop();
  });

  describe('When opened with product item', () => {
    it('should confirm', () => {
      return given((request) =>
        request
          .post(`/clients/${clientId}/shopping-carts/current/product-items`)
          .send(productItem),
      )
        .when((request) =>
          request.post(`/clients/${clientId}/shopping-carts/current/confirm`),
        )
        .then([expectResponse(204)]);
    });

    it('should return details', () => {
      return given((request) =>
        request
          .post(`/clients/${clientId}/shopping-carts/current/product-items`)
          .send(productItem),
      )
        .when((request) =>
          request.get(`/clients/${clientId}/shopping-carts/current`).send(),
        )
        .then([
          expectResponse(200, {
            body: {
              clientId,
              id: shoppingCartId,
              productItems: [
                {
                  quantity: productItem.quantity,
                  productId: productItem.productId,
                },
              ],
              status: ShoppingCartStatus.Opened,
            },
          }),
        ]);
    });
  });

  const now = new Date();

  const getRandomProduct = (): PricedProductItem => {
    return {
      productId: randomUUID(),
      unitPrice: unitPrice,
      quantity: Math.random() * 10,
    };
  };

  const productItem = getRandomProduct();
});

KUDOS also goes to Thiago Valentim for contribution and delivering E2E tests feature.

Check also the full Emmett getting started guide and see the full sample in Emmett repository.

TLDR

I really think that Event Sourcing, with its repeatable patterns and focus on business, can streamline development. Yet, learning it may be challenging. You need to learn both new approaches and tooling. That’s why I decided to publish Emmett to package safe defaults and my state-of-the-art. I don’t want to replace or compete with existing tooling but guide you through this journey.

See that tests are first-class citizens in Emmett. I think that’s a pretty rare thing. I want to make getting the trust in your code and yourself smoother while going through the Event Sourcing journey. The proportion between unit, integration and end-to-end tests is up to you. The most important part is that I’ve got you covered.

I’d be really curious about your thoughts on this approach. Feel free to comment here or Join our Discord and discuss it with me there!

Cheers!

Oskar

Join my Event Sourcing workshops at Techorama and DDD Europe and speed up your journey!

Event Sourcing is a pattern that is quickly gaining popularity. Many companies see the advantages it brings, e.g. business focus and keeping all information about the business process. It can also be a great input to analytics, providing increased diagnostics and tracing. Nowadays, that’s essential for running a system on the scale.

There are already decent materials teaching how to start the journey with Event Sourcing. I tried to cover a lot of aspects of it in this blog. So starting is easier, but it requires a ramp-up phase. Once you pass it, it’s not easy to get certainty that our design will work on production. Even when we get there, it’s worth revisiting lessons learned and discovering what we can improve in preparing for the upcoming challenges.

We can hit accidental complexity. That can be overwhelming. I learned that the hard way. I tackled those problems in my projects as:

a developer and architect,
consultant with my clients
tooling author.

I think that those versatile lessons could also be useful for you.

Do you want to speed up your journey? Do you like onsite conferences?

You can mash those two together, as I’ll be giving pre-conference workshops on my favourite conferences:

Techorama - 6th of May,
DDD Europe - 27th-29th of May.

Practical Introduction to Event Sourcing at Techorama

Date: 6th of May.

The workshop will be hands-on and teach you how to use Event Sourcing, giving you solid foundations. You will understand after them:

when and how to use it and what benefits it brings,
how to reflect your business logic in the code using events,
differences to the classical approach,
different tools such as Marten and EventStoreDB, and the differences between them,
how to use Event Sourcing on your system,
challenges related to Event Sourcing and recommended solutions.

The general plan looks like this:

Introduction to Event Sourcing. Basic terminology (event, stream of events, command), differences from the classical approach.
What is Event Sourcing, and how is it different from Event Streaming. Advantages and disadvantages.
Write model and data consistency guarantees.
Various ways of handling business logic: Aggregates, Command Handlers, functional approach.
Projections and best practices for building a read model on the.
Event Sourcing in the context of application architecture and integration with other approaches.
Saga, Choreography, Process Manager, handling distributed processes.
Good and bad practices in modelling and handling events.
Event Sourcing on production, evolution, event versioning, etc.

This should build a good foundation for your Event Sourcing journey. I won’t lie: you won’t be an expert after it, but you’ll be able to start applying it to your projects, learn what you don’t know, and have a plan for advancing it.

The requirements to join are simple:

understanding of the basic building blocks of the application design,
experience in one of the languages and platforms: C#, Java, TypeScript (code exercises will be done using them),
positive and open-minded attitude.

Interested? Sign up here.

Production-Grade Event Sourcing: Modelling, DevOps, Process

Dates: 27th-29th of May

The goal of this workshop is to strengthen all of the critical aspects of running event-sourced on production:

modelling (like keeping streams short, managing consistency and efficiently handling business logic, best practices and anti-patterns),
managing schema evolution, so events versioning,
advanced projections design and resiliency, so how to get the best from events to efficiently get read models and insights from them,
DevOps techniques (like traceability, blue-green projection rebuilt, archiving old data),
handling distributed processes and integrating with other systems,
putting event sourcing as a vital part of the whole architecture (so how to integrate it with non-event-sourced modules, defining private and public schema).

After the workshop, you’ll understand and practice techniques that will allow you to run your Event-Sourcing system on production.

All the exercises will be backed by the versatile experience I got throughout my career. I’ll share my experience building systems like Marten and EventStoreDB and working as an architect and consultant in many Event Sourcing projects.

We’ll start with the project of the system that looks fine as the first production deployment. Through group exercises, a mixture of modelling and practical coding exercises, we’ll analyse potential issues step by step. We’ll learn how to refactor it step by step without breaking it. That will allow us to fix the boundaries, consistency guarantees, and resiliency. We’ll practice both adding new capabilities and changing existing ones. We’ll practice schema evolution and DevOps practices like projection rebuilds. We’ll end with a more robust and prepared-for-the-next-challenges system.

Prerequisities?

The workshop is designed for people who already have experience with Event Sourcing. Especially for those who would like to assess the production-readiness of their design or already deployed system. I won’t start from basics but assume:

experience in how Event Sourcing works (storing events, building state from them),
knowledge of how to model business logic and projections/read models,
Understand that event sourcing is not event streaming and why streaming solutions like Kafka and Pulsar should not be used as event stores.

I assume you know modelling techniques like EventStorming, C4 and modularity concepts (microservices, monolith, CQRS, etc.). And that you were already building and designing web applications.

Most of the exercises will be focused on general understanding and will not be technology-specific, but practical will require knowledge of one of the languages: C#, Java, or TypeScript. We’ll use Marten and EventStoreDB as event stores examples.

If you took my Practical Introduction workshop, you’ll also be fine with taking the Advanced workshop. Especially that, you’ll also have three weeks to do some additional preparations. I can help you spend this time as well as you can!

Interested? Sign up here.

Can’t be there or want a private experience?

No worries, I also run private trainings and other forms of help like:

architecture review,
consultancy,
proof of concepts,
mentoring.

I’m happy to help!

If you don’t know what to choose, don’t hesitate [email protected]">to contact me via email. I’m here to help.

Check the details on my training page.

A workshop is the most effective way to jump-start.

You can also try to do Introduction to Event Sourcing - Self-Paced Kit on your own. I think that’s a decent path, but longer, the choice is yours!

Cheers!

Oskar

How to tackle compatibility issues in ECMA Script modules (and in general)

Do you recall moments when you’re sitting and closing dozens or more browser tabs? Most of them are Google, GitHub, Blogs, and others. You click and close them one by one, not even checking if they’re worth keeping. That’s a sign you learned more about the problem than you ever wanted. But that also means that you solved it. If you didn’t, you’d just add more open tabs. I have that feeling fresh now.

Yesterday, I solved one of those types of issues. It’s not so surprising knowing that I just started Emmett, a JS/TS library for event-driven applications. JavaScript Land has a lot of rabbit holes. And boy, you can go down in them!

JavaScript can be seen as wild, but at least they have the ECMAScript specification that unifies standards. It’s widely discussed before acceptance but also inert in application.

For instance, ES Modules (ECMAScript Modules) were introduced in ECMAScript 6 (ES6) in 2015. Yes, the year when Netflix released Narcos, Jon Snow died, and Bruno Mars’ Uptown Funk was on top of Billboard charts. Nice memories? Guess what? For the JS community, they’re still fresh, as migration to ES Modules is still ongoing.

ES Modules

What are ES Modules? They brought a standardised module system to JavaScript, addressing several limitations of the previously widely used module patterns (such as CommonJS and AMD). ES Modules were introduced for several reasons:

Standardisation: Before ES Modules, the JavaScript ecosystem was fragmented with multiple module systems (e.g., CommonJS for Node.js and AMD for asynchronous module loading in browsers). ES Modules provide a single, standardised module system used across different environments, enhancing interoperability.
Static Analysis and Tree Shaking ES Modules support static analysis, allowing tools to perform optimisations such as tree shaking (eliminating unused code) due to their static structure (import and export statements are at the top level and cannot be conditional).
Improved Performance: Due to the possibility of static analysis, JavaScript engines can optimise the loading and bundling of modules more efficiently, potentially improving load times and execution performance.
Better Code Organization. ES Modules encourage a more modular and maintainable code structure, making it easier to develop, understand, and maintain large codebases.
Native Browser Support. ES Modules are natively supported in modern browsers, allowing modules to be used directly without needing module bundlers or transpilers.

All that sounds great, but migration can be tedious. You need to change not only your build tooling but also the syntax in the code. For instance, in CommonJS, your module declaration could look like:

const add = (a, b) => a + b;
const subtract = (a, b) => a - b;

module.exports = { add, subtract };

And usage with imports as:

const math = require('./math');

console.log(math.add(2, 3));
console.log(math.subtract(5, 2));

In ES modules, the declaration looks:

export const add = (a, b) => a + b;
export const subtract = (a, b) => a - b;

And usage:

import { add, subtract } from './math.js';

console.log(add(2, 3));
console.log(subtract(5, 2));

With AMD, the leap is even bigger. So yeah, essentially, you need to go through all files and gradually migrate. For known reasons, that’s hard and takes time. Not always that’s justified financial-wise. The library creators have to support both, which, for obvious reasons, is tricky.

Do you know Quines? Per Wikipedia:

A quine is a computer program that takes no input and produces a copy of its own source code as its only output.

The variations of quines are Ouroboros programs:

The quine concept can be extended to multiple levels of recursion, giving rise to “ouroboros programs”, or quine-relays. This should not be confused with multiquines.

There are people, like Yusuke Endoh, that do things like Quine Relay:

QR.rb is a Ruby program that generates a Rust program that generates a Scala program that generates …(through 128 languages in total)… a REXX program that generates the original Ruby code again.

Yeah, people are having fun.

In JS land, you might not have fun, but you may need to write your quines. How do you write code that supports both styles, CommonJS and ES Modules? There are multiple options. You can use:

separate file extensions .mjs for ES Modules and .js for CommonJS,
TypeScript and ask the TS compiler to generate files compatible with both. That’s nice, as TypeScript is ECMA Script compatible by itself.
bundlers that shake and bake and deliver nice code to you.

Sweet, but all has its price. Having multiple files for the same code might lead to the dual package hazard, where both versions of your package could be loaded simultaneously, potentially leading to bugs and bloated bundles. If you’re a package creator, then that means your life gets wilder from time to time. Package Creator? Yeah, that’s me.

Issues with ES Modules

Getting back to my closing browser tabs habit.

I recently released several of Emmett’s versions. They brought:

EventStoreDB support,
Integration and Acceptance testing for WebAPI,
basic Fastify support,
state time travelling capabilities,
and more.

I was happy and kinda proud as external contributors delivered some of those changes. That’s motivating, especially in the early phases of the OSS project. But happiness has its deadline. For OSS, this deadline comes when someone tries to use a brand-new version.

Emmett appears incompatible with ES Modules. That was surprising, as I was using them in the code, and I thought I had configured that correctly. Famous last words!

While importing, you got:

import { getInMemoryEventStore } from '@event-driven-io/emmett';
         ^^^^^^^^^^^^^^^^^^^^^
SyntaxError: Named export 'getInMemoryEventStore' not found. The requested module '@event-driven-io/emmett' is a CommonJS module, which may not support all module.exports as named exports.
CommonJS modules can always be imported via the default export, for example using:

import pkg from '@event-driven-io/emmett';
const { getInMemoryEventStore } = pkg;

    at ModuleJob._instantiate (node:internal/modules/esm/module_job:132:21)
    at async ModuleJob.run (node:internal/modules/esm/module_job:214:5)
    at async ModuleLoader.import (node:internal/modules/esm/loader:329:24)
    at async loadESM (node:internal/process/esm_loader:28:7)
    at async handleMainPromise (node:internal/modules/run_main:113:12)

Of course, I couldn’t reproduce that locally, but “works on my box!” isn’t the answer your users would expect. So, I had to dig deeper. I’m using TypeScript. I checked the generated files, and they seemed fine. They had all those .js, .mjs, .ts, .mts files, etc. Yet, I found the potential issue in those closed tabs. I didn’t give a hint about which files are entry points for CommonJS imports and which for ES Modules.

Essentially, you need to declare it in your package.json file like this:

{
  "name": "@event-driven-io/emmett",
  "description": "Emmett - Event Sourcing development made simple",
  // (...)
  "exports": {
    ".": {
      "import": {
        "types": "./dist/index.d.mts",
        "default": "./dist/index.mjs"
      },
      "require": {
        "types": "./dist/index.d.ts",
        "default": "./dist/index.js"
      }
    }
  },
  "main": "./dist/index.js",
  "module": "./dist/index.mjs",
  "types": "./dist/index.d.ts",
  "files": [
    "dist"
  ],
}

Now, when someone uses require syntax from CommonJS, it goes to the js and ts files; if someone uses ES Modules, it goes to ts and mts. That’s cool, but I wasn’t still able to reproduce it.

Providing a quick fix is fine, but if you don’t know how to verify it, it is not a fix; it’s a rumour. That can lead to regression when you introduce a new change that breaks it again.

I wouldn’t like to have that. A good indicator of the OSS community quality is the engagement from users. And it sounds like Emmett already has a thriving community. I got reproducible steps with a project; Thanks Mateusz! But yeah, one problem solved, and a new one appears. Mateusz confirmed my suspicion about the missing exports in package.json was correct; that solved one issue, but another one popped up for him. Of we go to detective work, thanks again, Mateusz!

The next problem was:

Directory import 'emmett/packages/emmett/dist/commandHandling' is not supported resolving ES modules imported from emmett/packages/emmett/dist/index.mjs

I’m exporting them to the root ’.’ to provide a straightforward entry point for Emmett’s features without searching through nested structures. You can import all code like:

import type { Event, Command } from '@event-driven-io/emmett';
import { IllegalStateError, CommandHandler } from '@event-driven-io/emmett';

Yet, of course, I don’t keep all the source codes in a single file; I have a nested structure. I’m using a feature called “re-exporting” to achieve that. Essentially, I have a default index.ts file in a directory that imports files from subdirectories and exports them again under the flattened structure. This can look like that:

export * from './commandHandling';
export * from './errors';
export * from './eventStore';
export * from './serialization';
export * from './testing';
export * from './typing';
export * from './utils';
export * from './validation';

And here’s the deal. When working with TypeScript (.ts, .tsx) and ECMAScript Modules (.mjs) in Node.js, you might encounter challenges due to the differences in file extensions and module resolution between TypeScript and JavaScript, particularly when using export * from syntax.

TypeScript introduced .mts as the file extension for ECMAScript modules containing TypeScript code, aligning with .mjs for JavaScript ECMAScript modules. Yet, you may face the following issues:

Module Resolution: Node.js treats .mjs files as ES modules natively. TypeScript’s .mts extension for ES module files also aligns with this convention. However, when importing or re-exporting from these modules, you might encounter issues where TypeScript or Node.js can only correctly resolve the other module type with explicit hints.
Type Definitions: When using export * from in TypeScript to re-export ES module contents from a .mjs file, TypeScript may not be able to apply the correct types unless those types are explicitly declared or inferred from .d.ts (declaration files). This can lead to type-checking issues.
Tooling and Build Processes: Tooling might need additional configuration to handle .mts and .mjs files correctly, especially when both TypeScript and JavaScript modules are mixed in a project. This includes setting up TypeScript, bundlers (Webpack, Rollup), and other tools (Babel) to recognise and process these files appropriately.
Path Importing and File Extensions: TypeScript and Node.js have different default behaviours regarding file extensions. TypeScript might require explicit extensions (e.g., .mts) in import statements, which is not always true for .mjs in Node.js due to its native module resolution strategy.

Plenty of options to make it wrong, aye? Before we discuss the (embarrassing) fix, let’s learn how I reproduced it.

Checking the compatibility

We’re in the moment when I had a project on which I was able to reproduce the issue. I had some clues about what may be wrong (dozens of clues!). I decided to automate it before providing the fix. Such automation is more complex. As you know now, it’s not possible to ensure that the generated code by the TypeScript compiler is compatible with different module styles. The only way is to test it on the real project.

The test project I got was using Nuxt.js. Nuxt.js is a framework built on Vue.js that offers server-side rendering, static site generation, and automatic code splitting to simplify Vue app development. It’s chosen for its ease of use, SEO benefits, performance optimisation features, and ability to streamline project setup and deployment.

It’s actually not important that it uses this particular framework, but what it brings. As always, strong powers can also be weaknesses. The fact that it has built-in bundling and code splitting and allows using both the backend and front end in one place can cause more compatibility issues. Why?

When TypeScript code is bundled, tools like Webpack, Rollup, or Parcel take care of module resolution and dependencies and can transpile the code to ensure compatibility across different environments. These bundlers often smooth over discrepancies between module formats (e.g., CommonJS vs. ES Modules) and file extensions. However, when you work with unbundled TypeScript code in a project that uses ES Modules, several issues can arise, particularly related to module resolution, file extensions, and differences in syntax between TypeScript and native ES Modules in Node.js.

Technically, the flow looks as follows

Npm package TypeScript code is transpilled into JavaScript, then minified, bundled and packaged.
Now, if you’re using such a package, you need to load the proper JS code, find TypeScript type maps and import it into your application code. Then, if you’re using TypeScript and frameworks like Nuxt, it’ll be transpilled, minified, bundled, and packaged again.

It’s like Chinese Whispers game. As in this game, the solution may not be what we expect.

The other challenge is that Nuxt wasn’t failing during the build but when it was generating the static HTML files. That’s the power of it: you can run the same code in the backend and browser, but it also adds more trouble. Especially that when running its built-in command:

npm run generate --fail-on-error

It wasn’t failing. It silently succeeded in redirecting errors to Standard Error Output (stderr). No one said that’s going to be easy, but that’s why we’re here!

I decided to take the following path for checking the compatibility:

Put the sample application with import error for ES Modules into Emmett repository. This is important for the Red, Green Refactor. Knowing that it fails, we get certain that it’s fixed after the change.
Extend the existing GitHub Action pipeline that builds and tests all packages during each Pull Request and main branch commit.
Extend it by building Emmett’s npm package with npm pack. This creates a tarball of your package, simulating what would be uploaded to npm without actually publishing it. This step is crucial for testing the package as it would appear to consumers. I wouldn’t like to publish the package only to realise that I broke compatibility. I needed to do it before that.
Install such tarball file as a package reference in the sample application. Npm allows doing that without additional steps to set up the package server or file share. The soundest way to validate integration and compatibility is to install the package in a sample project that uses ES Modules and attempts to import code from your package.
Build the Sample Project and check if there are no errors now.

That sounds simple, but how do you tackle it? Let’s go through it step by step.

The first two steps are pretty straightforward. I copied the project as it is to ensure that I’m following the reproduction steps. I also had the initial pipeline. Let’s go to the next steps

Extend it by building Emmett’s npm package with npm pack.

In Emmett, I’m using a mono repo based on the NPM Workspaces to tame the complexity of building multiple packages. That’s a topic on its own, so let’s leave the details for another blog article. Most importantly, all packages are nested in the packages folder, and npm has built-in ways to work only on specific packages. To pack a particular package (in this case, core Emmett package), you need to call:

npm pack -w @event-driven-io/emmett

The -w parameter means we’re running the command for specific workspaces. In this case, the core package workspace. The command will generate a tarball in the root folder. That’s not perfect for our case, especially since it’ll have the package version in its name, for instance: event-driven-io-emmett-0.5.2.tgz. That means it’s not repeatable; when we change the version, it’ll have a different one.

Luckily npm pack have more options. We can specify the file destination and also return the result metadata in JSON format:

npm pack --json --pack-destination './e2e/esmCompatibility' -w @event-driven-io/emmett

The ./e2e/esmCompatibility is the location of my sample project.

Result JSON will contain all metadata about the generated package. We don’t need all of them; we just want to extract the file name. We’re running commands in GitHub actions, so Linux, why don’t we use some of the native goodies like pipelining and tools like jq? jq tool is a lightweight and flexible command-line JSON processor, enabling users to easily parse, filter, and transform JSON data. That’s precisely what we need!

We can do:

npm pack --json --pack-destination './e2e/esmCompatibility' -w @event-driven-io/emmett | \
jq -r '.[] | .filename'

This will return the raw value of the filename property of the object in the results array. Why do we need it? In the next step, we’ll need to pass it to the npm install command.

We must export the filename to some GitHub Actions environment variable to make it available in a separate step of the GitHub Action pipeline. GitHub action allows us to do that by:

- name: Pack Emmett locally to tar file
  shell: bash
  run: echo "PACKAGE_FILENAME=$(our bash script)" >> $GITHUB_ENV

So our whole step will look like:

- name: Pack Emmett locally to tar file
  shell: bash
  run: echo "PACKAGE_FILENAME=$(npm pack --json --pack-destination './e2e/esmCompatibility' -w @event-driven-io/emmett | jq -r '.[] | .filename')" >> $GITHUB_ENV

Install such tarball file as a package reference in the sample application

This part is simple; our sample application already has the tarball file with packed library code. We have the filename in our environment variable. Let’s use it:

- name: Use it in the compatibility test project
  working-directory: ./e2e/esmCompatibility
  run: npm install ${{ env.PACKAGE_FILENAME }}

Just to be sure that packages were refreshed, we can also run npm install:

- name: Install packages in the compatibility test project
  working-directory: ./e2e/esmCompatibility
  run: npm install

Build the Sample Project and check if there are no errors now

To reproduce the issue, we need to try to build a sample project. As mentioned earlier, the challenge with this Nuxt setup is that Nuxt does not fail during the build but when it generates the static HTML files. It silently succeeded in redirecting errors to Standard Error Output (stderr). The error lines will look as that:

Error:  [nuxt] [request error] [unhandled] [500] Directory import '/home/runner/work/emmett/emmett/e2e/esmCompatibility/node_modules/@event-driven-io/emmett/dist/commandHandling' is not supported resolving ES modules imported from /home/runner/work/emmett/emmett/e2e/esmCompatibility/node_modules/@event-driven-io/emmett/dist/index.mjs
Error:  [nuxt] [request error] [unhandled] [500] Directory import '/home/runner/work/emmett/emmett/e2e/esmCompatibility/node_modules/@event-driven-io/emmett/dist/commandHandling' is not supported resolving ES modules imported from /home/runner/work/emmett/emmett/e2e/esmCompatibility/node_modules/@event-driven-io/emmett/dist/index.mjs

So we’ll need to do more lifting and parse the output. Again, Linux tools like grep are pretty straightforward. As long as you know the syntax, which wasn’t my precise case, I had to learn. The grep command searches through input text, filtering for lines that match a specified pattern, and outputs those lines.

The challenge with grep is that we need to fail our script when matches are found, and grep doesn’t do that. It returns found lines. Yet, we can deal with that with a simple if. Our shell script can look as follows:

if npm run generate 2>&1 | grep -iF '[request error]'; then
  echo "Errors found, failing the step." && exit 1
else
   echo "No errors found, proceeding..."
fi

The script runs npm run generate, and 2>&1 error messages with the standard output to be scanned together. 2 stands for stderr and 1 for stdout. This merging is crucial because it ensures no message is missed, whether it’s an error or not.

After merging, the script uses the pipeline operator | to send this combined stream to grep, which looks for “[request error]. The —F means it treats the search phrase as a fixed string, not a regexp pattern. The -i param makes the search case-insensitive.

Finding this phrase triggers “Errors found, failing the step.” and stops the script with exit 1, indicating an error occurred. It is crucial to tell GitHub Actions that the step failed.

If the phrase isn’t found, “No errors found, proceeding…” is printed, and the step succeeds.

Complete pipeline

The complete pipeline with all the steps looks as follows:

name: Build and test

on:
  # run it on push to the default repository branch
  push:
    branches: [main]
  # run it during pull request
  pull_request:

jobs:
  build-and-test-code:
    name: Build application code
    # use system defined below in the tests matrix
    runs-on: ${{ matrix.os }}

    strategy:
      # define the test matrix
      matrix:
        # selected operation systems to run CI
        os: [ubuntu-latest] #, windows-latest, macos-latest]
        # selected node version to run CI
        node-version: [20.11.1]

    steps:
      - name: Check Out Repo
        uses: actions/checkout@v4

      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v4
        with:
          # use the node version defined in matrix above
          node-version: ${{ matrix.node-version }}
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Build TS
        run: npm run build:ts

      - name: Run linting (ESlint and Prettier)
        run: npm run lint

      - name: Build
        run: npm run build

      - name: Test
        run: npm run test

      - name: Pack Emmett locally to tar file
        shell: bash
        run: echo "PACKAGE_FILENAME=$(npm pack --json --pack-destination './e2e/esmCompatibility' -w @event-driven-io/emmett | jq -r '.[] | .filename')" >> $GITHUB_ENV

      - name: Use it in the compatibility test project
        working-directory: ./e2e/esmCompatibility
        run: npm install ${{ env.PACKAGE_FILENAME }}

      - name: Install packages in the compatibility test project
        working-directory: ./e2e/esmCompatibility
        run: npm install

      - name: Build the compatibility test project final
        working-directory: ./e2e/esmCompatibility
        shell: bash
        run: |
          if npm run generate 2>&1 | grep -iF '[request error]'; then
            echo "Errors found, failing the step." && exit 1
          else
            echo "No errors found, proceeding..."
          fi

The Fix

Ok, reproduction steps are great, but what did I do to fix the issue? The Fix and source of failure were somewhat embarrassing (as always).

I’m using tsup to bundle and minimise my package code. tsup is known for its simplicity and efficiency. It focuses on producing minimal and fast JavaScript output. It automatically handles TypeScript compilation and has built-in support for tree shaking, which helps reduce the bundle size. It’s a great and simple tool, but it can still fail in the hands of sloppy developers like me.

You configure it through the tsup.config.ts. My looked look like this:

import { defineConfig } from 'tsup';

const env = process.env.NODE_ENV;

export default defineConfig({
  splitting: true,
  clean: true, // clean up the dist folder
  dts: true, // generate dts files
  format: ['cjs', 'esm'], // generate cjs and esm files
  minify: env === 'production',
  bundle: env === 'production',
  skipNodeModulesBundle: true,
  entryPoints: ['src/index.ts'],
  watch: env === 'development',
  target: 'esnext',
  outDir: 'dist',
  entry: ['src/**/*.ts', '!src/**/*.spec.ts'], //include all files under src but not specs
  sourcemap: true,
  tsconfig: 'tsconfig.build.json', // workaround for https://github.com/egoist/tsup/issues/571#issuecomment-1760052931
});

Guess what I did wrong?

Yup, I tried to be sneaky. And writing sneaky code is not the smartest move. I didn’t set the env before publishing to production. Because of that, it wasn’t bundling and minimising code, but leaving it nested.

When code is bundled, it’s effectively placed in a single file (well, almost; it’s chunked to allow asynchronous, gradual loading). Thanks to that, you don’t get the issue of referencing the wrong files of the different module types. Depending on the tooling configuration, unbundled code may have issues locating the proper ES Modules or CommonJS files. With bundling, this problem disappears. So The Fix was just to change those two lines from:

import { defineConfig } from 'tsup';

const env = process.env.NODE_ENV;

export default defineConfig({
  // (...)
  minify: env === 'production',
  bundle: env === 'production',
  // (...)
});

import { defineConfig } from 'tsup';

const env = process.env.NODE_ENV;

export default defineConfig({
  // (...)
  minify: true,
  bundle: true,
  // (...)
});

Curtain!

TLDR

Reflecting on this whirlwind journey of making Emmett compatible with ES Modules and CommonJS has been quite the ride. Delivering OSS packages is a decent opportunity to learn new stuff. But not always; you’d like to have such opportunities so often. Despite being a 2015 initiative, the move to ES Modules feels as fresh and ongoing as ever. It’s like keeping pace with the never-ending updates in our favourite TV series.

One may say that JS land is terrible, and what I did is too much of a heavy lifting. And one could be right. But even though I got the hard lesson, I see that as positive. JS community is thriving. The diversity can sometimes be overwhelming, but at least you have a choice and tools to fix it. Having too big a choice is still better than monoculture and the single way. Especially if you have standards to which the community is moving.

Transitions are always hard, and compatibility is never easy. I hope this article explains how to deal with compatibility issues and harder-to-find bugs. It’s essential to find a repeatable way to avoid regression and then make the fix.

The approach explained can also give you a general mental framework for tackling compatibility issues in other environments.

Last but not least, go check Emmett. It’ll take your event-driven applications back to the future.

Join also our Discord to get a live stream of fun like this!

Keeping our overachieving freak on a leash

Being busy is not something we strive for; it’s an axiom. Social media are not helping; our whole lives are on the plate. And we’re chefs serving fast food.

We show how we dance on TikTok, we show where we go on Instagram, and we show how charming we are on Tinder. We show how intelligent and successful we are on LinkedIn.

We also show our engagement by checking Slack and work email after hours. And hey, are they really “after”?

Of course, we’re doing none of that to show up. We’re casually elegant and casually busy.

Sometimes, all of that is not intentional; sometimes, it “just happens”, and that’s something to watch for. If we influence others, we may unintentionally boost their grind for business.

You may say you have just 50 followers on Instagram and are not an influencer, so we’re good. You may be right, but… Maybe you’re a tech lead sending emails after hours or a friend sending work-related links to your colleagues. All of that, even if unintentional, can lead to building a false view of yourself as constantly busy. And you may create an impact that you don’t want to make.

I also should be more careful in what I’m posting and how I’m posting it. I’m not doing much overtime work now. One of the reasons why I went “solo” was to get more time for family life, be more flexible and be less stressed. I was almost burned out in my last regular job. So, I know how hard it can be and how important it is to take care of mental health.

Currently, I don’t have full-time assignments. Most of what I do nowadays is related to consulting and workshops. My running joke is that I’m earning money from consulting for my OSS passion (so Marten and Emmett).

That gives me more time to work on OSS and content, as that’s also my job now. Visibility helps to get new contracts. So, I spend more time blogging and doing OSS than the average person. Doing stuff in public helps me to motivate. I like to learn and build in public and share my findings. Plus, it seems to help some people.

So it’s not my intention to look busy. Yet…

Yet, it may look from the outside like I’m busier than I really am. And that’s dangerous as this can cause FOMO and make unhealthy comparisons.

So what to do about it? And what you can take from this blog?

If we want to respond on Slack/Discord/Email in the evening, hoping that someone will read it in the morning, let’s not do that. Just answer in the morning or schedule the message to be sent later.

If we want to share something, consider whether this message is worth it and if it has content besides showing ourselves as busy or grinding.

Let’s keep our overachieving freak on a leash.

Should you always keep streams short in Event Sourcing?

In the last article and others I did my best explaining why keeping streams short is important in Event Sourcing. I also showed you how. That should take you far enough, especially if you talk to your business. That will help you to find the lifecycle. But what if it doesn’t?

What if your entity doesn’t have such a lifecycle? Should we artificially find it for the sake of the specifics of the event model? Example? Personal or company data like names, addresses, etc.

The question sounds simple, but the answer is, as always, nuanced. I have the rule that if I answer “it depends”, then I should follow it up with “…on this and that”. Let’s then discuss those thises and thats.

Should you event-source everything?

You can, but will that give you a lot of additional benefits? What could be the benefit of event sourcing company data change events? Quite often, those data are kept in the event store to have them for read model updates or use them as a state-carried transfer. Those are valid reasons, as they can add some event-driven flavour and reactive way of integrating some of those changes with the rest of the application.

Maybe, in this case, it’s better to use a state-first approach for those entities. What I mean by that is treating the state as the source of truth, using it to validate business rules. We can still record events together with updating the state.

Even if we append those events to the event store, we won’t be doing Event Sourcing, and that’s fine! It’s not Event Sourcing, as events are not a source of truth but are side effects of changing the state. Our main intention is not to record events but to change state and notify others about them. Logically, we’re not appending events but publishing them.

We can still use event store for that and benefit from its publish-subscribe capabilities. We can also spare by adding yet another technology for that and spare ourselves another moving part.

If you’re doing that, then you’re doing Event Streaming, not Event Sourcing. In that case, the length of the stream doesn’t matter so much, as you’re not reading from it to get the state; you just subscribe to its tail. We’re using an event store as a queue to forward events to subscriptions/read models.

In Marten, it’s easier to keep consistency for that case than in EventStoreDB, as you can store the state as a document and append events in the same transaction.

In EventStoreDB, you don’t have built-in read models, so you would need to use an additional database. You could use user-defined projection and aggregate state from events. Then, you’d read the state from the last event in the projection stream. I saw one customer having 160,000,000 events in the stream, and the database survived. But please don’t do that. Projections are always run on the leader node, impacting the cluster’s performance. It’s also challenging to test them. If you want to do it, limit the maximum count of events.

Ensure also that you’re not doing CRUD Sourcing or Property Sourcing.

When does keeping streams longer become an issue?

If the stream doesn’t have a lot of events and performance is not critical here (fameous last words), then it may be okay to keep it living longer. That company details or address doesn’t change often. You probably won’t have a lot of events in such a stream. However, it may be living for a long time.

Most of the considerations on keeping streams short relate to those that:

have business lifecycles,
are actively accessed,
performance matters for their business case.

The first point relates to understanding the business process, and the others relate to technical performance requirements.

Company address or details won’t change too often, and performance is not critical, so it should be fine.

So, should I keep those streams short or not?

I think that it’s still worth setting some lifetime for our data and not trying to keep it forever. It’s about the data governance practices (but also storage size, which eventually may matter). So stuff that I described in the “GDPR series”:

I wrote there that:

Even for non-user data, most of us would have a challenge answering that, not even speaking about having policies. Yes, we don’t care much about data governance and privacy practices. The reality is we keep data until application logic removes it. We’re keeping all data because it may be useful in future, just in case. We’re starting to notice that when our database is overloaded or we don’t have space for backups.

Even though Company Data doesn’t have a clear business lifecycle, you don’t need to keep all the history of its updates for business logic. It’s still worth defining the data retention and archive strategy, e.g. once per year or another period, publish a summary event with the current data and truncate past events. That still allows you to rebuild your read models and get clear causality of processing.

This strategy is also useful for cases where you actually don’t need this data anymore for 99% of situations. You keep it because you’re legally obliged to allow users to modify closed/removed/entities for the next few years. Then, instead of keeping all past data, it’s better to keep a single event in the stream and keep it alive by that. You can append a summary event after a defined period and archive the old data.

In the end, it’s your choice; you know your business process and system better. I suggest playing with different approaches and being flexible. You don’t need to use the same strategy for each business process. I hope this article gives you a bit more nuanced explanation of your options for more state-based processes.

Cheers!

Oskar

Implementing Closing the Books pattern

No one knows who invented Event Sourcing. I overheard that Hammurabi did. Why? Because he standardised the first set of rules of accounting.

Event Sourcing works as bookkeeping; we record new entries for each business operation. Not surprisingly, accounting patterns also work well in event sourcing.

For instance, we can use the “Closing the Books” pattern to model the lifecycle process effectively. Its name comes from the accounting domain. All financial numbers are summarised and verified, and the final report is created at the end of each cycle (e.g., month and year). This data is used as a base for the next period. For accounting calculations, you don’t need to bring forward the entire history; it’s enough to summarise the crucial aspects to be carried forward, i.e. the opening balances, etc. The same pattern can be used for temporal modelling.

I explained this pattern in detail in Keep your streams short! Temporal modelling for fast reads and optimal data retention.

And the talk:

Closing the Books is the essence of Event Sourcing modelling. Thanks to that, we can keep things short and thus run our system efficiently. We’re slicing the lifetime of our process, marking the start and end of the lifecycle using events.

In this article, I assume you have read/watched, or skimmed the above resources. I’ll focus on the practical example using Marten. I know that’s not entirely to say “Go read that, before we talk”, but really: “Go read that”. It took me three months to write the article, and a bit less to prepare the talk. Of course, this article may be enough to grab the concept, but there you find more nuanced considerations.

Still, we’ll diverge from the accounting domain. I started explaining Event Sourcing like many of us: using a Bank Account example. But I stopped. It’s a business domain that most of us believe in knowing how it works, but it’s much different from what we may expect.

How much different? There’s no database transaction between multiple bank accounts while transferring money. A Bank Account is also not an entity or transactional boundary. In accounting, we add lifecycle and work on, e.g. Accounting Month.

Yes, there’s no spoon.

Let’s use a scenario close to a financial domain but not the same, plus not tied to a specific time period: cashiers in stores (also used in the original article above). We could try to model that by keeping all transactions for the particular cash register on the same stream, but if we’re building a system for bigger department stores, then that could quickly escalate. We might end up with a stream containing thousands of events. That quickly could make our processing inefficient and unmanageable.

Yet, if we talked with our domain experts, we could realise this is not how our business works. All payments are registered by specific cashiers, and the cashiers care only about what has happened in their shift. They don’t need to know the entire history of transactions, just the starting amount of cash in the drawer (called float) that was left from the previous shift.

Let’s try to reflect that in the simplified event model:

public abstract record CashierShiftEvent
{
    public record ShiftOpened(
        CashierShiftId CashierShiftId,
        string CashierId,
        decimal Float,
        DateTimeOffset StartedAt
    ): CashierShiftEvent;

    public record TransactionRegistered(
        CashierShiftId CashierShiftId,
        string TransactionId,
        decimal Amount,
        DateTimeOffset RegisteredAt
    ): CashierShiftEvent;

    public record ShiftClosed(
        CashierShiftId CashierShiftId,
        decimal DeclaredTender,
        decimal OverageAmount,
        decimal ShortageAmount,
        decimal FinalFloat,
        DateTimeOffset ClosedAt
    ): CashierShiftEvent;

    private CashierShiftEvent(){}
}

So we have ShiftOpened and ShiftClosed events marking the lifetime of the cashier shift and TransactionRegistered to register the payments. Of course, we would have more types of events possibly happening between shift opening and closing, but let’s keep it simple. Just assume that TransactionRegistered is an example of the events you may have throughout the shift lifecycle.

You could notice CashierShiftId as a class, not a primitive value. It’s an example of the strongly-typed key:

public record CashierShiftId(string CashRegisterId, int ShiftNumber)
{
    public static implicit operator string(CashierShiftId id) => id.ToString();

    public override string ToString() => $"urn:cashier_shift:{CashRegisterId}:{ShiftNumber}";
}

I’m not a big fan of the ceremony brought by strongly typed key wrappers, but it makes perfect sense here. It’s not just a wrapper for a Guid or string value. Our cashier shift id is built from two components:

cash register id,
shift number.

We’re using the Uniform Resource Name structure (URN).

urn:cashier_shift:{CashRegisterId}:{ShiftNumber}

You can use any format, but URN is the standardised way for handling ids with meaningful segments. Why reinvent the wheel if we have a standard for it? Our URN starts with the prefix and stream type. Then, we have cash register id and shift number segments.

We could add more components, e.g., date, if we’d like to have it sliced by a temporal aspect. So, having shift numbers reset every day. Most importantly, we’re defining the explicit lifetime for our cashier shift.

If you are using EventStoreDB, you’d need to structure it a bit differently: cashier_shift-{cashRegisterId}:{number}. EventStoreDB expects the first part to represent the category, so for our stream type, we need to have a dash followed by the unique stream ID part.

By default, Marten uses Guid as stream key, but it allows to use string. To do that, we need to change that in the configuration:

services.AddMarten(options =>
    {
        options.Events.StreamIdentity = StreamIdentity.AsString;

        // (...)
    });

Closing and opening shift as one operation

Cool stuff, but how to effectively perform it? Let’s start with composing closing and opening:

using CommandResult = (CashierShiftId StreamId, CashierShiftEvent[] Events);

public record CloseAndOpenCommand(
    CashierShiftId CashierShiftId,
    string CashierId,
    decimal DeclaredTender,
    DateTimeOffset Now
);

public static class CloseAndOpenShift
{
    public static (CommandResult, CommandResult) Handle(CloseAndOpenCommand command, CashierShift currentShift)
    {
        // close current shift
        var (currentShiftId, cashierId, declaredTender, now) = command;
        var closingResult = Decide(new CloseShift(currentShiftId, declaredTender, now), currentShift);

        // get new current shift state by applying the result event(s)
        currentShift = closingResult.Aggregate(currentShift, (current, @event) => current.Apply(@event));

        // open the next shift
        var openResult = Decide(new OpenShift(currentShiftId, cashierId, now), currentShift);

        // double check if it was actually opened
        var opened = openResult.OfType<ShiftOpened>().SingleOrDefault();
        if (opened == null)
            throw new InvalidOperationException("Cannot open new shift!");

        // return both results with respective stream ids
        return ((currentShiftId, closingResult), (opened.CashierShiftId, openResult));
    }
}

We’re running two commands sequentially, returning the results from both operations together with ids. Read also more in How to handle multiple commands in the same transaction.

If we’re using Marten, we can benefit from PostgreSQL transactional capabilities and Marten built-in Unit of Work.


public static class CloseAndOpenShift
{
    public static async Task<CashierShiftId> CloseAndOpenCashierShift(
        this IDocumentSession documentSession,
        CloseAndOpenCommand command,
        int version,
        CancellationToken ct
    )
    {
        var currentShift =
            await documentSession.Events.AggregateStreamAsync<CashierShift>(command.CashierShiftId, token: ct) ??
            new CashierShift.NonExisting();

        var (closingResult, openResult) = Handle(command, currentShift);

        // Append Closing result to the old stream
        if (closingResult.Events.Length > 0)
            documentSession.Events.Append(closingResult.StreamId, version, closingResult.Events.AsEnumerable());

        if (openResult.Events.Length > 0)
            documentSession.Events.StartStream<CashierShift>(openResult.StreamId, openResult.Events.AsEnumerable());

        await documentSession.SaveChangesAsync(ct);

        return openResult.StreamId;
    }
}

Thanks to the predictable structure of the id and calling the StartStream method, we’re ensuring that if the new shift has already been opened, then our operation will be rejected. We won’t have any duplicated shifts.

We can use this code in the endpoint:

app.MapPost("/api/cash-registers/{cashRegisterId}/cashier-shifts/{shiftNumber:int}/close-and-open",
    async (
        IDocumentSession documentSession,
        string cashRegisterId,
        int shiftNumber,
        CloseAndOpenShiftRequest body,
        [FromIfMatchHeader] string eTag,
        CancellationToken ct
    ) =>
    {
        var command = new CloseAndOpenCommand(
            new CashierShiftId(cashRegisterId, shiftNumber),
            body.CashierId,
            body.DeclaredTender,
            Now
        );

        var openedCashierId = await documentSession.CloseAndOpenCashierShift(command, ToExpectedVersion(eTag), ct);

        return Created(
            $"/api/cash-registers/{cashRegisterId}/cashier-shifts/{openedCashierId.ShiftNumber}",
            cashRegisterId
        );
    }
);

As you can see, we greatly benefit from the repeatability and composability of the Event Sourcing and Decider pattern and Marten’s transactional capabilities.

We’re also using optimistic concurrency with ETag to ensure we won’t face concurrency issues. Thanks to that, we will know we’re making decisions based on the expected state.

Closing and opening shifts as separate operations

The pattern of closing and opening as one operation can be suitable if our lifecycles are continuous. Yet, typically, they’re not. Occupancy of the cash registers by cashiers depends on the traffic. There may be periods when no one works on the particular cash register. How to handle that?

Let’s tackle that step by step, starting from the end, so from the closing. It’ll be a simple operation now:

app.MapPost("/api/cash-registers/{cashRegisterId}/cashier-shifts/{shiftNumber:int}/close",
    (
        IDocumentSession documentSession,
        string cashRegisterId,
        int shiftNumber,
        CloseShiftRequest body,
        [FromIfMatchHeader] string eTag,
        CancellationToken ct
    ) =>
    {
        var cashierShiftId = new CashierShiftId(cashRegisterId, shiftNumber);

        return documentSession.GetAndUpdate<CashierShift, CashierShiftEvent>(cashierShiftId, ToExpectedVersion(eTag),
            state => Decide(new CloseShift(cashierShiftId, body.DeclaredTender, Now), state), ct);
    }
);

I’m using here a simple wrapper that will load events from the stream, build the state from it, run business logic and append event result events if there are such:


public static class DocumentSessionExtensions
{
    public static Task GetAndUpdate<T, TEvent>(
        this IDocumentSession documentSession,
        string id,
        int version,
        Func<T, TEvent[]> handle,
        CancellationToken ct
    ) where T : class =>
        documentSession.Events.WriteToAggregate<T>(id, version, stream =>
        {
            var result = handle(stream.Aggregate);
            if (result.Length != 0)
                stream.AppendMany(result.Cast<object>().ToArray());
        }, ct);
}

Opening will be more complicated. We need to get the shift number and data from the last closed one (like float, the state of the cash in the drawer after the last shift). Of course, the cashier could provide the previous shift number, but that’s error-prone and potential vulnerability. It’d be better if we track that on our own. And the best way for that is to build a model that would cache the information. We could define an updated projection based on the registered events.

What should such a model contain? Potentially it could cache all needed information to open new shift, e.g.:

public record CashierShiftTracker(
    string Id, // Cash Register Id
    string LastClosedShiftNumber,
    string LastClosedShiftFloat,
    // (...) etc.
);

That looks fine, as we could start by querying it and getting the needed information from it. Yet, in my opinion, that solution is not scalable and can provide some issues while maintaining. Our process may change, and we’ll need to update this projection each time that impacts closing or opening. That can create issues with versioning this model, updating it as it goes etc.

I think it’d be better to generalise it and make it agnostic to the specifics of our process. Closing the Books pattern is repeatable:

Store summary of the closed streams needed for audit and for the next lifecycle period,
When opening, get data from the last event in the closed lifecycle and use it to start a new period and stream.

I wrote in Let’s talk about positions in event stores that Marten keeps the global sequence number. It’s a monotonic number. There may be gaps when an event is not added for some reason (e.g. a conflict or transient error). It’s unique for each event. We could use it as a reference to the closing event.

public record CashierShiftTracker(
    string Id, // Cash Register Id
    long? LastShiftClosedSequence
);

Now, let’s define the multi-stream projection that will start tracking progress when the cash register is initialised and update it with each cashier shift closing event. We’ll get it from the event metadata (Marten’s IEvent wrapper).

public class CashierShiftTrackerProjection: MultiStreamProjection<CashierShiftTracker, string>
{
    public CashierShiftTrackerProjection()
    {
        Identity<CashRegisterInitialized>(e => e.CashRegisterId);
        Identity<CashierShiftEvent.ShiftClosed>(e => e.CashierShiftId.CashRegisterId);
    }

    public CashierShiftTracker Create(CashRegisterInitialized initialized) =>
        new(initialized.CashRegisterId, null);

    public CashierShiftTracker Apply(IEvent<CashierShiftEvent.ShiftClosed> closed, CashierShiftTracker current) =>
        current with { LastShiftClosedSequence = closed.Sequence };
}

We need to register it in the Marten configuration.

services.AddMarten(options =>
    {
        options.Projections.Add<CashierShiftTrackerProjection>(ProjectionLifecycle.Async);

        // (...)
    });

The safest option for multi-stream projection is to register it as asynchronous. It will ensure that all events are run sequentially. Yet, it will introduce eventual consistency as it’ll be processed in the background. Getting the stale data is not dangerous. We’re using optimistic concurrency and have the uniqueness enforced by the predictable stream id structure. In the worst case, we may need to retry our handling.

The other option is to register it as inline. Then, the projection will be updated in the same transaction as the appended event. This will also work for our case because there cannot be parallel shift closings for the particular cash register. Still, as a general rule, we should try to avoid it, as optimistic concurrency will only ensure concurrency on a specific single stream. We have multiple streams, so they could override each other when we have a high load.

Pick your poison, depending on your use case.

Let’s discuss using this model to locate the last cashier shift. We’ll define a dedicated method for that to encapsulate the locator logic:

public static class LastCashierShiftLocator
{
    public static async Task<CashierShift> GetLastCashierShift(
        this IDocumentSession documentSession,
        string cashRegisterId
    )
    {
        var tracker = await documentSession.LoadAsync<CashierShiftTracker>(cashRegisterId);

        if (tracker is null)
            throw new InvalidOperationException("Unknown cash register!");

        var lastClosedShiftEvent = tracker.LastShiftClosedSequence.HasValue
            ? (await documentSession.Events.QueryAllRawEvents()
                .SingleAsync(e => e.Sequence == tracker.LastShiftClosedSequence.Value)).Data
            : null;

        return lastClosedShiftEvent is ShiftClosed closed
            ? new CashierShift.Closed(closed.CashierShiftId, closed.FinalFloat)
            : new CashierShift.NonExisting();
    }
}

As explained above, we’re getting the tracking information, loading the last event (if there’s such) and returning the ClosedShift or NonExisting if no shift was closed yet.

Such tracking logic is repeatable and could even be generalised to other scenarios if needed.

Having that, we can define the endpoint for opening the cashier shift as follows:

app.MapPost("/api/cash-registers/{cashRegisterId}/cashier-shifts",
    async (
        IDocumentSession documentSession,
        string cashRegisterId,
        OpenShiftRequest body,
        CancellationToken ct
    ) =>
    {
        var lastClosedShift = await documentSession.GetLastCashierShift(cashRegisterId);
        var result = Decide(new OpenShift(cashRegisterId, body.CashierId, Now), lastClosedShift);

        var opened = result.OfType<ShiftOpened>().SingleOrDefault();

        if (opened == null)
            throw new InvalidOperationException("Cannot Open Shift");

        await documentSession.Add<CashierShift, CashierShiftEvent>(opened.CashierShiftId, result, ct);

        return Created(
            $"/api/cash-registers/{cashRegisterId}/cashier-shifts/{opened.CashierShiftId.ShiftNumber}",
            cashRegisterId
        );
    }
);

We’re using a simple wrapper for appending new events similar to GetAndUpdate shown before.

public static class DocumentSessionExtensions
{
    public static Task Add<T, TEvent>(this IDocumentSession documentSession, string id, TEvent[] events,
        CancellationToken ct)
        where T : class
    {
        if (events.Length == 0)
            return Task.CompletedTask;

        documentSession.Events.StartStream<T>(id, result.Cast<object>().ToArray());
        return documentSession.SaveChangesAsync(token: ct);
    }
}

TLDR

Keeping streams short is the most important modelling practice in Event Sourcing. Closing the Books pattern is the biggest enabler for that. I hope that after this article, you’ll know how to implement that effectively using Marten.

There are other options to do it, that I described in Keep your streams short! Temporal modelling for fast reads and optimal data retention like:

opening periods asynchronously,
There may be scenarios where you have multiple open shifts (e.g. cash register in the restaurant used by multiple waiters),
you might not be able to get (or even want to have) predictable ids.

Those scenarios will require different ways to handle that, but I’m sure the techniques described above can be enough for you to implement that.

See the full code in my sample repository.

Most importantly: talk with the business experts and ask enough whys to understand the lifecycle of your business use case. Listen for the keywords like open/close/end, summary, daily, monthly, etc. For business experts, lifecycle may be so apparent that they won’t mention it straight away, but if you dig and ask enough questions, they’re typically more than happy to reveal it. Workshops like Event Storming can help you with that.

Check also the follow up article for more nuanced considerations on Should you always keep streams short in Event Sourcing?.

Cheers!

Oskar

Announcing Emmett! Take your event-driven applications back to the future!

Simple is not easy. Each person has its definition of it. For me, it means that when I look at the solution, I think: “So simple. Why didn’t I come up with it?”. That also means that it’s straightforward to understand.

In recent years, I went down the rabbit hole trying to explain to you and other readers that Event Sourcing and CQRS can be effective and simple. I wrote hundreds of articles, delivered samples in multiple languages and worked on tooling like Marten and EventStoreDB. I’m also doing workshops and consultancy.

Why do I do it? I don’t want to be a content creator or influencer. That’s not my goal. I just saw in my projects that Event Sourcing and CQRS are working and helping to build better applications. If applied in a simple way, they not only do not bring additional complexity but are cutting it.

Again, simple is not easy. But why?

Because it requires a lot of iterations and experience, it’s hard to learn from other people’s mistakes. Reading blogs is kinda like trying to do it. Thanks to that, you can avoid some, but not all. Wouldn’t it be nice to be walked through by hand sometimes?

Event Sourcing in CQRS having the guidance is easier than in other approaches. They have common and repeatable patterns like:

you read events and build the state from them,
you take this state and command and run business logic on them returning event or multiple events,
you append those events to the event stream,
rinse, repeat.

But well, you can always forget about it, or your colleague.

That’s why I decided to release Emmett. I gathered a catalogue of Node.js and TypeScript patterns in my blog and sample repository.

You can already use it by calling:

npm add @event-driven-io/emmett

It already has the first set of documentation: check here.

Why Node.js and TypeScript?

For me, it’s the environment I feel the most effective in. I like its minimalistic approach and flexibility, plus TypeScript is an excellent language with its shapeshifter capabilities. It doesn’t require much boilerplate; it cuts the cognitive load, so it aligns with my current vision of building applications.

Why Emmett?

Because I want to take your event-driven applications back to the future!.

Is it production-ready?

Kinda. What is here is already usable, but you’ll need to wait for the full production experience in all essential aspects.

What features does it have?

Essential building blocks for designing and running business and application logic like:

typings around events, commands, Deciders, Workflows, etc.
command handling wrappers for the application layer,
basic, in-memory event store implementation.

What features will it have?

We’ll see, but I’d like to have the following at some point:

building blocks for the Web Apis with Event Sourcing and CQRS,
implementation of event store using EventStoreDB, PostgreSQL, SQLite, etc.
abstractions for building read models,
building blocks for integration and running distributed processes,
built-in open telemetry,
GraphQL API for event stores,
Full stack development helpers with Next.js or HTMX,
running it serverless or on the web with SQLite,
streaming data through HTTP API (and enabling integration scenarios through it).
defining event transformations and projections with WebAssembly,
etc.

Would it be a competitor to other stores?

Probably not. For now, I’d like to have a safe playground to have fun, experiment and try out new ideas. Still, I expect what I deliver to be safe to use in production.

Why there’s no license?

Because I’m unsure how this will end, and I don’t want to expose it as an MIT license from the beginning.

So try it on your own, have fun and send me feedback!

Join our Discord to be a part of building it!

See also a follow up articles on Emmett:

Testing Event Sourcing, Emmett edition.

Read also more about Node.js and Event Sourcing to get what I want to provide out of the box:

Cheers.

Oskar

Production-Grade Event Sourcing Workshop - Modelling, DevOps, Process

Putting the career bets is not easy; I made boring and pragmatic choices for most of my career. They took me far. Yet, I always felt that something was missing and that my systems could be delivered and operated better. I tried many things; I even finished my post-graduate studies around project management.

All of that helped me make an impact, but the most significant A-ha moment for me was Event Sourcing. Why?

It helps to create the synergy between business and development. Thanks to that, we can get the fastest feedback loop, streamline development and operate with more insights and data correlation. We’re investing in the quality of the information in our system without losing much.

Event sourcing is already battle-tested by both big and small companies. They see the advantages it brings, e.g. business focus and keeping all information about the business process. It can also be a great input to analytics, providing increased diagnostics and tracing. Nowadays, that’s essential for running a system on the scale.

There are already decent materials teaching how to start the journey with Event Sourcing. Hell, I wrote over 100 of them on this blog. Yet it’s not easy to get certainty that our design will work on production. Even when we get there, it’s worth revisiting lessons learned and discovering what we can improve in preparing for the upcoming challenges.

That’s why I’m explaining the nuances, strategies, tips, and tricks of having your event-sourced system well-architected. We need more of that!

I want to invite you to my biggest initiative so far!

A three-day workshop on Production-Grade Event Sourcing: Modelling, DevOps, Process

The goal of this workshop is to strengthen all of the critical aspects of running event-sourced on production:

modelling (like keeping streams short, managing consistency and efficiently handling business logic, best practices and anti-patterns),
managing schema evolution, so events versioning,
advanced projections design and resiliency, so how to get the best from events to efficiently get read models and insights from them,
DevOps techniques (like traceability, blue-green projection rebuilt, archiving old data),
handling distributed processes and integrating with other systems,
putting event sourcing as a vital part of the whole architecture (so how to integrate it with non-event-sourced modules, defining private and public schema).

The workshop will happen May 27-29, 2024, as part of Domain Driven Design Europe. Do you know how beautiful Amsterdam is at that time of the year? You better check that!

You can sign up through the workshop page.

What will you learn

After the workshop, you’ll understand and practice techniques that will allow you to run your Event-Sourcing system on production. All the exercises will be backed by the versatile knowledge I gained through my projects and working with my clients.

I’ll share my experience building systems like Marten and EventStoreDB and working as an architect and consultant in many Event Sourcing projects.

Prerequisites

The workshop is designed for people who already have experience with Event Sourcing. Especially for those who would like to assess the production-readiness of their design or already deployed system. We won’t start from basics but assume:

experience in how Event Sourcing works (storing events, building state from them),
knowledge of how to model business logic and projections/read models,
Understand that event sourcing is not event streaming and why streaming solutions like Kafka and Pulsar should not be used as event stores.

I assume you know modelling techniques like EventStorming, C4 and modularity concepts (microservices, monolith, CQRS, etc.). And that you were already building and designing web applications. You don’t need to be an expert in that, but the more you know the easier it will be.

What if you don’t have Event Sourcing experience yet?

I’ve got you covered! Check my self-paced kit.

See the playlist with my talks:

Of course, that’s not the full experience of doing a workshop, but it can be a good starting point.

If you need more than that and want full coverage with all nuances of the private workshop, check the training page. Feel invited to contact me via [email protected]">email. I’m happy to help you via consultancy or paid mentoring!

If you’re not persuaded enough, check recommendations from my previous clients on my LinkedIn profile..

Here are some of them:

Again, feel invited to sign up for the workshop.. And [email protected]">contact me if you have any questions.

See you in Amsterdam!

Cheers.

Oskar

How TypeScript can help in modelling business workflows

TypeScript is an intriguing language. Some say that its type system, by itself, is Turing Complete. Some take it to the extreme and even implement Flappy Bird in TypeScript types. No worries, we won’t take that far today, but I’ll show you a few tricks I learned recently. Of course, I’ll use the event-driven examples, but you can use those techniques for other needs.

Let’s say that our system has two types of messages: commands and events. Commands represent the intention to run some business logic. Events are facts representing information about what has happened in our system. Essentially, events are the outcome of the command handling. (read a more nuanced take in What’s the difference between a command and an event?).

We could define command as:

export type Command<
  CommandType extends string = string,
  CommandData extends Record<string, unknown> = Record<string, unknown>,
> = Readonly<{
  type: Readonly<CommandType>;
  data: Readonly<CommandData>;
}>;

For instance, we can define the command representing the guest checkout as such:

export type CheckOut = Command<
  'CheckOut',
  {
    guestStayAccountId: string;
    now: Date;
    groupCheckoutId?: string;
  }
>;

and create the instance of it as:

const checkout: CheckOut = {
  type: 'CheckOut',
  data: {
    guestStayAccountId: uuid(),
    now: Date(),
    groupCheckoutId: uuid(),
  },
};

So, the command definition tells that the command has type and data. That’s represented by CommandType and CommandData in the Command type definition. The weirdly looking Record<string, unknown> just tells that command data can be any object with properties. We’re also marking it as read-only, as it’s a Data Transfer Object that carries data and, once created, should not be changed.

Accordingly, we could define the Event type as:

export type Event<
  EventType extends string = string,
  EventData extends Record<string, unknown> = Record<string, unknown>,
> = Readonly<{
  type: EventType;
  data: Readonly<EventData>;
}>;

with example:

export type GuestCheckedOut = Event<
  'GuestCheckedOut',
  {
    guestStayAccountId: string;
    checkedOutAt: Date;
    groupCheckoutId?: string;
  }
>;

const checkedOut: GuestCheckedOut = {
  type: 'GuestCheckedOut',
  data: {
    guestStayAccountId: uuid(),
    checkedOutAt: Date(),
    groupCheckoutId: uuid(),
  },
};

We could define the type for function making business logic decisions as :

export type Decide<
  State extends Record<string, unknown>,
  C extends Command,
  E extends Event,
> = (command: C, state: State) => E[];

We take a command and a state, returning one or more events (read more in Straightforward Event Sourcing with TypeScript and NodeJS). If we define all events and commands for Guest Stay, we can join those types with union.

export type GuestStayAccount =
  | { status: 'NotExisting' }
  | {
      status: 'Opened';
      balance: number;
    }
  | { status: 'CheckedOut' };

export type GuestStayCommand =
  | CheckIn
  | RecordCharge
  | RecordPayment
  | CheckOut;

export type GuestStayAccountEvent =
  | GuestCheckedIn
  | ChargeRecorded
  | PaymentRecorded
  | GuestCheckedOut
  | GuestCheckoutFailed;

This is a nice definition of the Guest Stay API. We can then encapsulate the business logic in:

export const decide: Decide<
  GuestStayAccount,
  GuestStayCommand,
  GuestStayAccountEvent
> = (
  { type, data: command }: GuestStayCommand,
  state: GuestStayAccount,
): GuestStayAccountEvent[] => {
  const { guestStayAccountId, now } = command;

  switch (type) {
    case 'CheckIn': {
      if (state.status !== 'NotExisting')
        throw Error('Guest is already checked-in!');

      return [
        {
          type: 'GuestCheckedIn',
          data: {
            guestStayAccountId,
            checkedInAt: now,
          },
        },
      ];
    }
    case 'RecordCharge': {
      if (state.status !== 'Opened')
        throw Error('Guest account is already checked out!');

      return [
        {
          type: 'ChargeRecorded',
          data: {
            guestStayAccountId,
            amount: command.amount,
            recordedAt: now,
          },
        },
      ];
    }
    case 'RecordPayment': {
      if (state.status !== 'Opened')
        throw Error('Guest account is already checked out!');

      return [
        {
          type: 'PaymentRecorded',
          data: {
            guestStayAccountId,
            amount: command.amount,
            recordedAt: now,
          },
        },
      ];
    }
    case 'CheckOut': {
      if (state.status !== 'Opened')
        return [
          {
            type: 'GuestCheckoutFailed',
            data: {
              guestStayAccountId,
              groupCheckoutId: command.groupCheckoutId,
              reason: 'NotOpened',
              failedAt: now,
            },
          },
        ];

      const isSettled = state.balance === 0;

      if (!isSettled)
        return [
          {
            type: 'GuestCheckoutFailed',
            data: {
              guestStayAccountId,
              groupCheckoutId: command.groupCheckoutId,
              reason: 'BalanceNotSettled',
              failedAt: now,
            },
          },
        ];

      return [
        {
          type: 'GuestCheckedOut',
          data: {
            guestStayAccountId,
            groupCheckoutId: command.groupCheckoutId,
            checkedOutAt: now,
          },
        },
      ];
    }

    default: {
      const _notExistingCommandType: never = type;
      throw new Error(`Unknown command type`);
    }
  }
};

Sweet, we have the pattern for handling the business logic in Event Sourcing. Thanks to its predictable nature, it’s quite straightforward.

Yet, things are getting more complex if we tackle more advanced business processes. Let’s try to describe the Group Checkout process that I modelled in the webinar:

In a nutshell, we want to speed up the people’s checkout by allowing the clerk to select multiple guest stays and run checkout as a batched, asynchronous process. Read also more in Event-driven distributed processes by example

Let’s try to reflect the process in code as described by Yves Reynhout in The Workflow Pattern.

The group checkout process can have the following inputs:

export type GroupCheckoutInput =
  | InitiateGroupCheckout
  | GuestCheckedOut
  | GuestCheckoutFailed
  | TimeoutGroupCheckout;

export type InitiateGroupCheckout = Command<
  'InitiateGroupCheckout',
  {
    groupCheckoutId: string;
    clerkId: string;
    guestStayAccountIds: string[];
    now: Date;
  }
>;

export type TimeoutGroupCheckout = Command<
  'TimeoutGroupCheckout',
  {
    groupCheckoutId: string;
    startedAt: Date;
    timeOutAt: Date;
  }
>;

export type GuestCheckedOut = Event<
  'GuestCheckedOut',
  {
    guestStayAccountId: string;
    checkedOutAt: Date;
    groupCheckoutId?: string;
  }
>;

export type GuestCheckoutFailed = Event<
  'GuestCheckoutFailed',
  {
    guestStayAccountId: string;
    reason: 'NotOpened' | 'BalanceNotSettled';
    failedAt: Date;
    groupCheckoutId?: string;
  }
>;

The clerk initiates the group checkout process after selecting a list of guests stays to checkout. Then, the next steps are triggered by guest checkout completion or failure. Once we get all events back, we can either complete the group checkout (if all guest checkouts succeeded) or fail it. We can also get the request to time out our workflow after the maximum processing time is reached.

The state of our workflow can look like this:

export type GroupCheckout =
  | { status: 'NotExisting' }
  | {
      status: 'Pending';
      guestStayAccountIds: Map<string, GuestStayStatus>;
    }
  | { status: 'Finished' };

export const getInitialState = (): GroupCheckout => {
  return {
    status: 'NotExisting',
  };
};

export enum GuestStayStatus {
  Pending = 'Pending',
  Completed = 'Completed',
  Failed = 'Failed',
}

It’s either not existing, pending or finished.

The output of our process can be defined as:

export type GroupCheckoutOutput =
  | GroupCheckoutInitiated
  | CheckOut
  | GroupCheckoutCompleted
  | GroupCheckoutFailed
  | GroupCheckoutTimedOut;

export type GroupCheckoutInitiated = Event<
  'GroupCheckoutInitiated',
  {
    groupCheckoutId: string;
    clerkId: string;
    guestStayAccountIds: string[];
    initiatedAt: Date;
  }
>;

export type GroupCheckoutCompleted = Event<
  'GroupCheckoutCompleted',
  {
    groupCheckoutId: string;
    completedCheckouts: string[];
    completedAt: Date;
  }
>;

export type GroupCheckoutFailed = Event<
  'GroupCheckoutFailed',
  {
    groupCheckoutId: string;
    completedCheckouts: string[];
    failedCheckouts: string[];
    failedAt: Date;
  }
>;

export type GroupCheckoutTimedOut = Event<
  'GroupCheckoutTimedOut',
  {
    groupCheckoutId: string;
    incompleteCheckouts: string[];
    completedCheckouts: string[];
    failedCheckouts: string[];
    timedOutAt: Date;
  }
>;

export type CheckOut = Command<
  'CheckOut',
  {
    guestStayAccountId: string;
    now: Date;
    groupCheckoutId?: string;
  }
>;

As mentioned, once the clerk initiated the Group checkout, we spun out the set of CheckOut commands, and then we published the final events once the work was done.

Let’s now try to generalise the workflow definition using the TypeScript type system.

Workflow is defined by the set of inputs, outputs and the state. Inputs and outputs can be either commands or events. The heuristic is that inputs are mostly events, and outputs are mostly commands.

Once the workflow is triggered by an input message, it needs to decide what to do next. The result is reflected by the list of output messages. Besides the payload (as Yves explained in his article) they can be enriched by the expected integration context (e.g. whether we’re sending commands, publishing events, scheduling delayed messages, etc.).

It can look as follows:

export type WorkflowOutput<TOutput extends Command | Event> =
  | { kind: 'Reply'; message: TOutput }
  | { kind: 'Send'; message: TOutput }
  | { kind: 'Publish'; message: TOutput }
  | {
      kind: 'Schedule';
      message: TOutput;
      when: { afterInMs: number } | { at: Date };
    }
  | { kind: 'Complete' }
  | { kind: 'Accept' }
  | { kind: 'Ignore'; reason: string }
  | { kind: 'Error'; reason: string };

I extended the original set of semantics with Ignore, Accept and Error. Why? Read more in No, it can never happen!.

Now, what if we’d like to be explicit and say that we can send only commands and publish only events? We would like the TypeScript compiler to tell us if we try to use the event for command and publish for the event. Our current type definition doesn’t allow us to do that; why? Because of the Structural Typing in TypeScript. Our command and event type definitions have exactly the same structure. Let’s get some flavour into it!

In TypeScript we have so called Flavouring and Branding techniques. Wojciech Baczyński explained that in detail in his article, but TLDR is that we can add a field that’ll differentiate our type. Branding requires providing this optional value; flavouring makes it optional. This can be reflected as:

export type Brand<K, T> = K & { readonly __brand: T };
export type Flavour<K, T> = K & { readonly __brand?: T };

Using this technique, we can update our Event and Command type definitions as:

export type Event<
  EventType extends string = string,
  EventData extends Record<string, unknown> = Record<string, unknown>,
> = Flavour<
  Readonly<{
    type: EventType;
    data: Readonly<EventData>;
  }>,
  'Event'
>;

export type Command<
  CommandType extends string = string,
  CommandData extends Record<string, unknown> = Record<string, unknown>,
> = Flavour<
  Readonly<{
    type: CommandType;
    data: Readonly<CommandData>;
  }>,
  'Command'
>;

This looks a bit weird, but we’re adding an additional field to tell the typescript compiler “Hey, I’m Event!” or “Hey, I’m Command!”.

Now, having that, we can tell TypeScript that having a set of inputs, I’d like to take the subset of it. This subset should contain only event types:

export type WorkflowEvent<Output extends Command | Event> = Extract<
  Output,
  { __brand?: 'Event' }
>;

or just command types:

export type WorkflowCommand<Output extends Command | Event> = Extract<
  Output,
  { __brand?: 'Command' }
>;

We’re using the built-in Extract utility type. It constructs a type by extracting all union members that are assignable to the union type provided as the type second parameter (so all with a specific brand: Event or Command).

Thanks to that, this won’t compile:

const event: WorkflowEvent<GroupCheckoutOutput> = {
  type: 'CheckOut',
  data: {
    guestStayAccountId: uuid(),
    now: Date(),
    groupCheckoutId: uuid(),
  },
};

but this will:

const event: WorkflowCommand<GroupCheckoutOutput> = {
  type: 'CheckOut',
  data: {
    guestStayAccountId: uuid(),
    now: Date(),
    groupCheckoutId: uuid(),
  },
};

We can change our workflow output definition to be explicit and say that we can send only commands and publish only events.

export type WorkflowOutput<TOutput extends Command | Event> =
  | { kind: 'Reply'; message: TOutput }
  | { kind: 'Send'; message: WorkflowCommand<TOutput> }
  | { kind: 'Publish'; message: WorkflowEvent<TOutput> }
  | {
      kind: 'Schedule';
      message: TOutput;
      when: { afterInMs: number } | { at: Date };
    }
  | { kind: 'Complete' }
  | { kind: 'Accept' }
  | { kind: 'Ignore'; reason: string }
  | { kind: 'Error'; reason: string };

And the whole Worfklow definition as:

export type Workflow<
  Input extends Event | Command,
  State,
  Output extends Event | Command,
> = {
  decide: (command: Input, state: State) => WorkflowOutput<Output>[];
  evolve: (currentState: State, event: WorkflowEvent<Output>) => State;
  getInitialState: () => State;
};

Besides what we have discussed so far, I added the requirement to provide the initial state of the workflow through the getInitialState method.

I also introduced the evolve function. We want to store all input and output events to have the full tracing. I’ll also use them to build the state of the workflow from them. Read more in How to get the current entity state from events?. That’s optional, but you should already know, that I like Event Sourcing. It can look as follows:

export const evolve = (
  state: GroupCheckout,
  {
    type,
    data: event,
  }: WorkflowEvent<GroupCheckoutInput | GroupCheckoutOutput>,
): GroupCheckout => {
  switch (type) {
    case 'GroupCheckoutInitiated': {
      if (state.status !== 'NotExisting') return state;

      return {
        status: 'Pending',
        guestStayAccountIds: event.guestStayAccountIds.reduce(
          (map, id) => map.set(id, GuestStayStatus.Pending),
          Map<string, GuestStayStatus>(),
        ),
      };
    }
    case 'GuestCheckedOut':
    case 'GuestCheckoutFailed': {
      if (state.status !== 'Pending') return state;

      return {
        ...state,
        guestStayAccountIds: state.guestStayAccountIds.set(
          event.guestStayAccountId,
          type === 'GuestCheckedOut'
            ? GuestStayStatus.Completed
            : GuestStayStatus.Failed,
        ),
      };
    }
    case 'GroupCheckoutCompleted':
    case 'GroupCheckoutFailed':
    case 'GroupCheckoutTimedOut': {
      if (state.status !== 'Pending') return state;

      return {
        status: 'Finished',
      };
    }
    default: {
      const _notExistingEventType: never = type;
      return state;
    }
  }
};

We’re having fun with the TypeScript system, saying that to evolve the state, we’ll just use events from workflow inputs and outputs. We can do that by simply calling:

const events: (GroupCheckoutInput | GroupCheckoutOutput)[]= 
  eventStore.readEvents(workflowId);
const initialState = getInitialState();
const state = events.reduce<GroupCheckout>(evolve, initialState);

Such state can be used in the decision making as:

export const decide = (
  input: GroupCheckoutInput,
  state: GroupCheckout,
): WorkflowOutput<GroupCheckoutOutput>[] => {
  const { type } = input;

  switch (type) {
    case 'InitiateGroupCheckout':
      return initiate(input, state);
    case 'GuestCheckedOut':
    case 'GuestCheckoutFailed':
      return tryComplete(input, state);
    case 'TimeoutGroupCheckout': {
      return timeOut(input, state);
    }
    default: {
      const _notExistingEventType: never = type;
      return [error('UnknownInputType')];
    }
  }
};

Togethere with handling methods:

const initiate = (
  { data: command }: InitiateGroupCheckout,
  state: GroupCheckout,
): WorkflowOutput<GroupCheckoutOutput>[] => {
  if (state.status !== 'NotExisting')
    return [ignore(IgnoredReason.GroupCheckoutAlreadyInitiated)];

  const checkoutGuestStays = command.guestStayAccountIds.map((id) => {
    return send<GroupCheckoutOutput>({
      type: 'CheckOut',
      data: {
        guestStayAccountId: id,
        now: command.now,
        groupCheckoutId: command.groupCheckoutId,
      },
    });
  });

  return [
    ...checkoutGuestStays,
    publish<GroupCheckoutOutput>({
      type: 'GroupCheckoutInitiated',
      data: {
        groupCheckoutId: command.groupCheckoutId,
        guestStayAccountIds: command.guestStayAccountIds,
        initiatedAt: command.now,
        clerkId: command.clerkId,
      },
    }),
  ];
};

const tryComplete = (
  { type, data: event }: GuestCheckedOut | GuestCheckoutFailed,
  state: GroupCheckout,
): WorkflowOutput<GroupCheckoutOutput>[] => {
  if (!event.groupCheckoutId)
    return [ignore(IgnoredReason.GuestCheckoutWasNotPartOfGroupCheckout)];

  if (state.status === 'NotExisting')
    return [ignore(IgnoredReason.GroupCheckoutDoesNotExist)];

  if (state.status === 'Finished')
    return [ignore(IgnoredReason.GuestCheckoutAlreadyFinished)];

  const { guestStayAccountId, groupCheckoutId } = event;

  const guestCheckoutStatus = state.guestStayAccountIds.get(guestStayAccountId);

  if (isAlreadyClosed(guestCheckoutStatus))
    return [ignore(IgnoredReason.GuestCheckoutAlreadyFinished)];

  const guestStayAccountIds = state.guestStayAccountIds.set(
    guestStayAccountId,
    type === 'GuestCheckedOut'
      ? GuestStayStatus.Completed
      : GuestStayStatus.Failed,
  );

  const now = type === 'GuestCheckedOut' ? event.checkedOutAt : event.failedAt;

  return areAnyOngoingCheckouts(guestStayAccountIds)
    ? [accept()]
    : [
        publish(finished(groupCheckoutId, state.guestStayAccountIds, now)),
        complete(),
      ];
};

const timeOut = (
  { data: event }: TimeoutGroupCheckout,
  state: GroupCheckout,
): WorkflowOutput<GroupCheckoutOutput>[] => {
  if (state.status === 'NotExisting')
    return [ignore(IgnoredReason.GroupCheckoutDoesNotExist)];

  if (state.status === 'Finished')
    return [ignore(IgnoredReason.GroupCheckoutAlreadyFinished)];

  const { groupCheckoutId, timeOutAt } = event;
  const { guestStayAccountIds } = state;

  return [
    publish<GroupCheckoutOutput>({
      type: 'GroupCheckoutTimedOut',
      data: {
        groupCheckoutId,
        incompleteCheckouts: checkoutsWith(
          guestStayAccountIds,
          GuestStayStatus.Pending,
        ),
        completedCheckouts: checkoutsWith(
          guestStayAccountIds,
          GuestStayStatus.Completed,
        ),
        failedCheckouts: checkoutsWith(
          guestStayAccountIds,
          GuestStayStatus.Failed,
        ),
        timedOutAt: timeOutAt,
      },
    }),
    complete(),
  ];
};

and helpers to make processing more explicit:

export const isAlreadyClosed = (status: GuestStayStatus | undefined) =>
  status === GuestStayStatus.Completed || status === GuestStayStatus.Failed;

const areAnyOngoingCheckouts = (
  guestStayAccounts: Map<string, GuestStayStatus>,
) => guestStayAccounts.some((status) => !isAlreadyClosed(status));

const areAllCompleted = (guestStayAccounts: Map<string, GuestStayStatus>) =>
  guestStayAccounts.some((status) => status === GuestStayStatus.Completed);

const checkoutsWith = (
  guestStayAccounts: Map<string, GuestStayStatus>,
  status: GuestStayStatus,
): string[] =>
  Array.from(guestStayAccounts.filter((s) => s === status).values());

const finished = (
  groupCheckoutId: string,
  guestStayAccounts: Map<string, GuestStayStatus>,
  now: Date,
): GroupCheckoutCompleted | GroupCheckoutFailed => {
  return areAllCompleted(guestStayAccounts)
    ? {
        type: 'GroupCheckoutCompleted',
        data: {
          groupCheckoutId,
          completedCheckouts: Array.from(guestStayAccounts.values()),
          completedAt: now,
        },
      }
    : {
        type: 'GroupCheckoutFailed',
        data: {
          groupCheckoutId,
          completedCheckouts: checkoutsWith(
            guestStayAccounts,
            GuestStayStatus.Completed,
          ),
          failedCheckouts: checkoutsWith(
            guestStayAccounts,
            GuestStayStatus.Failed,
          ),
          failedAt: now,
        },
      };
};

export enum IgnoredReason {
  GroupCheckoutAlreadyInitiated = 'GroupCheckoutAlreadyInitiated',
  GuestCheckoutWasNotPartOfGroupCheckout = 'GuestCheckoutWasNotPartOfGroupCheckout',
  GuestCheckoutAlreadyFinished = 'GuestCheckoutAlreadyFinished',
  GroupCheckoutAlreadyFinished = 'GroupCheckoutAlreadyFinished',
  GroupCheckoutDoesNotExist = 'GroupCheckoutDoesNotExist',
}

export type ErrorReason = 'UnknownInputType';

Thanks to the WorkflowOutput definition, we won’t be able to produce wrong values.

The final definition of our workflow will look as:

export const GroupCheckoutWorkflow: Workflow<
  GroupCheckoutInput,
  GroupCheckout,
  GroupCheckoutOutput
> = {
  decide,
  evolve,
  getInitialState,
};

And hey, we modelled the whole workflow, keeping it expressive and type-safe. We also had some fun with the TypeScript type system and reached the end of this article!

I encourage you to play more with TypeScript or similar languages; with advanced types of systems, you can safely model both domain and infrastructure code! Speaking about the code, see the full sample in my repository.

Also, stay tuned. In one of the following articles, I’ll show you how to plug that into the actual infrastructure!

Cheers!

Oskar

Event-Driven by Oskar Dudycz

Interactive Rubber Ducking with GenAI

Interactive Rubber Ducking example

My idea

Q1: Cache library choice — lru-cache, keyv, or custom interface?

Q2: Where should the cache live in Pongo’s hierarchy?

Q3: Inheritance/override behavior for cascading cache config?

Q4: Cache interface shape — async, batch ops, scoping?

Q5: Cache key strategy

Q6: Cache invalidation and consistency with optimistic concurrency

Q7: Should skipCache be available on read methods too?

Q8: Batch operations on handle — signature and handler shape

Q9: Defaults — max entries, TTL, enabled by default?

Q10: TTL or no TTL as staleness backstop?

Q11: Where does cache integration hook into existing code?

Q12: Cache interaction with transactions

Q13: Cascade priority — does session override collection?

Q14: Cache provider error handling

Q15: Cache warming, events/hooks, and delete operations

Interactive Rubber Ducking result

Pongo 2nd Level Cache — Specification

Overview

Cache Interface

PongoCacheProvider<T>

Cache key strategy

Event hooks

Configuration

CacheConfig

Cascading configuration

Passing a cache instance

Defaults

Integration points

Where: pongoCollection factory function

Read operations

Write operations

Optimistic concurrency

handle method

Transactions

Error handling

Future considerations (out of scope for v1)

Implementation approach

The End of Coding? Wrong Question

Parse, Don't Guess

The Shift

Compatibility FTW

Right decisions stack

How I cheated on transactions. Or how to make tradeoffs based on my Cloudflare D1 support

Dumbo

Cloudflare D1

Repeatable reads, batches, etc.

On rebuilding read models, Dead-Letter Queues and Why Letting Go is Sometimes the Answer

Everyone has a plan until they get punched in the mouth

The Rabbit Hole of “Fixes”

Attempt 1: Wait for In-Flight Transactions

Attempt 2: Use Transaction IDs as Boundaries

Attempt 3: Lock Appends During Transition

TLDR on Attempts

Stop Fighting, Start Tracking

The Final Flow

During rebuild

When rebuild catches up

Draining the skipped events

The Dead Letter Queue Pattern

The Broader Lesson

Rebuilding Event-Driven Read Models in a safe and resilient way

Add new vs update existing.

In place update, blue/green rebuild and backfilling data

Concurrency issues while (re)building read models

Distributed Locking and PostgreSQL advisory locks

Why advisory locks alone aren’t enough

The hybrid-locking approach

The scenarios walkthrough

TLDR

Multi-tenancy and dynamic messaging workload distribution

Distributing the workload dynamically between processors

What options do we have for such a multi-tenant setup?

Checkpointing the message processing

Consumers, projectors, reactors and all that messaging jazz in Emmett

Why Split Consumers and Processors?

How Consumers Work

Q7: Should `skipCache` be available on read methods too?

Q8: Batch operations on `handle` — signature and handler shape

`PongoCacheProvider<T>`

`CacheConfig`

Where: `pongoCollection` factory function

`handle` method