Ryan Martin's Blog

Understanding Recursive Ascent Parsers

2026-04-04T12:25:25Z

Recursive ascent is an obscure parsing technique I researched about a few years ago. It’s an interesting approach to bottom-up parsing, the parsing method used by parser generators like Bison or ANTLR. If you’re familiar with recursive descent parsing, recursive ascent parsing is the bottom-up version of it.

Unfortunately, there isn’t a lot of material about it on the internet. When I first started learning about it, there was only one blog post about it.^{[1]
[1]} There were also some academic papers about it, but these are very theory-heavy and difficult to read (at least for me).^{[2]
[2]}

I’m writing this post to share what I’ve learned about recursive ascent parsing; how it works and how to implement it. I’ll assume you have some familiarity with recursive descent parsing, as it’s usually the first parsing technique people learn. If not, I recommend reading up on it first.^{[3]
[3]}

The first half of this post will cover the theory behind bottom-up parsing, and the second half will delve into the code implementation of a recursive ascent parser.

Disclaimer: I’m not an expert. I didn’t take any classes on compiler theory, and I haven’t read the Dragon Book. Please reach out If you notice any mistakes in this post, especially in the first half!

Bottom-Up vs Top-Down Parsing

Parsing techniques can be roughly grouped into 2 types: top-down and bottom-up. Top-down parsers start from the highest level of the grammar rules, while bottom-up parsers start from the lowest level. Generally, top-down parsing algorithms are easier to understand and implement, but are less powerful and accept a smaller set of grammars than bottom-up algorithms.

A nice way to visualise the difference between top-down and bottom-up approaches, which I just found recently, is to correspond top-down parsing with the Polish notation, and bottom-up parsing with the reversed Polish notation.^{[4]
[4]} In Polish notation, the operators come before the operands. In reversed Polish notation, it’s the opposite; the operands come before the operators.

Let’s use this math expression 1 + 2 * 3 as an example. We can also represent this expression as a tree:

A top-down parser builds the parse tree starting from the root node and ending in the leaf nodes. In computer science lingo, this is also called a pre-order traversal. If you write down the nodes in the order it is traversed, you’ll end up with + 1 * 2 3, the Polish notation of the previous expression.

On the other hand, a bottom-up parser builds the parse tree starting from the leaf nodes up to the root node, aka, post-order traversal. The order the nodes are traversed will form the reversed Polish notation of the original expression: 2 3 * 1 +.

You’re probably already familiar with the top-down approach, as that is how recursive descent parsers work. Personally, I find it difficult to reverse this logic and understand how a bottom-up approach works. Unlike top-down algorithms, which determine what to do next based on the current grammar rule, how does a bottom-up algorithm know what rule it is in and what it should do next?

LR Algorithm

LR is a bottom-up parsing algorithm used in most parser generators. More accurately, it is a family of parsing algorithms that follow a similar approach. Recursive ascent parsers also use this algorithm. There are many variants, such as LR(k), SLR, LALR, etc.,^{[5]
[5]} but all of them are based on the same two fundamental actions: shift and reduce.

Shift means to advance or “shift” to the next token from the input stream. If the shifted tokens match one of the rules of the grammar, they will be converted or “reduced” into that rule. An LR parser will continue shifting and reducing until the entire input is consumed and transformed into a single parse tree. This concept may sound abstract, so let’s use an example to illustrate it.

We’ll use this grammar:

expr -> expr '+' term
expr -> expr '-' term
expr -> term
term -> INTEGER

This grammar defines a language that consists only of additions and subtractions of integers. The table below shows the actions an LR parser takes when parsing the input 1 + 2 - 3 according to the grammar above:

Parse stack	Unparsed	Action
empty	1 + 2 - 3	Shift 1
1	+ 2 - 3	Reduce term -> INTEGER
term	+ 2 - 3	Reduce expr -> term
expr	+ 2 - 3	Shift +
expr +	2 - 3	Shift 2 (because there are no rules that match expr ‘+’)
expr + 2	- 3	Reduce term -> INTEGER
expr + term	- 3	Reduce expr -> expr ‘+’ term
expr	- 3	Shift -
expr -	3	Shift 3
expr - 3		Reduce term -> INTEGER
expr - term		Reduce expr -> expr - term
expr		Accept

Here, I used the term “parse stack” for the shifted inputs. That’s because most LR parsers use a stack to keep track of the shifted tokens. A shift is a push to the stack, and a reduce pops tokens that match a rule and pushes the corresponding rule to the stack.

How does the parser know which rule it should reduce to? Most LR parsers construct a parse table, which is a table of parser instructions based on the grammar. Entries in the table tell the parser whether to shift or reduce based on the current token in the input stream. The entire combination of actions a parser can take based on any input fits in this one table. This is possible because there are only a finite number of possible states an LR parser can assume for a given grammar. These are called the LR states.

Because the number of states is finite, an LR parser is essentially just a finite state machine (FSM) modelled after the LR states. The parse table contains the state transitions. The parse stack is used to keep track of the current state. In fact, parser generators are just programs that convert a grammar into a table-based state machine.

What about recursive ascent parsers then? A recursive ascent parser is just the “functional” version of a traditional LR table-based state machine. Instead of rows in a table, state transitions are encoded as functions. A function call is a shift action, and a reduce action happens when the function returns. The parse stack is the function call stack!!

I had an Aha! moment when I first realised this about recursive ascent parsers. This might not make sense now, but once we get to the code implementation, it will hopefully make this more clear.

Building the State Machine

Before getting into the code, first, we’ll need to understand how to generate a parse table. This is a process that is automated by parser generators, but in recursive ascent, we have to do this manually. The first step is to compute all the possible LR states for a given grammar, in other words, build a finite state machine. Again, we’ll use an example grammar to illustrate this.

We’ll modify the grammar from the previous section to support multiplications and drop subtractions:^{[6]
[6]}

S -> expr eof
expr -> expr '+' term
expr -> term
term -> term '*' factor
term -> factor
factor -> INTEGER

Note the extra rule S (start), which will be used for a reduction when the parser has accepted the whole input. eof indicates the end of the input stream.

Next, we’ll need to construct the LR(0) items, which are grammar rules with a dot (•) to mark the current position of the parser.^{[7]
[7]} E.g. the rule S -> expr eof contains one LR(0) item: S -> • expr eof. Everything to the left of the dot has already been parsed or shifted.^{[8]
[8]} For brevity’s sake, I’ll shorten “LR(0) item” to just “item” going forward.

For each item, we then need to compute its closure. The closure of an item is the set of all possible items that can be derived from applying the appropriate grammar rules to it. Here’s how it works:

Start with a set of items. For our grammar, this will be { S -> • expr eof }.
For each item, if the symbol on the right of the dot is a non-terminal, expand its rules based on the grammar and turn the expanded rules into items by adding a dot. From our first item, we expand expr into expr -> • expr '+' term and expr -> • term.
Add the expanded items to the original set. Our set should now look like { S -> • expr eof, expr -> • expr '+' term, expr -> • term }.
Repeat steps 2 and 3 until no new items can be added to the set.

A non-terminal is any symbol that represents a grammar rule, e.g. expr, term, or factor. The opposite of this is a terminal, which is an actual value or token that is produced by the lexer, e.g. '+', '-', and INTEGER. If we apply closure to the first item in the grammar S -> • expr eof, we’ll end up with the first LR state:

# State 0
S -> • expr eof
expr -> • expr '+' term
expr -> • term
term -> • term '*' factor
term -> • factor
factor -> • INTEGER

We can then transition from this state into the other states by accepting different values and advancing the dot. The values that the parser can accept are the terminals and non-terminals on the right side of the dot. So, for state 0, the values it can accept are expr, term, factor, and INTEGER. Let’s say the parser receives an expr. This will become our second state:

# State 1 (State 0 + expr)
S -> expr • eof (Accept)
expr -> expr • '+' term

When the dot reaches the end of the starting rule, it is called the accept state. It means that the input has been parsed fully and the parsing is complete. If we had instead received a term in the first state, then we’d end up with this state:

# State 2 (State 0 + term)
expr -> term •
term -> term • '*' factor

And we just repeat this process until all dots are at the end of each rule. There is an exception to this though, which I’ll get to in a bit. Anyways, here are some more states:

# State 3 (State 0 + factor)
term -> factor •

# State 4 (State 0 + INTEGER)
factor -> INTEGER •

# State 5 (State 1 + '+')
expr -> expr '+' • term
term -> • term '*' factor
term -> • factor
factor -> • INTEGER

# State 6 (State 2 + '*')
term -> term '*' • factor
factor -> • INTEGER

# State 7 (State 5 + term)
expr -> expr '+' term •
term -> term • '*' factor

Now, it might make sense to do this for the next state:

# State 8 (State 1 + factor)
term -> factor •

Notice that this is the same state as state 3. If we encounter a state that we have seen before, we don’t have to create a new state. Instead, we’ll just jump or goto the state that we’ve already found before. This will make more sense when we build the actual parse table. Now, let’s finish the last state.^{[9]
[9]}

# State 8 (State 6 + factor)
term -> term '*' factor •

And we’re done! Here’s a diagram to help you visualise the state transitions:

The Action and Goto Table

Now that we have the state machine, we can translate it into the parse table. A transition between states represents a shift action. Items with a dot at the end represent reduce actions. The parse table consists of two parts, the action table and the goto table. The action table tells the parser whether to shift or to reduce based on a lookahead terminal. The goto table tells the parser which state to go to, given a non-terminal. Here’s what the action and goto table looks like for our grammar:

	Action				Goto
State	+	*	INTEGER	eof	expr	term	factor
State 0			s4		gt1	gt2	gt3
State 1	s5			Accept
State 2	r3	s6	r3	r3
State 3	r5	r5	r5	r5
State 4	r6	r6	r6	r6
State 5			s4			gt7	gt3
State 6			s4				gt8
State 7	r2	s6	r2	r2
State 8	r4	r4	r4	r4

Each row represents a state in the state machine, each column in the action table represents a terminal, and each column in the goto table represents a non-terminal. sN means “shift and go to state N”, rN means “reduce using rule N”, and gtN means “go to state N”. Unlike shift, goto doesn’t take in a token from the input stream and just transitions to the next state.

Here are the grammar rules again for reference, with the rule numbers added for reductions:

1. S -> expr eof
2. expr -> expr '+' term
3. expr -> term
4. term -> term '*' factor
5. term -> factor
6. factor -> INTEGER

The parse table is usually much larger than the grammar, and languages with a complex grammar will produce huge tables which are almost impossible to compute by hand. This is why people create parser generators. Based on how the states and parse table are generated, the resulting parser can be called either a simple LR (SLR), look-ahead LR (LALR), or a canonical LR parser (LR(k)). I’m not going to go into detail about these algorithms here, as it can lead to a rabbit hole. Just know that we have a working LR(0) parse table now.^{[10]
[10]}

The next step of the LR algorithm is to run the actual parser using a real input. Traditional LR parsers will run a loop and parse a token stream following the rules of the parse table. In our case, we’ll implement this mechanism as mutually recursive functions.

Writing The Parser

At this point, we’re ready to build the actual parser. I’ll be writing the code in Python because it’s easy to read. To make sure our parser is correct, we’ll also implement an interpreter for the expressions we parsed. First, let’s start with the program’s entry point:

recursive-ascent.py

import sys

def main():
    if len(sys.argv) != 2:
        print("usage: {} [STRING]".format(sys.argv[0]))
        return
    tokens = lex(sys.argv[1])
    ast = parse(tokens)
    result = evaluate(ast)
    print(result)

if __name__ == "__main__":
    main()

This program will receive a string as an argument and print out the evaluated string. Don’t worry about the unimplemented functions for now. We’ll get to them soon.

A Simple Lexer

Since we don’t have a complex grammar, our lexer can just scan tokens separated by whitespace. This works, but it also means that we need to separate the terminals in our input string with spaces. A better lexer knows whether a token can be scanned regardless of whitespaces.^{[11]
[11]}

recursive-ascent.py

def lex(string: str):
    return string.split()

Representing the Abstract Syntax Tree (AST)

We’ll also need a type to represent the AST that will be produced by the parser:

recursive-ascent.py

class AST:
    def __init__(self, left: "AST | None", right: "AST | None", value: str, kind: str):
        self.left = left
        self.right = right
        self.value = value
        self.kind = kind

If you’ve noticed, the AST class above looks like a binary tree. That’s because all the expressions we have are binary operations, which can be represented as a binary tree. The final AST should look like this diagram from an earlier section. For more complex grammars, you might need to use a more complex data structure for the AST.

Converting the Parse Table into Functions

Our parse function will start from the first state. If you recall the previous parse table, there are 4 possible actions the parser can take:

	Action				Goto
State	+	*	INTEGER	eof	expr	term	factor
State 0			s4		gt1	gt2	gt3

We can translate this to code like so:

recursive-ascent.py

def state0(tokens: list[str]):
    token = peek(tokens)
    if token.isdigit():
        # shift token, move dot forwards
        tokens = tokens[1:]
        # go to state 4
        ast = state4(token)
    else:
        panic(token)
    if ast.kind == "factor":
        # go to state 3
        ast = state3(ast)
    if ast.kind == "term":
        # go to state 2
        ast, tokens = state2(ast, tokens)
    # go to state 1
    return state1(ast, tokens)

def parse(tokens):
    return state0(tokens)

Note, we don’t actually need to check the ast.kind here because if ast = state4(token) succeeds, then we’re guaranteed to get a factor. The same goes for ast = state3(ast), which is guaranteed to return a term. So, we can rewrite state0 like this:

recursive-ascent.py

def state0(tokens: list[str]):
    token = peek(tokens)
    if token.isdigit():
        # shift token, move dot forwards
        tokens = tokens[1:]
        # go to state 4
        ast = state4(token)
    else:
        panic(token)
    # go to state 3
    ast = state3(ast)
    # go to state 2
    ast, tokens = state2(ast, tokens)
    # go to state 1
    return state1(ast, tokens)

Here are the helper function definitions. peek returns the next token, while panic prints an error message if an unexpected token is found and terminates the program:^{[12]
[12]}

recursive-ascent.py

def peek(tokens: list[str]):
    return "" if len(tokens) == 0 else tokens[0]

def panic(token: str):
    print("unexpected token: {}".format(token))
    exit()

As mentioned before, reduce happens when returning from a function:

	Action				Goto
State	+	*	INTEGER	eof	expr	term	factor
State 3	r5	r5	r5	r5
State 4	r6	r6	r6	r6

recursive-ascent.py

def state3(ast: AST):
    # reduce using rule 5: term -> factor
    return AST(ast.left, ast.right, ast.value, "term")

def state4(token: str):
    # reduce using rule 6: factor -> INTEGER
    return AST(None, None, token, "factor")

Now here’s where things get more interesting:

	Action				Goto
State	+	*	INTEGER	eof	expr	term	factor
State 1	s5			accept
State 2	r3	s6	r3	r3

recursive-ascent.py

def state1(ast: AST, tokens: list[str]):
    # loop to handle left recursion
    while peek(tokens) == "+":
        # shift token
        tokens = tokens[1:]
        # go to state 5
        ast, tokens = state5(ast, tokens)
    # accept
    return ast

def state2(ast: AST, tokens: list[str]):
    while peek(tokens) == "*":
        tokens = tokens[1:]
        ast, tokens = state6(ast, tokens)
    # reduce using rule 3: expr -> term
    return AST(ast.left, ast.right, ast.value, "expr"), tokens

The grammar we used has rules that contain left recursion. This happens when a rule starts with a symbol that is itself, e.g. expr -> expr '+' term and term -> term '*' factor. Top-down parsers can’t handle this type of grammar. If you’ve implemented a recursive descent parser before, a left-recursive rule will cause the parse function to immediately call itself before doing anything else, leading to infinite recursion and a stack overflow.

When using top-down parsers, you have to refactor the grammar to be right-recursive or tail-recursive. LR parsers can handle left recursion just fine, so this isn’t a problem for us. We can use a loop and a condition to check whether the recursive calls are done.

In state 1 and state 2, the parser has parsed the left side of the binary operation. In the next states, the parser will parse the right side of the binary operation:

	Action				Goto
State	+	*	INTEGER	eof	expr	term	factor
State 5			s4			gt7	gt3
State 6			s4				gt8

recursive-ascent.py

def state5(left: AST, tokens: list[str]):
    token = peek(tokens)
    if token.isdigit():
        tokens = tokens[1:]
        right = state4(tokens)
    else:
        panic(token)
    right = state3(right)
    return state7(left, right, tokens)

def state6(left: AST, tokens: list[str]):
    token = peek(tokens)
    if token.isdigit():
        tokens = tokens[1:]
        right = state4(tokens)
    else:
        panic(token)
    return state8(left, right), tokens

Once the right side is parsed, we can then reduce these rules in state 7 and state 8:

	Action				Goto
State	+	*	INTEGER	eof	expr	term	factor
State 7	r2	s6	r2	r2
State 8	r4	r4	r4	r4

recursive-ascent.py

def state7(left: AST, right: AST, tokens: list[str]):
    while peek(tokens) == "*":
        tokens = tokens[1:]
        right = state6(right, tokens)
    # reduce using rule 2: expr -> expr '+' term
    return AST(left, right, "+", "expr"), tokens

def state8(left: AST, right: AST):
    # reduce using rule 4: term -> term '*' factor
    return AST(left, right, "*", "term")

And that’s all of the code for the parser! Now we just have to check whether the output is correct. Let’s add a method in the AST class to pretty-print itself, so that it’s easier to visualise:

recursive-ascent.py

class AST:
    # ...

    def pprint(self):
        if not self.value.isdigit():
            print("( {} ".format(self.value), end="")
        else:
            print(" {} ".format(self.value), end="")
        if self.left:
            self.left.pprint()
        if self.right:
            self.right.pprint()
        if not self.value.isdigit():
            print(" )", end="")

I’m printing the AST as s-expressions so that it’s easy to see the expressions follow the correct operator precedence.^{[13]
[13]} Given an input like 1 + 2 * 3 + 4, this method will print out the following: ( + ( + 1 ( * 2 3 ) ) 4 ). This looks right. All that’s left now is to implement the evaluate function and test if it works as expected.

Interpreting the AST

Here’s the evaluate function:

recursive-ascent.py

def evaluate(ast):
    if ast.value == "+":
        return evaluate(ast.left) + evaluate(ast.right)
    elif ast.value == "*":
        return evaluate(ast.left) * evaluate(ast.right)
    else:
        return int(ast.value)

It will recursively evaluate all the nodes in the AST. Let’s run the program and check if the output is correct:

$ python3 recursive-ascent.py '1 + 2 * 3 + 4'
# 11

Good, it works! Now, let’s write some generated tests to have more confidence in the program. Here’s a function that would generate random numbers and combinations of operators (+ and *), run the evaluate function on them, and compare the result to Python’s eval function:

recursive-ascent.py

def tests():
    errors = 0
    for _ in range(1000):
        k = random.randint(2, 10)
        nums = [str(i) for i in random.sample(range(10_000), k)]
        ops = [random.choice(["+", "*"]) for _ in range(k - 1)]
        expr = [i for j in zip(nums, ops) for i in j] + [nums[-1]]
        input = " ".join(expr)
        res = evaluate(parse(lex(input)))
        py_res = eval(input)
        if res != py_res:
            print("incorrect output for:", input)
            print("got:", res)
            print("expected:", py_res)
            errors += 1
    if errors == 0:
        print("all tests passed!")

# also some changes to main to allow users to run the tests
def main():
    if len(sys.argv) != 2:
        print("usage: {} [STRING]".format(sys.argv[0]))
        return
    input = sys.argv[1]
    if input == "test":
        tests()
        return
    tokens = lex(input)
    ast = parse(tokens)
    result = evaluate(ast)
    print(result)

The eval function will run any Python expression. Since Python can also evaluate math expressions, we can use it as a reference to verify the correctness of our evaluate function. If you run python3 main.py test, you should get this output: all tests passed!.

End

And that’s all. You can find the complete code on this GitHub repo. Hopefully, you have a good idea of how LR and recursive ascent parsers work now. You probably won’t ever use this technique when building a real compiler, but at least you learned something new. If you’d like to learn more about this topic or explore other parsing techniques, check out the links in the footnotes.

This blog post by Russel Johnston about implementing a JSON parser in Rust using recursive ascent really helped me when I first started. It’s probably the only blog post about recursive ascent on the internet besides this one. ↩︎
Here are some papers about recursive ascent parsers that you could read if you’re interested in the theory. ↩︎
Crafting Interpreters is a really good introductory book on writing compilers, and its section on parsing is all about implementing a recursive descent parser. It’s also probably the most influential book in my programming journey. I learned a lot from it, which I applied in areas other than in writing compilers. Highly recommend reading it if you’re in software engineering. ↩︎
I found this analogy from this blog post by Josh Haberman. He went deep into the details of both LL and LR algorithms in his blog post while keeping it easy to read. ↩︎
Here’s a nice overview of parsing algorithm literature you can explore if you are interested in the different variants of LR. ↩︎
This is done to show that the parser can handle operator precedence based on the grammar rules. Subtraction is removed to keep the section short and readable. The number of possible states will grow exponentially the more complex your grammar is. This is why it is inconvenient to write a bottom-up parser by hand, and why people create parser generators. ↩︎
A grammar rule is often called a “production” in academic literature. I just use the term “grammar rules” because it’s more intuitive. ↩︎
My explanation of the algorithm for creating the parse table might not click with you, so here are some other resources you can read if you’re still confused. ↩︎
Note that the order of the states doesn’t matter. The parser will still work if the states were created in a different order, e.g. state 1 = state 0 + term. ↩︎
You can read more about why this is on this Wikipedia page. It should be a good starting point to explore the different variants of LR and the types of parse conflicts. For this article, I intentionally chose a grammar that is accepted by LR(0) to keep it simple. ↩︎
You can read Crafting Interpreters for a better idea of what production lexers do. ↩︎
This is a very basic approach to handling parse errors. Production-grade parsers will usually report errors, ignore them, and continue parsing. This is to catch as many errors as it can so that the programmer doesn’t have to recompile to find more errors. ↩︎
What better way to visualise AST than with s-expressions? When you read a lisp program, you’re looking directly at the program’s AST, which is just a lisp data structure! This is what clojurists often mean when they say “It’s just data”. ↩︎

Rewriting My Tests With Generative Testing

2026-04-04T12:25:25Z

Over the past few months, I rewrote most of my test suite. It was a cleanup effort as well as a chance to rethink how I approach testing, something I’ve been neglecting for most of my programming career.

There were two main reasons for rewriting my tests. The first reason was that the codebase I’ve been working on has grown, and more people are now working on it, but the existing tests weren’t adequate. At the time, they were the largest source of tech debt in the project, and I’m mostly to blame for that… Most of the tests were half-broken, vibe-coded slop just there to satisfy Codecov.^{[1]
[1]}

The other reason was that I wanted to try generative testing. I read and consume a lot of content about generative testing, e.g. from TigerBeetle and Antithesis, and I’ve always wanted to apply it to my own projects.

These eventually pushed me to delete my existing tests and rewrite them the way I imagined they should be. This post goes through that process and what I learned from it.

Notes on Terminology

Before going into more details, I just want to clarify some terms I’ll use. There are a lot of overloaded terms in testing (and software in general), and people often use the same words to mean different things. You may disagree with these, that’s fine. This is just what I mean when I use them.

I use only two scopes for tests:

Unit tests: Tests scoped to any component or subsystem of the full system. The size doesn’t matter. It can be a single function, a module, or a group of modules. Some people will call tests that span multiple units “integration tests”, but I disagree. That definition is too ambiguous. I use the term “integration” to describe a different concept.
System tests: Tests scoped to the entire system. I also consider tests that replace production components with test versions to still be system tests, as long as the overall system is exercised as a whole.

I use the term integration tests for tests (any scope) that interact with “black boxes”, e.g. third-party services, filesystems, networks, databases, etc. Or, in functional programming terms, impure tests. Tests without “integration” in their name implicitly mean pure tests. Also, I use the term end-to-end (E2E) tests specifically for system-scoped integration tests.

Rethinking My Testing Approach

Before starting the rewrite, I spent some time thinking about how to approach it and what my tests should look like. As you can already tell from the title of this post, most of the rewrite involves writing generative tests.

Generative testing is basically a catch-all term for tests where you write code that generates test cases, instead of you specifying the test cases themselves. This includes randomised approaches like property-based testing (PBT) or fuzzing, and non-random ones where you try every possible input.

Note, I’ll use the terms “PBT” and “fuzzing” interchangeably going forward. Both involve using randomised input to test some property of the system. There are subtle differences, but in practice they’re minor enough that I just consider them the same. ^{[2]
[2]}

Here are some of the main ideas that guided the rewrite:

Properties Over Examples

Property-based testing is a testing approach where you specify properties of your code and throw randomised input at it to break those properties. I’m not going to explain it in detail here as there are already plenty of good resources online, and I don’t have anything new to add.^{[3]
[3]} Here, I’ll just talk about my case for PBT.

While the system is still small, I wanted to catch most bugs early so they don’t compound as the system grows. The existing example-based tests are limited in this regard, as they only cover cases I already know. PBT is much more effective as they uncover cases and scenarios that we forgot or may never have thought of.

In other words, property-based tests find bugs; example-based tests catch regressions.

My plan was to replace most of the existing tests with property-based tests, keeping a few select example-based tests, which are still useful for:

Smoke tests: Sanity checks of the happy path, sometimes including common error cases . I mainly use them for code that doesn’t need extensive testing, e.g. thin wrappers or simple glue code.
Documentation: Example-based tests are more readable, so I leave some to illustrate API usage or internal behaviour.
Regression tests: Dedicated tests to ensure new changes don’t reintroduce known bugs.

System-Wide Fuzzing

System tests are usually the most valuable as they validate your system as a whole. The hardest bugs usually emerge only when components are used together. PBT/fuzzing is scope-agnostic, so it makes a whole lot of sense to also apply it in system tests.

There are two approaches I went with for this: the infamous deterministic simulation testing (DST) and traditional full-system fuzzing.^{[4]
[4]}

DST is basically just a system-scoped property-based test. Sure, it involves extra steps like replacing non-deterministic components (e.g. network, disk, clock, etc.) with deterministic counterparts, but the overall flow is the same as regular fuzzing or stateful PBT. If you want to learn more about DST, I recommend reading Antithesis’s DST primer or Phil Eaton’s blog post.

The main pitch of DST is that it finds the most outlandish bugs for you while keeping them perfectly reproducible, but as I eventually learned, it’s not the silver bullet I initially thought it was. More on this later!

The other approach is an E2E test where we fuzz the system in its production configuration, i.e. without replacing non-deterministic components. This means that the test runs in wall clock time and that they may be flaky, but it provides more coverage as the bugs it catches may not show up in DST. This follows the same principles as TigerBeetle’s Vortex.

Testing with Intent

This isn’t related to generative testing, but I still think it’s worth including. One thing I started to pay more attention to now in my tests is the intent or thought behind it.

It’s a similar concept to design. If you read design case studies or write-ups, you might be surprised at the amount of thought put into seemingly trivial details. Every decision is made on purpose. It’s the same for code. I think through the program’s architecture and how to structure the code, but I’ve always neglected doing it for tests.

You may argue that tests already have a purpose by default, e.g. to verify correctness, catch regressions, etc. I’m talking about stuff beyond that. Stuff like:

Is this test testing intended behaviour, or is it just copying your implementation? The latter gives you false confidence, e.g. it can cover every branch and still be wrong.
How should I test this? Not everything needs the same level of testing. Sometimes a simple smoke test is enough vs a full-blown exhaustive test.
Do I need a test for this? Do I care enough about this to add a test? Every code you add is code you have to maintain.

Thinking about a test’s intent also forces you to reason about the system under test (SUT). I treat it as a design exercise; some people describe it like an experiment’s hypothesis. I’ve also started adding comments on my tests to make them easier for other developers to understand, e.g. why it exists, what behaviour/properties it checks, how it actually verifies them, etc.

The System Under Test

Here’s a brief overview of the SUT for context in the examples later. All examples in this post are drawn from Cardinal, the game server component of the World Engine stack. It’s written in Go.

To be more specific, Cardinal is a tick-based game server framework built around an archetypal entity-component-system (ECS) core. Users implement game logic as systems, which are executed each tick to advance the world state. Cardinal communicates with other services through NATS and persists state using periodic snapshots.

Every tick, Cardinal injects commands received from external services (e.g. game clients) into the ECS core and runs the registered systems. These systems process commands, update the world state, and emit events. After the tick ends, event handlers publish the emitted events to NATS for other services to consume.

Here’s a diagram showing Cardinal’s high-level architecture:

Ok, hope that made sense. Now, on to how I wrote the tests.

Property-Based Testing Techniques

I started by writing unit tests first, on small units, just to familiarise myself with PBT before moving on to larger units and the full system. This section lists most of the techniques I learned while doing it.

The hardest part of PBT is figuring out what properties to test, so I think one of the best ways to learn is to just see a lot of examples to start recognising patterns. Most of the things mentioned here came from TigerBeetle’s blog and source code, as that’s where I learned them. If you’re also interested in PBT/fuzzing, definitely give them a read!

Testing Data Structures

Cardinal is mostly made up of data structures. Testing them is a bit different from testing pure functions often used in PBT examples. With pure functions, you simply assert on the output. With data structures, there are two ways to do it:

Assert internal properties (invariants). An invariant is a condition that must always be true. You can test these if your tests have access to internal fields, or by placing assertions directly in the implementation.
Assert against a model. This is called model-based testing (MBT), or stateful PBT, state machine testing, differential testing, etc. In MBT, you compare the behaviour of your implementation against a model with the same semantics. The model can be a simplified implementation or a trusted reference, e.g. a data structure from your language’s standard library.

The idea is simple: run the same operations on both your implementation and the model, and assert that their states match.

I’ll use Cardinal’s componentManager to illustrate. Here’s what it looks like:

pkg/cardinal/internal/ecs/component.go

type componentManager struct {
	nextID    componentID            // The next available component ID
	catalog   map[string]componentID // Component name -> component ID
	factories []columnFactory        // Component ID -> column factory
}

// register registers a new component type and returns its ID.
func (cm *componentManager) register(name string, factory columnFactory) (componentID, error) {}

// getID returns a component's ID given its name.
func (cm *componentManager) getID(name string) (componentID, error) {}

While the manager does more internally, the bulk of its work is component registrations, which basically involves mapping component IDs to their names. Since this behaviour is essentially like a map, I used Go’s built-in map as the model. Here’s a simplified version of the test:

pkg/cardinal/internal/ecs/component_internal_test.go

func TestComponent_RegisterModelFuzz(t *testing.T) {
	prng := testutils.NewRand(t)
	impl := newComponentManager()
	model := make(map[string]componentID)

	operations := []string{opRegister, opGetID} // Just string constants
	for range 10_000 {
		switch testutils.RandOp(prng, weights) { // Pick a random operation
		case opRegister:
			name := randValidComponentName(prng)
			implID, implErr := impl.register(name, nil)
			modelID, modelExists := model[name]

			require.NoError(t, implErr)
			if modelExists {
				// Property: duplicate registrations return same ID.
				assert.Equal(t, modelID, implID)
			} else {
				model[name] = implID
			}
		case opGetID:
			name := randProbablyRegisteredComponentName(prng) // Can return non-existent keys
			implID, implErr := impl.getID(name)
			modelID, modelExists := model[name]

			// Property: getID returns same existence and value as model.
			assert.Equal(t, modelExists, implErr == nil)
			if modelExists {
				assert.Equal(t, modelID, implID)
			}
	}

	// Property: implementation matches model.
	assert.Len(t, impl.catalog, len(model))
	for name, modelID := range model {
		implID, err := impl.getID(name)
		require.NoError(t, err)
		assert.Equal(t, modelID, implID)
	}
}

Now you might be thinking: wait, doesn’t the implementation also use a map? The model is the same as the implementation. Your test essentially just compares two maps. Is it even doing anything?

I had similar doubts too when I first wrote this. The benefits of this test aren’t obvious at first (to me at least), but it becomes clear once you change the implementation. For example, I could replace the map with an exotic data structure, e.g. to get a 100x performance boost. This test will still work (as long as the semantics are the same, of course) because it tests the behaviour, not the implementation.

Note, MBT and internal assertions aren’t mutually exclusive. Ideally, you should use both. It’s not shown in the example above, but I also have assertions in the implementation to guard against invalid states. They check for different things: internal assertions ensure invariants hold, MBT ensures it behaves correctly.

Swarm Testing

I complement all my model-based tests with swarm testing. In the previous example, the test picks a random operation, where every operation has the same chance of being selected (a uniform distribution). In practice, this usually doesn’t lead to interesting scenarios.

The idea behind swarm testing is to randomise the test configuration. In our case, that means randomising which operations are performed. We can do this by picking a subset of all possible operations and assigning each a random weight (probability of being picked).

TigerBeetle uses Zig’s metaprogramming magic to automatically derive operations from a struct’s fields. You could probably do this in Go using runtime reflection, which I only realised while writing this… What I ended up doing was writing helper functions that take a hardcoded list of operations and randomises them:

pkg/testutils/random.go

type OpWeights = map[string]uint64 // Operation names -> weights

// RandOpWeights selects a random subset of operations and assigns random weights to them.
// At minimum 1 operation is enabled, at most all operations are enabled.
func RandOpWeights(r *rand.Rand, ops []string) OpWeights {
	assert.That(len(ops) > 0, "you need multiple operations to randomize")

	// Randomly select how many operations to enable (1 to len(ops)).
	numEnabled := 1 + r.IntN(len(ops))

	// Shuffle and take the first numEnabled operations.
	shuffled := slices.Clone(ops)
	r.Shuffle(len(shuffled), func(i, j int) {
		shuffled[i], shuffled[j] = shuffled[j], shuffled[i]
	})
	weights := make(map[string]uint64, numEnabled)
	for i := range numEnabled {
		weights[shuffled[i]] = uint64(1 + r.IntN(100))
	}
	return weights
}

// RandWeightedOp returns a random weighted operation from a OpWeights.
func RandWeightedOp(r *rand.Rand, ops OpWeights) string {
	var total uint64
	for _, weight := range ops {
		total += weight
	}
	pick := r.Uint64N(total)
	for op, weight := range ops {
		if pick < weight {
			return op
		}
		pick -= weight
	}
	panic("unreachable")
}

Here’s what the previous test looks like with swarm testing:

pkg/cardinal/internal/ecs/component_internal_test.go

func TestComponent_RegisterModelFuzz(t *testing.T) {
	prng := testutils.NewRand(t)

	// Get a random weighted subset of your operations.
	operations := []string{opRegister, opGetID}
	weights := testutils.RandOpWeights(prng, operations)

	for range 10_000 {
		op := testutils.RandWeightedOp(prng, weights)
		switch op {
		case opRegister:
			// ...
		case opGetID:
			// ...
	}
	// ...
}

Initially, I thought regular MBT was enough, until I found a bug in my code. It’s the type of bug that can be found faster through swarm testing. Here’s a snippet of the buggy code from Cardinal’s event manager type. Can you spot the issue?

pkg/cardinal/internal/event/event.go

func NewManager(channelCapacity int) Manager {
	return Manager{
		channel: make(chan Event, channelCapacity),
		// ...
	}
}

// Enqueue enqueues an event into the Manager.
func (m *Manager) Enqueue(event Event) {
	e.channel <- event
}

The bug here is that Enqueue blocks when the channel capacity is reached.

The existing tests never found it, and it never happened during local development, so this bug went unnoticed for a long time. While I was writing the model tests, I experimented with increasing the number of operations. The tests eventually got stuck, revealing the bug.

This is exactly the kind of bug swarm testing helps catch. The original test runs both enqueue and dequeue operations, so the channel never fills up. If the dequeue operation is disabled or is less likely to be picked, the channel quickly reaches capacity, and the bug becomes obvious!

After fixing the bug, I added a regression test to ensure it doesn’t happen again:

pkg/cardinal/internal/event/event_test.go

func TestEvent_EnqueueChannelFull(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		const channelCapacity = 16
		const totalEvents = channelCapacity * 3 // Well beyond channel capacity

		impl := event.NewManager(channelCapacity)

		// Enqueue more events than channel capacity.
		go func() {
			for i := range totalEvents {
				impl.Enqueue(event.Event{Kind: event.KindDefault, Payload: i})
			}
		}()

		// Wait for all goroutines to complete or durably block.
		// If enqueue blocks, synctest.Test will detect deadlock and fail.
		synctest.Wait()

		// Extra property checks...
}

This test uses Go’s new testing/synctest to detect deadlocks. Unlike traditional timeout-based tests, goroutines spawned inside the synctest.Test bubble are tracked, which allows the runtime to detect deadlocks/leaks quicker and deterministically.

Concerns with Model-Based Testing

A big concern I had when writing these tests was how do I know whether my model is correct. I’m usually confident when the models are from Go’s standard library, since they’ve probably been tested to death by maintainers and by users in production.

But sometimes the standard library doesn’t have anything with the same behaviour as the data structure I’m testing. In these cases, I’m forced to write my own simplified model implementation. The hard part was verifying the model. How do I know if it’s correct? Do I test my model? What do I test it against? Should I test my tests? It’s turtles all the way down.

Turns out this is a well-known problem called the test oracle problem. A test oracle is basically what determines the correct output of a test. In example-based tests, you are the oracle because you specify the expected output. In MBT, the model is the oracle. As far as I know, there’s no complete solution to this problem.^{[5]
[5]}

My only practical suggestion is to make your models (and tests in general) “obviously correct”. Prioritise simplicity and clarity above everything else. Keep the code short and easy to audit. It doesn’t matter if it’s slow or inelegant. Ideally, you should be able to convince yourself (and others) that the code is correct just by reading it.

Reducing the State Space

Some behaviours are surprisingly difficult to test, especially when they depend on many interacting variables. The number of possible variable combinations grows quickly, leading to a state space explosion. When this happens, uniform random sampling becomes less effective as most samples fall into “empty space” rather than edges and boundaries where bugs often hide.

Here’s an example. Cardinal’s ECS has the concept of archetypes. An archetype represents a unique combination of component types. It stores a list of entities belonging to that archetype as well as their component data. Here’s what it looks like:

pkg/cardinal/internal/ecs/archetype.go

type archetype struct {
	id         archetypeID      // Corresponds to the index in the archetypes array
	components bitmap.Bitmap    // Bitmap of components contained in this archetype
	rows       sparseSet        // Maps entity ID -> row index in entities and columns
	entities   []EntityID       // List of entities of this archetype
	columns    []abstractColumn // List of columns containing component data
}

When components are added to and/or removed from an entity, it is moved to a different archetype with the required component set. This operation involves creating the entity in the destination archetype, copying its components, and removing it from the source archetype:

pkg/cardinal/internal/ecs/archetype.go

// moveEntity moves an entity from one archetype to another.
func (a *archetype) moveEntity(destination *archetype, eid EntityID) {
	if a == destination {
		return
	}
	row, exists := a.rows.get(eid)
	assert.That(exists, "entity is not in archetype")

	// Create a new entity in the destination archetype with the id.
	destination.newEntity(eid)
	newRow, exists := destination.rows.get(eid)
	assert.That(exists, "new entity isn't created in the destination archetype")

	// Move entity's components to the new archetype.
	for _, dst := range destination.columns {
		for _, src := range a.columns {
			if dst.name() == src.name() {
				value := src.getAbstract(row)
				dst.setAbstract(newRow, value)
			}
		}
	}
	// Remove the entity from the current archetype, which also updates the row mapping.
	a.removeEntity(eid)
}

There are a lot of variables involved in a move operation:

The source and destination archetypes.
The number of components added, removed, or copied.
The number of entities in both archetypes.
Which entity is being moved.

If we only consider the source and destination archetypes, the input space is already huge. If there are $N$ component types, there are $2^{N}$ possible archetypes (distinct subsets). Cardinal supports up to $2^{32}$ component types, so the theoretical number of archetypes is $2^{2^{32}}$ . This means there are $2^{2^{32}} \times 2^{2^{32}}$ source-destination pairs. That’s a number with 258,583 digits!

To make testing this more effective, I intentionally limited the size of the state space. The rationale is based on the small scope hypothesis, which argues that most bugs can be found within small input sizes.^{[6]
[6]} Larger inputs don’t necessarily lead to new execution paths. Borrowing the words of Matklad:

Most of the bugs involve small, tricky examples. If a sorting routine breaks on some array with ten thousand elements it’s highly likely that there’s a much smaller array (a handful of elements), which exposes the same bug.

With a smaller scope, we can feasibly test all possible variable combinations exhaustively. Some of you might already know where I’m heading here. Yes, I’m talking about Exhaustigen. It’s a helper library for generating combinations of multiple variables exhaustively. I ported TigerBeetle’s Zig version to Go, which you can find here.

For my archetype move test, I reduced my variables to:

Maximum 6 component types, which results in 64 possible archetypes. This produces $64^{2} = 4096$ source-destination pairs. I arrived at this number after some experimentation. Higher counts lead to exponentially longer test runs.
1-3 entities in the source and 0-2 entities in the destination. The source must have at least one entity to move.
The entity to be moved is randomly chosen from the source. Entities and components are stored in slices under the hood. This verifies moves work correctly regardless of whether the entity is at the beginning, middle, or end of the slice.

With that, my number of test cases becomes: $4096 \times 3 \times 3 \times 3 = 110, 592$ . This is small enough to run in under a second, but large enough to go beyond the simplest cases.

Here’s a simplified version of the test:

pkg/cardinal/internal/ecs/archetype_internal_test.go

func TestArchetype_MoveExhaustive(t *testing.T) {
	prng := testutils.NewRand(t)

	// Generate all possible subsets (archetype) with N component types.
	const maxComponents = 6
	pool := newArchetypePool(maxComponents)

	// Exhaustively test combinations of source * destination * entity count * moved entity row.
	gen := testutils.NewGen()
	for !gen.Done() {
		src := pool[gen.Index(len(pool))] // Randomize source
		dst := pool[gen.Index(len(pool))] // Randomize destination

		// Populate source with 1-3 entities, destination with 0-2 entities.
		countSrc := gen.Range(1, 3) // Randomize source entity count
		for eid := range countSrc {
			src.newEntity(EntityID(eid))
		}
		countDst := gen.Range(0, 2) // Randomize destination entity count
		for eid := range countDst {
			dst.newEntity(EntityID(countSrc + eid)) // Entity IDs are globally unique
		}

		// Pick which entity to move.
		row := gen.Index(len(src.entities)) // Randomize moved entity row
		eid := src.entities[row]

		if src == dst {
			assert.NotPanics(t, func() {
				src.moveEntity(dst, eid)
			}, "self move should be a no-op and must not panic")
			continue
		}

		src.moveEntity(dst, eid) // Do the actual move.

		// Assert properties here...
	}
}

Exhaustigen’s API is so nicely designed. It still amazes me that this small amount of code is enough to cover all variable combinations. It does some intricate machinery under the hood, so if you’re curious about how it works, check out the original blog post!

On Tooling

I didn’t use any third-party PBT libraries, e.g. Rapid, or Go’s testing/quick and testing.F. For my use cases, Go’s rand.Rand PRNG already provided most of the functionality I needed.

As I wrote the tests, I gradually extracted utilities once the patterns became clear. These helpers live in a small testutils package. The first thing I added was support for a global seed for the test PRNGs.

Since PBT relies on randomness, rerunning the same test can produce different results. To make failures reproducible, I seed all test PRNGs with the same value. If a test fails, I can reproduce it by rerunning it using the same seed:

TEST_SEED=0x0123456789abcdef go test ./...

The seed is printed if you run the tests with the -v (verbose) flag. Here’s the code for it:

pkg/testutils/random.go

var Seed uint64 // Global test seed

func init() {
	Seed = uint64(time.Now().UnixNano())
	if envSeed := os.Getenv("TEST_SEED"); envSeed != "" {
		parsed, err := strconv.ParseUint(envSeed, 0, 64)
		if err == nil { // Only set using the env if it's valid
			Seed = parsed
		}
	}
	fmt.Printf("to reproduce: TEST_SEED=0x%x\n", Seed)
}

func NewRand(t *testing.T) *rand.Rand {
	t.Helper()
	// We derive a unique seed for each test by XOR-ing the global seed with a hash of the test name.
	h := fnv.New64a()
	_, err := h.Write([]byte(t.Name()))
	require.NoError(t, err)
	perTest := h.Sum64()
	return rand.New(rand.NewPCG(Seed^perTest, Seed^(perTest<<1)))
}

Tests use NewRand as their source of randomness. Each test also uses a different seed, which is derived from the same original seed. This prevents parallel tests from generating the same random values. While this isn’t a problem for isolated tests, some tests share a common resource, e.g. a server or database. Using the same seed could cause them to pick the same resources, leading to collisions and tests corrupting other tests.^{[7]
[7]}

(Almost) Deterministic Simulation Testing

After I finished with the unit tests, I moved on to implement DST. Cardinal wasn’t initially designed with DST in mind, so it took quite a few refactors to get it mostly there. Almost all of the non-deterministic parts are stubbed out, except for the concurrent systems scheduler.

It’s a bit difficult to replace, so I just left it there.^{[8]
[8]} To guard against scheduler bugs that may not be 100% reproducible, I wrote extra tests to verify the scheduler’s internal concurrency works correctly. This way, I can still have confidence in its behaviour without needing to fully eliminate its non-determinism.

I’ve always thought it would be hard to implement DST, considering my first reference is TigerBeetle’s VOPR, which has a huge number of components and configurations. Then, I stumbled upon this blog post by Oskar Wickstrom about implementing a lightweight DST for Amp.^{[9]
[9]} It showed me that at its core, DST isn’t really a complicated concept, which made DST more attemptable.

I had the same realisation while writing the model-based tests. They looked a whole lot like the lightweight DST from the blog post. And they are! DST is basically just a bigger model-based test, on the whole system, with non-deterministic components stubbed out. Since I already wrote a lot of them, implementing DST didn’t turn out to be too hard.^{[10]
[10]}

How it Works

I don’t remember where I first heard this, but TigerBeetle is structured so that it moves in fixed ticks, like a game engine.^{[11]
[11]} Time is dependency-injected into the tick, not the other way around like in traditional code. In production use, this uses a regular wall clock, but in simulation, it uses a simulated clock. Well, Cardinal is a tick-based game engine! It was already well-prepared for DST in this regard.

Cardinal only updates its state on every call to Tick:

pkg/cardinal/cardinal.go

// NOTE: The World type is Cardinal.

func (w *World) Tick(ctx context.Context, timestamp time.Time) {
	w.currentTick.timestamp = timestamp

	// Collect the commands for this tick.
	_ = w.commands.Drain()

	// Tick ECS world.
	w.world.Tick()

	// Emit events.
	if err := w.events.Dispatch(); err != nil {
		w.tel.Logger.Warn().Err(err).Msg("errors encountered dispatching events")
	}

	// Publish snapshot every N ticks.
	if w.currentTick.height%uint64(w.options.SnapshotRate) == 0 {
		snapshotCtx, cancel := context.WithTimeout(ctx, 2*time.Second)
		w.snapshot(snapshotCtx, timestamp)
		cancel()
	}

	// Increment tick height.
	w.currentTick.height++
}

In production use, Cardinal’s StartGame method handles its lifecycle. It basically runs an infinite loop, calling Tick every N seconds. For the simulation test, all I had to do was just replace StartGame and have the test harness control Cardinal using Tick.

Here’s a simplified version of the DST harness:

pkg/cardinal/dst.go

func RunDST(t *testing.T, setup func(*World)) {
	prng := testutils.NewRand(t)
	cfg := newDSTConfig(prng)
	fix := newDSTFixture(t, cfg, setup)

	// Add the world's commmands as operations in the dst config.
	cfg.addCommandsAsOperations(fix.world)
	cfg.log(t) // Log the test config

	// The main test loop:
	tick := 0
	for tick < cfg.Ticks {
		op := testutils.RandWeightedOp(prng, cfg.OpWeights)
		switch {
		case op == opTick:
			timestamp := time.Unix(int64(tick), 0)
			fix.world.Tick(context.Background(), timestamp)
			// Assert structural ECS invariants after every tick.
			ecs.CheckWorld(t, fix.world.world)
			tick++

		case strings.HasPrefix(op, opCommandPrefix):
			cmdName := strings.TrimPrefix(op, opCommandPrefix)
			cmd := fix.randCommand(t, prng, cmdName)
			require.NoError(t, fix.world.commands.Enqueue(cmd))

		case op == opRestart:
			fix.world.reset()

		case op == opSnapshotRestore:
			fix.world.reset()
			require.NoError(t, fix.world.restore(context.Background()))
		}
	}

	// Final validation after all randomized operations complete.
	ecs.CheckWorld(t, fix.world.world)
}

This does a few things:

Create a test configuration with random values. There are only a few ATM, and no fancy fault injections, but it’s a start.
Initialise Cardinal with that configuration and replace non-deterministic components (e.g. NATS handlers, snapshot storage) with deterministic fakes that contain assertions. The non-deterministic parts are tested separately in integration tests.
Register systems using the user-provided setup function. I’ll talk more about this soon.
Extract type information from the command types used in the systems using reflection. We’ll use this to generate random commands to send to the Cardinal.
Run for N ticks (configurable) and perform a random operation each time.
We switch between several operations: tick the world, send a random command, reset the world, and restore from a snapshot. The tick operation is always enabled, but the rest are random (as in swarm testing).
We validate the internal state every tick and at the end of the test.

While implementing this, there was a major design choice I had to make. Cardinal is a framework, and its input is user-provided game logic in the form of systems. I’ll refer to this set of systems as just “game”. The test harness above can only verify Cardinal’s internal invariants. It cannot verify whether the game is logically correct. I had to make a choice between:

Keeping the harness specific to a single, hand-coded game designed to test Cardinal. This allows us to add more instrumentation to verify some extra invariants and also to validate the game logic, but also means it must be a private type to access the required internal fields.
Make the harness game-agnostic and public. This gives a few benefits, namely, users can use it with their games, and we also get free inputs to test Cardinal on. We can only verify a subset of all our properties, but we’re more likely to get something we didn’t expect.

In the end, I went with the second design, betting that this choice would find more bugs through user-provided inputs. I’m hoping that someone somewhere would do something so weird and unpredictable that it crashes the test. Sticking with a single input might cause the harness to be biased.

So now, we have an importable DST harness that users can use, but not to test their game logic, rather to test that Cardinal works correctly with it. It’s not a substitute for properly testing your game logic. Here’s what its usage looks like:

example_test.go

import "github.com/argus-labs/world-engine/pkg/cardinal"

func YourTestDST(t *testing.T) {
	cardinal.RunDST(t, func(w *cardinal.World) {
		// Register your systems here...
		cardinal.RegisterSystem(w, system.PlayerSpawnerSystem, cardinal.WithHook(cardinal.Init))
		cardinal.RegisterSystem(w, system.CreatePlayerSystem)
		cardinal.RegisterSystem(w, system.RegenSystem)
		cardinal.RegisterSystem(w, system.AttackPlayerSystem)
	})
}

This will run the simulation for 1000 ticks by default. You can configure this using the -dst.ticks flag.

Expectations VS Reality

So initially, I tested with a simple game that I asked AI to write. I was really excited because of all the hype around DST, like how it seemingly found all the bugs in FoundationDB. The first time I ran it, the test froze. When I reran it, it froze again! My DST has found its first bug!

After some investigation, I found out that it was a bug in Cardinal’s scheduler. It’s a deadlock that happens only when the world is reset after an odd number of ticks. A ridiculous bug! I probably would’ve caught this myself in local development, but the fact that the simulator caught it this fast was amazing.

Then I ran the test again, a lot of times. Each iteration has a different seed and configuration. Nothing. Sometimes I get a bug, but most turned out to be bugs in the harness rather than Cardinal. I tried tuning the test configuration a little bit, but still nothing. At that point, I wasn’t sure what was wrong. Was my harness too simple to find more bugs? Or was my code just bug-free (highly unlikely)? Was it finding bugs, but I didn’t have the correct assertions to catch them?

I don’t know the answer. And it seems I wasn’t the only one with this issue. The other day, I was watching the BugBash Podcast episode with the co-authors of DDIA (2nd edition), and Chris Riccomini talked about his experience adding DST to SlateDB. He mentioned things like DST being high-investment but low-return, didn’t explore that much state, and that it only found three known bugs. This matched perfectly with my (short) experience!

Chris also mentioned that a better approach might be to apply DST on subcomponents much earlier in the process instead of going all in with full-system DST, which would’ve given more value much earlier. Coincidentally, this is what I did (if you count the model-based tests as DST), and it also aligns with my experience. I had caught several bugs from those tests. It’s probably why my DST didn’t catch many bugs, but that’s just my wishful thinking.

I’ll also add this quote from Will Wilson, the CEO of Antithesis:

The biggest challenge of DST in my experience is that tuning all the random distributions, the parameters of your system, the workload, the fault injection, etc. so that it produces interesting behaviour is very challenging and very labor intensive. As with fuzzing or PBT, it’s terrifyingly easy to build a DST system that appears to be doing a ton of testing, but actually never explores very much of the state space of your system. At FoundationDB, the vast majority of the work we put into the simulator was an iterative process of hunting for what wasn’t being covered by our tests and then figuring out how to make the tests better. This process often resembles science more than it does engineering.

I’m still not giving up, though. My DST harness is still small, which means it can only grow from here. I already have several improvements in mind, so I’ll stick with it for a bit longer.

End-to-End Fuzzing

The final test I added was the E2E tests. This actually followed a similar structure to the DST, where we set up Cardinal with user-provided game logic and bombard it with randomised commands. The biggest difference is that it runs with the same, non-deterministic components as it would in production.

The actual harness code is pretty gnarly, so here’s an extremely simplified version of it:

pkg/cardinal/e2e.go

func RunE2E(t *testing.T, setup E2ESetupFunc) {
	prng := testutils.NewRand(t)
	cfg := newE2EConfig()
	fix := newE2EFixture(t, setup)

	// Add the world's commmands as operations in the dst config.
	cfg.addCommandsAsOperations(fix.world)
	cfg.log(t) // Log the test config

	// Start the world's run loop in a background goroutine.
	ctx, cancel := context.WithCancel(context.Background())
	runErr := make(chan error, 1)
	go func() {
		runErr <- fix.world.run(ctx)
	}()

	deadline := time.After(cfg.Duration)
sendLoop:
	for {
		select {
		case <-deadline:
			err := stopWorld()
			if err != nil && !errors.Is(err, context.Canceled) {
				t.Fatalf("world.run returned unexpected error: %v", err)
			}
			break sendLoop
		default:
			name := testutils.RandWeightedOp(prng, cfg.CommandWeights)
			cmd := fix.randCommand(t, prng, name)
			fix.sendCommand(t, cmd)
		}
	}

	// Final validation after the world has fully stopped.
	ecs.CheckWorld(t, fix.world.world)
}

All this does is run Cardinal in a separate goroutine and send it randomised requests. Unlike in DST, where you control the number of ticks to run, here you control the test duration. Since it uses wall clock time, it’ll run much slower than other tests. This is why I made the E2E tests disabled by default. To run them, pass the -e2e.run flag to go test.

The usage API is also similar to DST, but instead of passing a function to register systems, you’d return the actual cardinal.World instead. In DST, we randomise Cardinal’s config, but in the E2E test, we want it to have the same config as production to test more realistically:

example_test.go

import "github.com/argus-labs/world-engine/pkg/cardinal"

func TestE2E(t *testing.T) {
	cardinal.RunE2E(t, func() *cardinal.World {
		world, err := cardinal.NewWorld(cardinal.WorldOptions{
			// Your world configuration...
		})
		require.NoError(t, err)

		// Register systems here...

		return world
	})
}

What’s Next?

Phew… That was long. I had cut out a lot of things, including a section on testing concurrent code, but I still ended up with around 7000 words. If you read the entire thing, thank you and big props to you.

Here’s the source code if you want to take a closer look. After all this, I’m even more excited about software correctness. There are so many more things I want to try, e.g. improve my DST harness, test using a deterministic hypervisor :), formal methods, etc.

Only the tests were slop, I actually put a lot of thought into the code itself! ↩︎
I like comparing PBT and fuzzing to convergent evolution, which is when species of different lineages evolve similar features. PBT came from functional programming, fuzzing came much earlier from systems/security. The approach is similar, but their focus is different (program correctness vs security issues). Yes, I got this from Antithesis. ↩︎
I recommend reading Antithesis’s PBT primer first, and then F# for Fun and Profit’s PBT series for some more practical tips. ↩︎
My real reason for adopting DST is just because I think it’s cool. It’s what got me interested in TigerBeetle in the first place. ↩︎
There are some approaches that address this problem (somewhat), e.g. metamorphic testing. It solves this by using oracles derived from the tests themselves (through special properties called metamorphic relations), but it’s not a universal solution. ↩︎
This is probably why shrinking in pbt works. While the original intent is to create a simpler representation for debugging, the fact remains that the same bugs found in larger inputs can usually be reproduced in smaller ones. ↩︎
I tried to keep the explanation short earlier, so here’s a concrete example. Some of my integration tests share a single, in-memory NATS server. When tests used the same seed, I noticed they would sometimes fail. The reason for this is that multiple tests were accessing the same NATS subjects, even though the tests were meant to be isolated. Using a different seed for each test solves this.

Also, before the rewrite, each test would create its own NATS server. This is simpler, but it fails when there are a lot of parallel tests. The tests would use up all memory in CI (and sometimes in my local machine) and OOM. ↩︎
It’s technically easy to replace, and I’m also considering replacing it with a simpler, sequential version, but first, I need to do some benchmarks with a lot of different system schedules to understand the implications. It’ll take a while, so I decided to just leave it there so I could get the DST set up first. ↩︎
Which is also my coding agent of choice :) ↩︎
There’s another way to do DST in Go. This blog post by Polar Signals explains how they implemented DST in Go by compiling their program to WASM and (slightly) modifying the Go runtime. ↩︎
Here’s the code, check for yourself! ↩︎

Build and Deploy Web Apps With Clojure and FLy.io

2026-04-04T12:25:25Z

This post walks through a small web development project using Clojure, covering everything from building the app to packaging and deploying it. It’s a collection of insights and tips I’ve learned from building my Clojure side projects, but presented in a more structured format.

As the title suggests, we’ll be deploying the app to Fly.io. It’s a service that allows you to deploy apps packaged as Docker images on lightweight virtual machines.^{[1]
[1]} My experience with it has been good; it’s easy to use and quick to set up. One downside of Fly is that it doesn’t have a free tier, but if you don’t plan on leaving the app deployed, it barely costs anything.

This isn’t a tutorial on Clojure, so I’ll assume you already have some familiarity with the language as well as some of its libraries.^{[2]
[2]}

Project Setup

In this post, we’ll be building a barebones bookmarks manager for the demo app. Users can log in using basic authentication, view all bookmarks, and create a new bookmark. It’ll be a traditional multi-page web app and the data will be stored in a SQLite database.

Here’s an overview of the project’s starting directory structure:

.
├── dev
│   └── user.clj
├── resources
│   └── config.edn
├── src
│   └── acme
│       └── main.clj
└── deps.edn

And the libraries we’re going to use. If you have some Clojure experience or have used Kit, you’re probably already familiar with all the libraries listed below.^{[3]
[3]}

deps.edn

{:paths ["src" "resources"]
 :deps {org.clojure/clojure               {:mvn/version "1.12.0"}
        aero/aero                         {:mvn/version "1.1.6"}
        integrant/integrant               {:mvn/version "0.11.0"}
        ring/ring-jetty-adapter           {:mvn/version "1.12.2"}
        metosin/reitit-ring               {:mvn/version "0.7.2"}
        com.github.seancorfield/next.jdbc {:mvn/version "1.3.939"}
        org.xerial/sqlite-jdbc            {:mvn/version "3.46.1.0"}
        hiccup/hiccup                     {:mvn/version "2.0.0-RC3"}}
 :aliases
 {:dev {:extra-paths ["dev"]
        :extra-deps  {nrepl/nrepl    {:mvn/version "1.3.0"}
                      integrant/repl {:mvn/version "0.3.3"}}
        :main-opts   ["-m" "nrepl.cmdline" "--interactive" "--color"]}}}

I use Aero and Integrant for my system configuration (more on this in the next section), Ring with the Jetty adaptor for the web server, Reitit for routing, next.jdbc for database interaction, and Hiccup for rendering HTML. From what I’ve seen, this is a popular “library combination” for building web apps in Clojure.^{[4]
[4]}

The user namespace in dev/user.clj contains helper functions from Integrant-repl to start, stop, and restart the Integrant system.

dev/user.clj

(ns user
  (:require
   [acme.main :as main]
   [clojure.tools.namespace.repl :as repl]
   [integrant.core :as ig]
   [integrant.repl :refer [set-prep! go halt reset reset-all]]))

(set-prep!
 (fn []
   (ig/expand (main/read-config)))) ;; we'll implement this soon

(repl/set-refresh-dirs "src" "resources")

(comment
  (go)
  (halt)
  (reset)
  (reset-all))

Systems and Configuration

If you’re new to Integrant or other dependency injection libraries like Component, I’d suggest reading “How to Structure a Clojure Web”. It’s a great explanation of the reasoning behind these libraries. Like most Clojure apps that use Aero and Integrant, my system configuration lives in a .edn file. I usually name mine as resources/config.edn. Here’s what it looks like:

resources/config.edn

{:server
 {:port #long #or [#env PORT 8080]
  :host #or [#env HOST "0.0.0.0"]
  :auth {:username #or [#env AUTH_USER "john.doe@email.com"]
         :password #or [#env AUTH_PASSWORD "password"]}}

 :database
 {:dbtype "sqlite"
  :dbname #or [#env DB_DATABASE "database.db"]}}

In production, most of these values will be set using environment variables. During local development, the app will use the hard-coded default values. We don’t have any sensitive values in our config (e.g., API keys), so it’s fine to commit this file to version control. If there are such values, I usually put them in another file that’s not tracked by version control and include them in the config file using Aero’s #include reader tag.

This config file is then “expanded” into the Integrant system map using the expand-key method:

src/acme/main.clj

(ns acme.main
  (:require
   [aero.core :as aero]
   [clojure.java.io :as io]
   [integrant.core :as ig]))

(defn read-config
  []
  {:system/config (aero/read-config (io/resource "config.edn"))})

(defmethod ig/expand-key :system/config
  [_ opts]
  (let [{:keys [server database]} opts]
    {:server/jetty (assoc server :handler (ig/ref :handler/ring))
     :handler/ring {:database (ig/ref :database/sql)
                    :auth     (:auth server)}
     :database/sql database}))

The system map is created in code instead of being in the configuration file. This makes refactoring your system simpler as you only need to change this method while leaving the config file (mostly) untouched.^{[5]
[5]}

My current approach to Integrant + Aero config files is mostly inspired by the blog post “Rethinking Config with Aero & Integrant” and Laravel’s configuration. The config file follows a similar structure to Laravel’s config files and contains the app configurations without describing the structure of the system. Previously, I had a key for each Integrant component, which led to the config file being littered with #ig/ref and more difficult to refactor.

Also, if you haven’t already, start a REPL and connect to it from your editor. Run clj -M:dev if your editor doesn’t automatically start a REPL. Next, we’ll implement the init-key and halt-key! methods for each of the components:

src/acme/main.clj

;; src/acme/main.clj
(ns acme.main
  (:require
   ;; ...
   [acme.handler :as handler]
   [acme.util :as util])
   [next.jdbc :as jdbc]
   [ring.adapter.jetty :as jetty]))
;; ...

(defmethod ig/init-key :server/jetty
  [_ opts]
  (let [{:keys [handler port]} opts
        jetty-opts (-> opts (dissoc :handler :auth) (assoc :join? false))
        server     (jetty/run-jetty handler jetty-opts)]
    (println "Server started on port " port)
    server))

(defmethod ig/halt-key! :server/jetty
  [_ server]
  (.stop server))

(defmethod ig/init-key :handler/ring
  [_ opts]
  (handler/handler opts))

(defmethod ig/init-key :database/sql
  [_ opts]
  (let [datasource (jdbc/get-datasource opts)]
    (util/setup-db datasource)
    datasource))

The setup-db function creates the required tables in the database if they don’t exist yet. This works fine for database migrations in small projects like this demo app, but for larger projects, consider using libraries such as Migratus (my preferred library) or Ragtime.

src/acme/util.clj

(ns acme.util 
  (:require
   [next.jdbc :as jdbc]))

(defn setup-db
  [db]
  (jdbc/execute-one!
   db
   ["create table if not exists bookmarks (
       bookmark_id text primary key not null,
       url text not null,
       created_at datetime default (unixepoch()) not null
     )"]))

For the server handler, let’s start with a simple function that returns a “hi world” string.

src/acme/handler.clj

(ns acme.handler
  (:require
   [ring.util.response :as res]))

(defn handler
  [_opts]
  (fn [req]
    (res/response "hi world")))

Now all the components are implemented. We can check if the system is working properly by evaluating (reset) in the user namespace. This will reload your files and restart the system. You should see this message printed in your REPL:

:reloading (acme.util acme.handler acme.main)
Server started on port  8080
:resumed

If we send a request to http://localhost:8080/, we should get “hi world” as the response:

$ curl localhost:8080/
# hi world

Nice! The system is working correctly. In the next section, we’ll implement routing and our business logic handlers.

Routing, Middleware, and Route Handlers

First, let’s set up a ring handler and router using Reitit. We only have one route, the index / route that’ll handle both GET and POST requests.

src/acme/handler.clj

(ns acme.handler
  (:require
   [reitit.ring :as ring]))

(def routes
  [["/" {:get  index-page
         :post index-action}]])

(defn handler
  [opts]
  (ring/ring-handler
   (ring/router routes)
   (ring/routes
    (ring/redirect-trailing-slash-handler)
    (ring/create-resource-handler {:path "/"})
    (ring/create-default-handler))))

We’re including some useful middleware:

redirect-trailing-slash-handler to resolve routes with trailing slashes,
create-resource-handler to serve static files, and
create-default-handler to handle common 40x responses.

Implementing the Middlewares

If you remember the :handler/ring from earlier, you’ll notice that it has two dependencies, database and auth. Currently, they’re inaccessible to our route handlers. To fix this, we can inject these components into the Ring request map using a middleware function.

src/acme/handler.clj

;; ...

(defn components-middleware
  [components]
  (let [{:keys [database auth]} components]
    (fn [handler]
      (fn [req]
        (handler (assoc req
                        :db database
                        :auth auth))))))
;; ...

The components-middleware function takes in a map of components and creates a middleware function that “assocs” each component into the request map.^{[6]
[6]} If you have more components such as a Redis cache or a mail service, you can add them here.

We’ll also need a middleware to handle HTTP basic authentication.^{[7]
[7]} This middleware will check if the username and password from the request map match the values in the auth map injected by components-middleware. If they match, then the request is authenticated and the user can view the site.

src/acme/handler.clj

(ns acme.handler
  (:require
   ;; ...
   [acme.util :as util]
   [ring.util.response :as res]))
;; ...

(defn wrap-basic-auth
  [handler]
  (fn [req]
    (let [{:keys [headers auth]} req
          {:keys [username password]} auth
          authorization (get headers "authorization")
          correct-creds (str "Basic " (util/base64-encode
                                       (format "%s:%s" username password)))]
      (if (and authorization (= correct-creds authorization))
        (handler req)
        (-> (res/response "Access Denied")
            (res/status 401)
            (res/header "WWW-Authenticate" "Basic realm=protected"))))))
;; ...

A nice feature of Clojure is that interop with the host language is easy. The base64-encode function is just a thin wrapper over Java’s Base64.Encoder:

src/acme/util.clj

(ns acme.util
   ;; ...
  (:import java.util.Base64))

(defn base64-encode
  [s]
  (.encodeToString (Base64/getEncoder) (.getBytes s)))

Finally, we need to add them to the router. Since we’ll be handling form requests later, we’ll also bring in Ring’s wrap-params middleware.

src/acme/handler.clj

(ns acme.handler
  (:require
   ;; ...
   [ring.middleware.params :refer [wrap-params]]))
;; ...

(defn handler
  [opts]
  (ring/ring-handler
   ;; ...
   {:middleware [(components-middleware opts)
                 wrap-basic-auth
                 wrap-params]}))

Implementing the Route Handlers

We now have everything we need to implement the route handlers or the business logic of the app. First, we’ll implement the index-page function, which renders a page that:

Shows all of the user’s bookmarks in the database, and
Shows a form that allows the user to insert new bookmarks into the database

src/acme/handler.clj

(ns acme.handler
  (:require
   ;; ...
   [next.jdbc :as jdbc]
   [next.jdbc.sql :as sql]))
;; ...

(defn template
  [bookmarks]
  [:html
   [:head
    [:meta {:charset "utf-8"
            :name    "viewport"
            :content "width=device-width, initial-scale=1.0"}]]
   [:body
    [:h1 "bookmarks"]
    [:form {:method "POST"}
     [:div
      [:label {:for "url"} "url "]
      [:input#url {:name "url"
                   :type "url"
                   :required true
                   :placeholer "https://en.wikipedia.org/"}]]
     [:button "submit"]]
    [:p "your bookmarks:"]
    [:ul
     (if (empty? bookmarks)
       [:li "you don't have any bookmarks"]
       (map
        (fn [{:keys [url]}]
          [:li
           [:a {:href url} url]])
        bookmarks))]]])

(defn index-page
  [req]
  (try
    (let [bookmarks (sql/query (:db req)
                               ["select * from bookmarks"]
                               jdbc/unqualified-snake-kebab-opts)]
      (util/render (template bookmarks)))
    (catch Exception e
      (util/server-error e))))
;; ...

Database queries can sometimes throw exceptions, so it’s good to wrap them in a try-catch block. I’ll also introduce some helper functions:

src/acme/util.clj

(ns acme.util
  (:require
   ;; ...
   [hiccup2.core :as h]
   [ring.util.response :as res])
  (:import java.util.Base64))
;; ...

(defn preprend-doctype
  [s]
  (str "<!doctype html>" s))

(defn render
  [hiccup]
  (-> hiccup h/html str preprend-doctype res/response (res/content-type "text/html")))

(defn server-error
  [e]
  (println "Caught exception: " e)
  (-> (res/response "Internal server error")
      (res/status 500)))

render takes a hiccup form and turns it into a ring response, while server-error takes an exception, logs it, and returns a 500 response.

Next, we’ll implement the index-action function:

src/acme/handler.clj

;; ...

(defn index-action
  [req]
  (try
    (let [{:keys [db form-params]} req
          value (get form-params "url")]
      (sql/insert! db :bookmarks {:bookmark_id (random-uuid) :url value})
      (res/redirect "/" 303))
    (catch Exception e
      (util/server-error e))))
;; ...

This is an implementation of a typical post/redirect/get pattern. We get the value from the URL form field, insert a new row in the database with that value, and redirect back to the index page. Again, we’re using a try-catch block to handle possible exceptions from the database query.

That should be all of the code for the controllers. If you reload your REPL and go to http://localhost:8080, you should see something that looks like this after logging in:

The last thing we need to do is to update the main function to start the system:

src/acme/main.clj

;; ...

(defn -main [& _]
  (-> (read-config) ig/expand ig/init))

Now, you should be able to run the app using clj -M -m acme.main. That’s all the code needed for the app. In the next section, we’ll package the app into a Docker image to deploy to Fly.

Packaging the App

While there are many ways to package a Clojure app, Fly.io specifically requires a Docker image. There are two approaches to doing this:

Build an uberjar and run it using Java in the container, or
Load the source code and run it using Clojure in the container

Both are valid approaches. I prefer the first since its only dependency is the JVM. We’ll use the tools.build library to build the uberjar. Check out the official guide for more information on building Clojure programs. Since it’s a library, to use it, we can add it to our deps.edn file with an alias:

deps.edn

{;; ...
 :aliases
 {;; ...
  :build {:extra-deps {io.github.clojure/tools.build 
                       {:git/tag "v0.10.5" :git/sha "2a21b7a"}}
          :ns-default build}}}

Tools.build expects a build.clj file in the root of the project directory, so we’ll need to create that file. This file contains the instructions to build artefacts, which in our case is a single uberjar. There are many great examples of build.clj files on the web, including from the official documentation. For now, you can copy+paste this file into your project.

build.clj

(ns build
  (:require
   [clojure.tools.build.api :as b]))

(def basis (delay (b/create-basis {:project "deps.edn"})))
(def src-dirs ["src" "resources"])
(def class-dir "target/classes")

(defn uber
  [_]
  (println "Cleaning build directory...")
  (b/delete {:path "target"})

  (println "Copying files...")
  (b/copy-dir {:src-dirs   src-dirs
               :target-dir class-dir})

  (println "Compiling Clojure...")
  (b/compile-clj {:basis      @basis
                  :ns-compile '[acme.main]
                  :class-dir  class-dir})

  (println "Building Uberjar...")
  (b/uber {:basis     @basis
           :class-dir class-dir
           :uber-file "target/standalone.jar"
           :main      'acme.main}))

To build the project, run clj -T:build uber. This will create the uberjar standalone.jar in the target directory. The uber in clj -T:build uber refers to the uber function from build.clj. Since the build system is a Clojure program, you can customise it however you like. If we try to run the uberjar now, we’ll get an error:

# build the uberjar
$ clj -T:build uber
# Cleaning build directory...
# Copying files...
# Compiling Clojure...
# Building Uberjar...

# run the uberjar
$ java -jar target/standalone.jar
# Error: Could not find or load main class acme.main
# Caused by: java.lang.ClassNotFoundException: acme.main

This error occurred because the Main class that is required by Java isn’t built. To fix this, we need to add the :gen-class directive in our main namespace. This will instruct Clojure to create the Main class from the -main function.

src/acme/main.clj

(ns acme.main
  ;; ...
  (:gen-class))
;; ...

If you rebuild the project and run java -jar target/standalone.jar again, it should work perfectly. Now that we have a working build script, we can write the Dockerfile:

Dockerfile

# install additional dependencies here in the base layer
# separate base from build layer so any additional deps installed are cached
FROM clojure:temurin-21-tools-deps-bookworm-slim AS base

FROM base AS build
WORKDIR /opt
COPY . .
RUN clj -T:build uber

FROM eclipse-temurin:21-alpine AS prod
COPY --from=build /opt/target/standalone.jar /
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "standalone.jar"]

It’s a multi-stage Dockerfile. We use the official Clojure Docker image as the layer to build the uberjar. Once it’s built, we copy it to a smaller Docker image that only contains the Java runtime.^{[8]
[8]} By doing this, we get a smaller container image as well as a faster Docker build time because the layers are better cached.

That should be all for packaging the app. We can move on to the deployment now.

Deploying with Fly.io

First things first, you’ll need to install flyctl, Fly’s CLI tool for interacting with their platform. Create a Fly.io account if you haven’t already. Then run fly auth login to authenticate flyctl with your account.

Next, we’ll need to create a new Fly App:

$ fly app create
# ? Choose an app name (leave blank to generate one): 
# automatically selected personal organization: Ryan Martin
# New app created: blue-water-6489

Another way to do this is with the fly launch command, which automates a lot of the app configuration for you. We have some steps to do that are not done by fly launch, so we’ll be configuring the app manually. I also already have a fly.toml file ready that you can straight away copy to your project.

fly.toml

# replace these with your app and region name
# run `fly platform regions` to get a list of regions
app = 'blue-water-6489' 
primary_region = 'sin'

[env]
  DB_DATABASE = "/data/database.db"

[http_service]
  internal_port = 8080
  force_https = true
  auto_stop_machines = "stop"
  auto_start_machines = true
  min_machines_running = 0

[mounts]
  source = "data"
  destination = "/data"
  initial_sie = 1

[[vm]]
  size = "shared-cpu-1x"
  memory = "512mb"
  cpus = 1
  cpu_kind = "shared"

These are mostly the default configuration values with some additions. Under the [env] section, we’re setting the SQLite database location to /data/database.db. The database.db file itself will be stored in a persistent Fly Volume mounted on the /data directory. This is specified under the [mounts] section. Fly Volumes are similar to regular Docker volumes but are designed for Fly’s micro VMs.

We’ll need to set the AUTH_USER and AUTH_PASSWORD environment variables too, but not through the fly.toml file as these are sensitive values. To securely set these credentials with Fly, we can set them as app secrets. They’re stored encrypted and will be automatically injected into the app at boot time.

$ fly secrets set AUTH_USER=hi@ryanmartin.me AUTH_PASSWORD=not-so-secure-password
# Secrets are staged for the first deployment

With this, the configuration is done and we can deploy the app using fly deploy:

$ fly deploy
# ...
# Checking DNS configuration for blue-water-6489.fly.dev
# Visit your newly deployed app at https://blue-water-6489.fly.dev/

The first deployment will take longer since it’s building the Docker image for the first time. Subsequent deployments should be faster due to the cached image layers. You can click on the link to view the deployed app, or you can also run fly open, which will do the same thing. Here’s the app in action:

If you made additional changes to the app or fly.toml, you can redeploy the app using the same command, fly deploy. The app is configured to auto stop/start, which helps to cut costs when there’s not a lot of traffic to the site. If you want to take down the deployment, you’ll need to delete the app itself using fly app destroy <your app name>.

Adding a Production REPL

This is an interesting topic in the Clojure community, with varying opinions on whether or not it’s a good idea. Personally, I find having a REPL connected to the live app helpful, and I often use it for debugging and running queries on the live database.^{[9]
[9]} Since we’re using SQLite, we don’t have a database server we can directly connect to, unlike Postgres or MySQL.

If you’re brave, you can even restart the app directly without redeploying from the REPL. You can easily go wrong with it, which is why some prefer not to use it.

For this project, we’re gonna add a socket REPL. It’s very simple to add (you just need to add a JVM option) and it doesn’t require additional dependencies like nREPL. Let’s update the Dockerfile:

Dockerfile

# ...
EXPOSE 7888
ENTRYPOINT ["java", "-Dclojure.server.repl={:port 7888 :accept clojure.core.server/repl}", "-jar", "standalone.jar"]

The socket REPL will be listening on port 7888. If we redeploy the app now, the REPL will be started, but we won’t be able to connect to it. That’s because we haven’t exposed the service through Fly proxy. We can do this by adding the socket REPL as a service in the [services] section in fly.toml.

However, doing this will also expose the REPL port to the public. This means that anyone can connect to your REPL and possibly mess with your app. Instead, what we want to do is to configure the socket REPL as a private service.

By default, all Fly apps in your organisation live in the same private network. This private network, called 6PN, connects the apps in your organisation through WireGuard tunnels (a VPN) using IPv6. Fly private services aren’t exposed to the public internet but can be reached from this private network. We can then use Wireguard to connect to this private network to reach our socket REPL.

Fly VMs are also configured with the hostname fly-local-6pn, which maps to its 6PN address. This is analogous to localhost, which points to your loopback address 127.0.0.1. To expose a service to 6PN, all we have to do is bind or serve it to fly-local-6pn instead of the usual 0.0.0.0. We have to update the socket REPL options to:

Dockerfile

# ...
ENTRYPOINT ["java", "-Dclojure.server.repl={:port 7888,:address \"fly-local-6pn\",:accept clojure.core.server/repl}", "-jar", "standalone.jar"]

After redeploying, we can use the fly proxy command to forward the port from the remote server to our local machine.^{[10]
[10]}

$ fly proxy 7888:7888
# Proxying local port 7888 to remote [blue-water-6489.internal]:7888

In another shell, run:

$ rlwrap nc localhost 7888
# user=>

Now we have a REPL connected to the production app! rlwrap is used for readline functionality, e.g. up/down arrow keys, vi bindings. Of course, you can also connect to it from your editor.

Deploy with GitHub Actions

If you’re using GitHub, we can also set up automatic deployments on pushes/PRs with GitHub Actions. All you need is to create the workflow file:

.github/workflows/fly.yaml

name: Fly Deploy
on:
  push:
    branches:
      - main
  workflow_dispatch:

jobs:
  deploy:
    name: Deploy app
    runs-on: ubuntu-latest
    concurrency: deploy-group
    steps:
      - uses: actions/checkout@v4
      - uses: superfly/flyctl-actions/setup-flyctl@master
      - run: flyctl deploy --remote-only
        env:
          FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}

To get this to work, you’ll need to create a deploy token from your app’s dashboard. Then, in your GitHub repo, create a new repository secret called FLY_API_TOKEN with the value of your deploy token. Now, whenever you push to the main branch, this workflow will automatically run and deploy your app. You can also manually run the workflow from GitHub because of the workflow_dispatch option.

End

As always, all the code is available on GitHub. Originally, this post was just about deploying to Fly.io, but along the way, I kept adding on more stuff until it essentially became my version of the user manager example app. Anyway, hope this post provided a good view into web development with Clojure. As a bonus, here are some additional resources on deploying Clojure apps:

The way Fly.io works under the hood is pretty clever. Instead of running the container image with a runtime like Docker, the image is unpacked and “loaded” into a VM. See this video explanation for more details. ↩︎
If you’re interested in learning Clojure, my recommendation is to follow the official getting started guide and join the Clojurians Slack. Also, read through this list of introductory resources. ↩︎
Kit was a big influence on me when I first started learning web development in Clojure. I never used it directly, but I did use their library choices and project structure as a base for my own projects. ↩︎
There’s no “Rails” for the Clojure ecosystem (yet?). The prevailing opinion is to build your own “framework” by composing different libraries together. Most of these libraries are stable and are already used in production by big companies, so don’t let this discourage you from doing web development in Clojure! ↩︎
There might be some keys that you add or remove, but the structure of the config file stays the same. ↩︎
“assoc” (associate) is a Clojure slang that means to add or update a key-value pair in a map. ↩︎
For more details on how basic authentication works, check out the specification. ↩︎
Here’s a cool resource I found when researching Java Dockerfiles: WhichJDK. It provides a comprehensive comparison of the different JDKs available and recommendations on which one you should use. ↩︎
Another (non-technically important) argument for live/production REPLs is just because it’s cool. Ever since I read the story about NASA’s programmers debugging a spacecraft through a live REPL, I’ve always wanted to try it at least once. ↩︎
If you encounter errors related to WireGuard when running fly proxy, you can run fly doctor, which will hopefully detect issues with your local setup and also suggest fixes for them. ↩︎

How to Implement Automatic CLI Login

2026-04-04T12:25:25Z

I’ve recently had the chance to work on an “automatic CLI login” feature for a CLI tool at my day job. What I mean by automatic CLI login here is when a CLI tool automatically logs in a user, so it can perform actions on behalf of the user.

It usually works like this. When you type a command like cli login in your terminal, a login page will open in your web browser. After you log in, the page will show a message like “Your CLI is connected now”. When you go back to the terminal, it will say that it’s “connected” and you can now perform commands that require authentication.

The connection here is often in the form of an access token or an API key. This allows the CLI to stay logged in and perform authenticated requests to a remote server.

You’ve probably experienced this type of interaction before. Many modern CLI tools have some sort of command to allow you to log in from the command line. Examples include Supabase CLI, GitHub CLI, Graphite CLI, and Fly.io’s flyctl. It’s a UX pattern that I like, as it hides a lot of complexity from the user.

There are probably many different ways to implement this, each with different trade-offs. In this post, I’ll write about just one way of doing this, which is also the same approach used by Supabase and Fly.io.^{[1]
[1]} It works for my particular use case and does not require refactoring my existing authentication logic.

How it Works

At first glance, it looks like there is a two-way communication between the CLI tool and the browser. When you log in to the browser, you’ll receive an access token. Then the browser sends it to the CLI in some way?. Maybe the CLI starts up an HTTP server with an endpoint to receive the access token from the web page, like the diagram below?

This probably works, though I haven’t actually tested it yet. This isn’t the approach that I implemented, but it could be a good area for future experimentation.

How it works in Supabase CLI and flyctl is that it uses a “one-way” communication scheme instead of a two-way one.^{[2]
[2]} Instead of the remote server sending the access token after the user has logged in, the CLI will keep polling for the access token in the background. At a high level, the flow should look like this diagram:

The CLI will provide a session ID and ask the server to create a “CLI login session” when the user logs in successfully. An access token will be created for this specific session. Then, the CLI will request this access token from the server and store it for future requests.

Since this approach uses regular HTTP polling, it’s a potential source of DDOS attacks. In a perfect world, a user needs to log in to the CLI only once after they install the tool. The server is unlikely to get too many login requests at a time. In the real world, attackers can keep sending login requests to the server to slow or take it down. We can mitigate this by rate-limiting the endpoints and blocking malicious IP addresses, but it’s not a perfect defence.

This is a trade-off between security and code simplicity. HTTP polling is much simpler to implement than other solutions like WebSockets or SSE. This might not work for you depending on your threat model. But hey, if it works for Supabase, which has around 450,000 users,^{[3]
[3]} it’ll probably work for me too.

Now that we know how it works, let’s get into the code implementation. I’ll build a web server and a CLI tool for a fictional developer tools company, ACME Corp. The CLI tool will be called acme.

Project Setup

I’ll use SvelteKit for this project, but the concepts apply to any web framework. I’ll try to avoid using any Svelte-specific features or terminologies so that you can apply this to any language or web framework you like.

# create a new sveltekit project & install dependencies
$ pnpm create svelte@latest acme
$ cd acme
$ pnpm install

I’ve heard good things about Drizzle ORM, so I’ll also use it as the ORM for this project. The database will be SQLite using the better-sqlite3 library. We’ll need to install these too.

$ pnpm add drizzle-orm better-sqlite3
$ pnpm add -D drizzle-kit

Drizzle requires some setup steps before we can use it. First, we’ll create a Drizzle client:

src/lib/db/db.js

import Database from 'better-sqlite3'
import { drizzle } from 'drizzle-orm/better-sqlite3'

const sqlite = new Database('data/database.db')

export const db = drizzle(sqlite)

And define the database schema:

src/lib/db/schema.js

import { sqliteTable, text } from 'drizzle-orm/sqlite-core'

export const users = sqliteTable('users', {
  userId: text('user_id')
    .primaryKey()
    .$defaultFn(() => crypto.randomUUID()),
  email: text('email').unique().notNull(),
  password: text('password').notNull(),
})

export const sessions = sqliteTable('sessions', {
  sessionId: text('session_id')
    .primaryKey()
    .$defaultFn(() => crypto.randomUUID()),
  userId: text('user_id')
    .notNull()
    .references(() => users.userId),
})

The users table will store the user details, and the sessions table will store the authentication sessions. The app will use traditional session-based authentication.

Then, we need to update the config file:

drizzle.config.js

import { defineConfig } from 'drizzle-kit'

export default defineConfig({
  dialect: 'sqlite',
  schema: './src/lib/db/schema.js',
  dbCredentials: {
    url: './data/database.db',
  },
})

To apply the changes to the database, run:

# create database directory
$ mkdir data

# apply changes to database
$ pnpm drizzle-kit push

This will create the data/database.db SQLite database with our schema. You should only use drizzle-kit push for prototyping or local development. If you need database migrations, use drizzle-kit generate and drizzle-kit migrate. See the official documentation for more info.

Finally, start the development server by running pnpm dev. The server should now be running at http://localhost:5173.

A Basic Authentication System

As the focus of this post is on automatic CLI login, we won’t be doing anything fancy with user authentication. It’ll use a database-backed session-based authentication. I’ll also skip hashing the password and just store it in plaintext. You should never store passwords like this in a production system. Either use a third-party authentication service or follow the OWASP guide to password storage.

First, let’s build a login page. For this app, the login page will serve as both the login and the signup page.^{[4]
[4]}

src/routes/login/+page.svelte

<h1>log in</h1>
<form method="POST">
  <div>
    <label for="email">email</label>
    <input required id="email" name="email" type="email" placeholder="john.doe@email.com" />
  </div>
  <div>
    <label for="password">password</label>
    <input required id="password" name="password" type="password" />
  </div>
  <button>log in</button>
</form>

The login form sends a POST request with the user’s email and password to this handler:

src/routes/login/+page.server.js

import { db } from '$lib/db/db'
import { sessions, users } from '$lib/db/schema'
import { fail, redirect } from '@sveltejs/kit'
import { eq } from 'drizzle-orm'

/** @type {import('./$types').Actions} */
export const actions = {
  default: async ({ request, cookies, url }) => {
    const form = await request.formData()
    const email = form.get('email')
    const password = form.get('password')

    if (!email || !password) {
      return fail(400, { message: 'missing email or password' })
    }

    const userData = await db.select().from(users).where(eq(users.email, email))
    const user = userData[0]
    const next = url.searchParams.get('next') ?? '/'

    // register user if not exists & create session
    if (!user) {
      const newUserData = await db.insert(users).values({ email, password }).returning()
      const session = await db
        .insert(sessions)
        .values({ userId: newUserData[0].userId })
        .returning()
      cookies.set('session', session[0].sessionId, { path: '/' })
      redirect(303, next)
    }

    // else login
    if (email === user.email && password === user.password) {
      const session = await db.insert(sessions).values({ userId: user.userId }).returning()
      cookies.set('session', session[0].sessionId, { path: '/' })
      redirect(303, next)
    }

    return fail(400, { message: 'invalid email or password' })
  },
}

The exported actions map is SvelteKit’s version of form action handlers. You can just focus on the function in the default action.

If the user record doesn’t exist in the database, this function will create a new user and session in the database, set the required cookie in the response header, and redirect the user to the value in the next search param. If next is not specified, it will redirect to the home page instead. If the user is already registered, it will directly create the session, set the cookie, and redirect the user.

We’ll also add route protection to the home page. Unauthenticated requests to the home page will be redirected to the login page.

src/routes/+page.server.js

import { db } from '$lib/db/db'
import { sessions, users } from '$lib/db/schema'
import { redirect } from '@sveltejs/kit'
import { eq } from 'drizzle-orm'

/** @type {import('./$types').PageServerLoad} */
export async function load({ cookies }) {
  const sessionCookie = cookies.get('session')
  if (!sessionCookie) redirect(302, '/login')

  const sessionData = await db.select().from(sessions).where(eq(sessions.sessionId, sessionCookie))
  const session = sessionData[0]
  if (!session) redirect(302, '/login')

  const user = await db.select().from(users).where(eq(users.userId, session.userId))

  return {
    user: user[0],
  }
}

The load function is a function that runs before the page is rendered. It’s similar to page/route controllers in other web frameworks like Django and Ruby on Rails. If you open http://localhost:5173 in your browser, you should be redirected to the login page.

Now let’s get to the main focus of this post. We’ll need to add new tables in our database schema to support this feature:

src/lib/db/schema.js

export const accessTokens = sqliteTable('access_tokens', {
  tokenId: text('token_id')
    .primaryKey()
    .$defaultFn(() => crypto.randomUUID()),
  userId: text('user_id')
    .notNull()
    .references(() => users.userId),
})

export const cliSessions = sqliteTable('cli_sessions', {
  cliSessionId: text('cli_session_id')
    .primaryKey()
    .$defaultFn(() => crypto.randomUUID()),
  tokenId: text('token_id')
    .notNull()
    .references(() => accessTokens.tokenId),
})

The access_tokens table will store the users’ access tokens, while the cli_sessions table will store the “login sessions” triggered by the CLI tool. When a user runs the command acme login, it will trigger a login session and an access token to be created on the server.

Run pnpm drizzle-kit push again to apply the changes to the database. If you encounter a SQLite read-only error in SvelteKit, just restart the dev server.

We’ll also need to add new endpoints, /cli/login and cli/login/:session. Before we implement these endpoints, we’ll need a page to show the “Your CLI is now connected” message once the user has logged in.

src/routes/cli/login/+page.svelte

<p>Your CLI is connected now. You can close this tab.</p>

Now, here’s the code for the /cli/login route handler:

src/routes/cli/login/+page.server.js

import { db } from '$lib/db/db'
import { accessTokens, cliSessions, sessions, users } from '$lib/db/schema'
import { error, redirect } from '@sveltejs/kit'
import { eq } from 'drizzle-orm'

/** @type {import('./$types').PageServerLoad} */
export async function load({ cookies, url }) {
  const sessionCookie = cookies.get('session')
  const next = encodeURIComponent(url.pathname + url.search)

  if (!sessionCookie) redirect(302, `/login?next=${next}`)

  const sessionData = await db.select().from(sessions).where(eq(sessions.sessionId, sessionCookie))
  const session = sessionData[0]

  // if there is no valid session, redirect to login
  if (!session) redirect(302, `/login?next=${next}`)

  // get user from session id
  const userData = await db
    .select()
    .from(users)
    .leftJoin(sessions, eq(users.userId, sessions.userId))
    .where(eq(sessions.sessionId, sessionCookie))
  const user = userData[0].users

  const params = url.searchParams
  const cliSessionId = params.get('session')

  if (!cliSessionId) {
    return error(400, { message: 'missing search params' })
  }

  const cliSessionData = await db
    .select()
    .from(cliSessions)
    .where(eq(cliSessions.cliSessionId, cliSessionId))
  const cliSession = cliSessionData[0]

  // create cli session & access token if it doesn't exist
  if (!cliSession) {
    const token = await db.insert(accessTokens).values({ userId: user.userId }).returning()
    await db.insert(cliSessions).values({ cliSessionId, tokenId: token[0].tokenId })
  }
}

This function first checks if the user is logged in using the cookie. The mechanism is almost the same as on the home page, with the only difference being that we’re setting the next search parameter to point back to this URL. This will redirect the user back to /cli/login after they have logged in.

If the user is already logged in, a CLI session and an access token will be inserted into the database. Then, it’ll render the success message page from before. Note, in SvelteKit, after the load function in +page.server.js is called, the page in +page.svelte will be rendered. That’s why we don’t have to tell this function to render the page.

Next is the /cli/login/:session route handler:

src/routes/cli/login/[session]/+server.js

import { db } from '$lib/db/db'
import { cliSessions } from '$lib/db/schema'
import { error } from '@sveltejs/kit'
import { eq } from 'drizzle-orm'

/** @type {import('./$types').RequestHandler} */
export async function GET({ params }) {
  const { session } = params

  const cliSessionData = await db
    .select()
    .from(cliSessions)
    .where(eq(cliSessions.cliSessionId, session))
  const cliSession = cliSessionData[0]

  if (!cliSession) return error(404)

  return Response.json({ token: cliSession.tokenId })
}

This function checks if the session in the URL’s path parameter is a valid CLI session in the database. If it is, it’ll return the access token in the response body. This token will then be stored by the CLI tool and used in subsequent requests.

Writing the CLI Tool

Our acme CLI tool will just be a single Node.js script. It’ll have only one command, acme login. As mentioned in the previous section, the CLI will request a login session and poll for the access token:

acme.js

import crypto from 'crypto'
import { exec } from 'child_process'

const [_node, _acme, argv1] = process.argv

// exit if program isn't run with the login command
if (!argv1 || argv1 !== 'login') {
  console.log('usage: node acme.js login')
  process.exit(1)
}

const host = 'http://localhost:5173'
const session = crypto.randomUUID()

// open the URL in the default browser
const url = `${host}/cli/login?session=${session}`
exec(`xdg-open '${url}'`, () => console.log(`Here's your login link:\n\n${url}\n`))

// poll for the access token
const interval = setInterval(() => {
  fetch(`${host}/cli/login/${session}`)
    .then((res) => (res.ok ? res.json() : null))
    .then((data) => {
      if (!data) return

      console.log(data)

      clearInterval(interval)
    })
}, 1000)

The acme CLI will generate a random UUID for the session ID and include it as a search parameter in a URL to /cli/login. Then, it’ll call exec, which will run an OS process. In my case, it uses xdg-open, a Linux program that opens a file or URL with the user’s default application. If you’re using a Mac, replace this with open, and if you’re on Windows, replace it with start.

If you’re not already logged in, you will be redirected to the login page. Once you’ve logged in and seen the success message, the CLI should have already received the access token from the /cli/login/:session endpoint. If you run this script, you should get something like this:

Nice! We now have a working automatic CLI login system. Note, I cleared the cookies in the video above so that I can log in again. If you didn’t do this, you’ll go straight to the success message as you’re already logged in. I also created a shell alias to make the script look like a “proper” CLI tool:

# create the alias
$ alias acme='node acme.js'

# then you can run
$ acme login

Security Considerations

I’ve mentioned DDOS attacks before as one of the vulnerabilities of this login flow. I’ll mention a few more and also explain the ways to mitigate them. This isn’t a comprehensive list of security vulnerabilities though. You might need to conduct security tests or assessments depending on your threat model.

One type of attack we’re vulnerable to is the man-in-the-middle attack. A man-in-the-middle attack can happen when we try to log in over an insecure connection. An attacker can listen to your network traffic and steal your access tokens. Thankfully, this kind of attack is easy to mitigate by using a secure connection like HTTPS. HTTPS encrypts your network traffic, preventing a third party from reading your network activity.

Another kind of attack is a brute force attack. Since the endpoints to poll for the access tokens don’t require authentication, attackers can brute force UUID values until they get an access token. This can be mitigated by expiring the CLI sessions and applying the same strategies as with DDOS attacks.

Both of these attacks have one thing in common. If an attacker gets a hold of your access token, you’re hacked. We can increase the security of our app by encrypting the access tokens. This way, even if attackers obtain an encrypted access token, they won’t be able to decrypt it.

Encrypting the tokens presents some challenges. How do we ensure that only the CLI can decrypt the token? What encryption scheme should be used? What if an attacker obtained the decryption key?

ECDH Key Exchange & Encryption

One possible solution is to use ECDH key exchange. ECDH (Elliptic Curve Diffie–Hellman Key Exchange) is a method that allows two parties to establish a shared secret over an insecure channel. ECDH is based on ECC (elliptic curve cryptography), a modern family of public-key cryptography and is considered the modern successor of RSA.^{[5]
[5]}

Here’s a high-level explanation of how ECDH works:^{[6]
[6]}

Alice generates a random ECC key pair: alicePriv & alicePub.
Bob generates a random ECC key pair: bobPriv & bobPub.
Alice and Bob exchange their public keys over an insecure channel like the Internet.
Alice calculates the secret = alicePriv * bobPub.
Bob calculates the secret = bobPriv * alicePub.
Alice and Bob now have the same shared secret.

This works because, in the ECDH algorithm, alicePriv * bobPub is equal to bobPriv * alicePub.

Unlike RSA, ECC doesn’t directly provide a way to do asymmetric encryption. Instead, we have to devise our own hybrid encryption scheme using a combination of ECDH and symmetric encryption. We can use any symmetric encryption algorithms we want like AES or ChaCha20-Poly1305. Here’s how the hybrid encryption scheme works:

Alice generates a random ECC key pair: alicePriv & alicePub.
Bob generates a random ECC key pair: bobPriv & bobPub.
Alice and Bob exchange their public keys over the internet.
Alice calculates the secret = alicePriv * bobPub.
Alice encrypts a plaintext using the secret as the symmetric encryption key.
Alice sends the ciphertext to Bob over the Internet.
Bob calculates the secret = bobPriv * alicePub.
Bob decrypts the ciphertext using the secret as the decryption key.

While the underlying maths might be complex, the encryption scheme is straightforward. Now, let’s implement this in our code. We’ll use the WebCrypto API as the cryptography library.

Securing the Endpoints

First, we need to update our cli_sessions table definition to store the CLI’s public key.

src/lib/db/schema.js

// ...

export const cliSessions = sqliteTable('cli_sessions', {
  cliSessionId: text('cli_session_id')
    .primaryKey()
    .$defaultFn(() => crypto.randomUUID()),
  tokenId: text('token_id')
    .notNull()
    .references(() => accessTokens.tokenId),
  pubKey: text('pub_key').notNull(), // <-- new column
})

Run pnpm drizzle-kit push again. This time, you should be prompted to delete some records. As we’re just testing, we can always wipe the DB clean to start over.

Next, we’ll update the /cli/login endpoint to get the public key from the search parameter and insert it into the database:

src/routes/cli/login/+page.server.js

// ...

/** @type {import('./$types').PageServerLoad} */
export async function load({ cookies, url }) {
  // ...

  const params = url.searchParams
  const cliSessionId = params.get('session')
  const pubKey = params.get('pub_key')

  if (!cliSessionId || !pubKey) {
    return error(400, { message: 'missing search params' })
  }

  const token = await db.insert(accessTokens).values({ userId: user.userId }).returning()
  await db.insert(cliSessions).values({ cliSessionId, pubKey, tokenId: token[0].tokenId })
}

As for the /cli/login/:session endpoint, instead of returning the access token directly, we’ll need to encrypt it first. We’ll refactor this endpoint’s handler function to implement the encryption:

src/routes/cli/login/[session]/+server.js

import { bytesToHex, hexToBytes } from '$lib/utils'
// ...

/** @type {import('./$types').RequestHandler} */
export async function GET({ params }) {
  // ...

  if (!cliSession) return error(404)

  // generate ECC key pair
  const { privateKey, publicKey } = await crypto.subtle.generateKey(
    { name: 'ECDH', namedCurve: 'P-256' },
    true,
    ['deriveKey'],
  )
  const publicKeyBuffer = await crypto.subtle.exportKey('raw', publicKey)
  const remotePublicKey = await crypto.subtle.importKey(
    'raw',
    hexToBytes(cliSession.pubKey),
    { name: 'ECDH', namedCurve: 'P-256' },
    false,
    [],
  )

  // encrypt the access token using aes-gcm-256 and the secret key
  const enc = new TextEncoder()
  const secret = await crypto.subtle.deriveKey(
    { name: 'ECDH', public: remotePublicKey },
    privateKey,
    { name: 'AES-GCM', length: 256 },
    false,
    ['encrypt', 'decrypt'],
  )
  // iv in aes-gcm-256 must be 12 bytes long
  const iv = crypto.getRandomValues(new Uint8Array(12))
  const encrypted = await crypto.subtle.encrypt(
    { name: 'AES-GCM', iv: iv },
    secret,
    enc.encode(cliSession.tokenId),
  )

  return Response.json({
    token: bytesToHex(encrypted),
    pubKey: bytesToHex(publicKeyBuffer),
    iv: bytesToHex(iv),
  })
}

Now, this function will generate a random ECC key pair to calculate the secret using the public key provided by the CLI. I’m using the NIST P-256 curve as the basis of the ECC key pair, as the WebCrypto API only supports NIST curves at the moment.^{[7]
[7]}

This secret is then used as the key for the symmetric encryption algorithm. I use AES-GCM here because it’s also directly supported in WebCrypto. The secret here is 32 bytes long, which means we’ll use a 256-bit key. AES-GCM encryption with a 256-bit key requires a nonce value (iv) of 12 bytes, which we also need to send to the user along with the server’s public key.

The private and public keys are byte arrays here, so we also need some helper functions to decode byte arrays from/to hex strings:

src/lib/utils.js

export const bytesToHex = (bytes) => {
  return [...new Uint8Array(bytes)].map((x) => x.toString(16).padStart(2, '0')).join('')
}

export const hexToBytes = (hex) => {
  const hexBytes = hex.match(/.{1,2}/g) ?? []
  return Uint8Array.from(hexBytes.map((byte) => parseInt(byte, 16)))
}

Finally, we have to update the CLI code to decrypt the access token:

acme.js

import { bytesToHex, hexToBytes } from './src/lib/utils.js'
// ...

const randomKeyPair = async () => {
  return await crypto.subtle.generateKey({ name: 'ECDH', namedCurve: 'P-256' }, true, ['deriveKey'])
}

const decrypt = async (privateKey, publicKey, iv, encrypted) => {
  const remotePublicKey = await crypto.subtle.importKey(
    'raw',
    publicKey,
    { name: 'ECDH', namedCurve: 'P-256' },
    false,
    [],
  )
  const secret = await crypto.subtle.deriveKey(
    { name: 'ECDH', public: remotePublicKey },
    privateKey,
    { name: 'AES-GCM', length: 256 },
    false,
    ['encrypt', 'decrypt'],
  )
  const decrypted = await crypto.subtle.decrypt({ name: 'AES-GCM', iv: iv }, secret, encrypted)
  const dec = new TextDecoder()
  return dec.decode(decrypted)
}

const login = async () => {
  const host = 'http://localhost:5173'
  const session = crypto.randomUUID()
  const { privateKey, publicKey } = await randomKeyPair()
  const publicKeyBuffer = await crypto.subtle.exportKey('raw', publicKey)

  const url = `${host}/cli/login?session=${session}&pub_key=${bytesToHex(publicKeyBuffer)}`
  exec(`xdg-open '${url}'`, () => console.log(`Here's your login link:\n\n${url}\n`))

  const interval = setInterval(() => {
    fetch(`${host}/cli/login/${session}`)
      .then((res) => (res.ok ? res.json() : null))
      .then((data) => {
        if (!data) return

        const { token, pubKey, iv } = data

        decrypt(privateKey, hexToBytes(pubKey), hexToBytes(iv), hexToBytes(token)).then(
          (accessToken) => console.log('Your access token:', accessToken),
        )

        clearInterval(interval)
      })
  }, 1000)
}

const [_node, _acme, argv1] = process.argv

if (!argv1 || argv1 !== 'login') {
  console.log('usage: node acme.js login')
  process.exit(1)
}

login()

The decrypt function follows the same steps as the encryption process, just reversed. And that’s it. When we run the script now, we should see that the login flow still works the same:

You can also see that the decrypted token is the same as the token in the database.

What’s Next?

You can find the code for this project on GitHub. I’ve only covered the minimum required to implement automatic CLI login with encryption. There’s always room for improvement. E.g., you can implement additional features such as CLI session expiry, rate-limiting, token expiry, and more. Good luck!

I learned about how this works mostly by digging into the Supabase CLI code, so I’m biased toward this approach. You can also see Supabase’s blog post about this and Fly’s guide on implementing this in Laravel. ↩︎
I call it one-way communication here, but what I meant is that only one participant, this being the CLI, is making the requests. Contrast this to the other approach, where both participants are sending requests to each other. ↩︎
450,000 is the number of registered developers on their website. It’s a vanity metric, but it’s still an impressive number. ↩︎
I made this choice just to keep the code as small as possible. You shouldn’t do this in a real app unless you have a really specific reason. This UX is unfamiliar and can confuse users. ↩︎
Check out the section on ECDH from the Practical Cryptography for Developers book for more details on how the algorithm works. It’s also a great resource for learning about cryptography in general without going too deep into the maths. ↩︎
This post assumes you know about RSA and basic cryptography concepts. It’s fine if you don’t, but the concepts explained will make more sense if you do. Here’s a good introduction to cryptography article that explains the basic concepts, as well as resources if you want to learn further. ↩︎
If I’m using a third-party library like @noble/curves, I’ll use X25519 for ECDH. I’m not an expert in ECC, and most of my algorithm/curve choices are based on this blog post on choosing an elliptic curve signature algorithm. Although it’s about digital signatures (ECDSA), I still find it helpful for ECDH. ↩︎

Advent of Code 2024 in Zig

2026-04-04T12:25:25Z

This post is about ~~six~~ seven months late, but here are my takeaways from Advent of Code 2024. It was my second time participating, and this time I actually managed to complete it.^{[1]
[1]} My goal was to learn a new language, Zig, and to improve my DSA and problem-solving skills.

If you’re not familiar, Advent of Code is an annual programming challenge that runs every December. A new puzzle is released each day from December 1st to the 25th. There’s also a global leaderboard where people (and AI) race to get the fastest solves, but I personally don’t compete in it, mostly because I want to do it at my own pace.

I went with Zig because I have been curious about it for a while, mainly because of its promise of being a better C and because TigerBeetle (one of the coolest databases now) is written in it. Learning Zig felt like a good way to get back into systems programming, something I’ve been wanting to do after a couple of chaotic years of web development.

This post is mostly about my setup, results, and the things I learned from solving the puzzles. If you’re more interested in my solutions, I’ve also uploaded my code and solution write-ups to my GitHub repository.

Project Setup

There were several Advent of Code templates in Zig that I looked at as a reference for my development setup, but none of them really clicked with me. I ended up just running my solutions directly using zig run for the whole event. It wasn’t until after the event ended that I properly learned Zig’s build system and reorganised my project.

Here’s what the project structure looks like now:

.
├── src
│   ├── days
│   │   ├── data
│   │   │   ├── day01.txt
│   │   │   ├── day02.txt
│   │   │   └── ...
│   │   ├── day01.zig
│   │   ├── day02.zig
│   │   └── ...
│   ├── bench.zig
│   └── run.zig
└── build.zig

The project is powered by build.zig, which defines several commands:

Build
- zig build - Builds all of the binaries for all optimisation modes.
Run
- zig build run - Runs all solutions sequentially.
- zig build run -Day=XX - Runs the solution of the specified day only.
Benchmark
- zig build bench - Runs all benchmarks sequentially.
- zig build bench -Day=XX - Runs the benchmark of the specified day only.
Test
- zig build test - Runs all tests sequentially.
- zig build test -Day=XX - Runs the tests of the specified day only.

You can also pass the optimisation mode that you want to any of the commands above with the -Doptimize flag.

Under the hood, build.zig compiles src/run.zig when you call zig build run, and src/bench.zig when you call zig build bench. These files are templates that import the solution for a specific day from src/days/dayXX.zig. For example, here’s what src/run.zig looks like:

src/run.zig

const std = @import("std");
const puzzle = @import("day"); // Injected by build.zig

pub fn main() !void {
    var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator);
    defer arena.deinit();
    const allocator = arena.allocator();

    std.debug.print("{s}\n", .{puzzle.title});
    _ = try puzzle.run(allocator, true);
    std.debug.print("\n", .{});
}

The day module imported is an anonymous import dynamically injected by build.zig during compilation. This allows a single run.zig or bench.zig to be reused for all solutions. This avoids repeating boilerplate code in the solution files. Here’s a simplified version of my build.zig file that shows how this works:

build.zig

const std = @import("std");

pub fn build(b: *std.Build) void {
    const target = b.standardTargetOptions(.{});
    const optimize = b.standardOptimizeOption(.{});

    const run_all = b.step("run", "Run all days");
    const day_option = b.option(usize, "ay", ""); // The `-Day` option

    // Generate build targets for all 25 days.
    for (1..26) |day| {
        const day_zig_file = b.path(b.fmt("src/days/day{d:0>2}.zig", .{day}));

        // Create an executable for running this specific day.
        const run_exe = b.addExecutable(.{
            .name = b.fmt("run-day{d:0>2}", .{day}),
            .root_source_file = b.path("src/run.zig"),
            .target = target,
            .optimize = optimize,
        });

        // Inject the day-specific solution file as the anonymous module `day`.
        run_exe.root_module.addAnonymousImport("day", .{ .root_source_file = day_zig_file });

        // Install the executable so it can be run.
        b.installArtifact(run_exe);

        // ...
    }
}

My actual build.zig has some extra code that builds the binaries for all optimisation modes.

This setup is pretty barebones. I’ve seen other templates do cool things like scaffold files, download puzzle inputs, and even submit answers automatically. Since I wrote my build.zig after the event ended, I didn’t get to use it while solving the puzzles. I might add these features to it if I decided to do Advent of Code again this year with Zig.

Self-Imposed Constraints

While there are no rules to Advent of Code itself, to make things a little more interesting, I set a few constraints and rules for myself:

The code must be readable. By “readable”, I mean the code should be straightforward and easy to follow. No unnecessary abstractions. I should be able to come back to the code months later and still understand (most of) it.
Solutions must be a single file. No external dependencies. No shared utilities module. Everything needed to solve the puzzle should be visible in that one solution file.
2.
The original constraint was that each solution must run in under one second. As it turned out, the code was faster than I expected, so I increased the difficulty. ↩︎

The total runtime must be under one second.^{[2]
[2]} All solutions, when run sequentially, should finish in under one second. I want to improve my performance engineering skills.
Parts should be solved separately. This means: (1) no solving both parts simultaneously, and (2) no doing extra work in part one that makes part two faster. The aim of this is to get a clear idea of how long each part takes on its own.
No concurrency or parallelism. Solutions must run sequentially on a single thread. This keeps the focus on the efficiency of the algorithm. I can’t speed up slow solutions by using multiple CPU cores.
No ChatGPT. No Claude. No AI help. I want to train myself, not the LLM. I can look at other people’s solutions, but only after I have given my best effort at solving the problem.
Follow the constraints of the input file. The solution doesn’t have to work for all possible scenarios, but it should work for all valid inputs. If the input file only contains 8-bit unsigned integers, the solution doesn’t have to handle larger integer types.
Hardcoding is allowed. For example: size of the input, number of rows and columns, etc. Since the input is known at compile-time, we can skip runtime parsing and just embed it into the program using Zig’s @embedFile.

Most of these constraints are designed to push me to write clearer, more performant code. I also wanted my code to look like it was taken straight from TigerBeetle’s codebase (minus the assertions).^{[3]
[3]} Lastly, I just thought it would make the experience more fun.

Favourite Puzzles

From all of the puzzles, here are my top 3 favourites:

Day 6: Guard Gallivant - This is my slowest day (in benchmarks), but also the one I learned the most from. Some of these learnings include: using vectors to represent directions, padding 2D grids, metadata packing, system endianness, etc.
Day 17: Chronospatial Computer - I love reverse engineering puzzles. I used to do a lot of these in CTFs during my university days. The best thing I learned from this day is the realisation that we can use different integer bases to optimise data representation. This helped improve my runtimes in the later days 22 and 23.
Day 21: Keypad Conundrum - This one was fun. My gut told me that it can be solved greedily by always choosing the best move. It was right. Though I did have to scroll Reddit for a bit to figure out the step I was missing, which was that you have to visit the farthest keypads first. This is also my longest solution file (almost 400 lines) because I hardcoded the best-moves table.

Honourable mention:

Day 24: Crossed Wires - Another reverse engineering puzzle. Confession: I didn’t solve this myself during the event. After 23 brutal days, my brain was too tired, so I copied a random Python solution from Reddit. When I retried it later, it turned out to be pretty fun. I still couldn’t find a solution I was satisfied with though.

Programming Patterns and Zig Tricks

During the event, I learned a lot about Zig and performance, and also developed some personal coding conventions. Some of these are Zig-specific, but most are universal and can be applied across languages. This section covers general programming and Zig patterns I found useful. The next section will focus on performance-related tips.

Comptime

Zig’s flagship feature, comptime, is surprisingly useful. I knew Zig uses it for generics and that people do clever metaprogramming with it, but I didn’t expect to be using it so often myself.

My main use for comptime was to generate puzzle-specific types. All my solution files follow the same structure, with a DayXX function that takes some parameters (usually the input length) and returns a puzzle-specific type, e.g.:

src/days/day01.zig

fn Day01(comptime length: usize) type {
    return struct {
        const Self = @This();
        
        left: [length]u32 = undefined,
        right: [length]u32 = undefined,

        fn init(input: []const u8) !Self {}

        // ...
    };
}

This lets me instantiate the type with a size that matches my input:

src/days/day01.zig

// Here, `Day01` is called with the size of my actual input.
pub fn run(_: std.mem.Allocator, is_run: bool) ![3]u64 {
    // ...
    const input = @embedFile("./data/day01.txt");
    var puzzle = try Day01(1000).init(input);
    // ...
}

// Here, `Day01` is called with the size of my test input.
test "day 01 part 1 sample 1" {
    var puzzle = try Day01(6).init(sample_input);
    // ...
}

This allows me to reuse logic across different inputs while still hardcoding the array sizes. Without comptime, I have to either create a separate function for all my different inputs or dynamically allocate memory because I can’t hardcode the array size.

I also used comptime to shift some computation to compile-time to reduce runtime overhead. For example, on day 4, I needed a function to check whether a string matches either "XMAS" or its reverse, "SAMX". A pretty simple function that you can write as a one-liner in Python:

example.py

def matches(pattern, target):
    return target == pattern or target == pattern[::-1]

Typically, a function like this requires some dynamic allocation to create the reversed string, since the length of the string is only known at runtime.^{[4]
[4]} For this puzzle, since the words to reverse are known at compile-time, we can do something like this:

src/days/day04.zig

fn matches(comptime word: []const u8, slice: []const u8) bool {
    var reversed: [word.len]u8 = undefined;
    @memcpy(&reversed, word);
    std.mem.reverse(u8, &reversed);
    return std.mem.eql(u8, word, slice) or std.mem.eql(u8, &reversed, slice);
}

As a bonus, I was curious as to what this looks like compiled, so I listed all the functions in this binary in GDB and found:

72: static bool day04.Day04(140).matches__anon_19741;
72: static bool day04.Day04(140).matches__anon_19750;

It does generate separate functions! ↩︎

This creates a separate function for each word I want to reverse.^{[5]
[5]} Each function has an array with the same size as the word to reverse. This removes the need for dynamic allocation and makes the code run faster. As a bonus, Zig also warns you when this word isn’t compile-time known, so you get an immediate error if you pass in a runtime value.

Optional Types

A common pattern in C is to return special sentinel values to denote missing values or errors, e.g. -1, 0, or NULL. In fact, I did this on day 13 of the challenge:

src/days/day13.zig

// We won't ever get 0 as a result, so we use it as a sentinel error value.
fn count_tokens(a: [2]u8, b: [2]u8, p: [2]i64) u64 {
    const numerator = @abs(p[0] * b[1] - p[1] * b[0]);
    const denumerator = @abs(@as(i32, a[0]) * b[1] - @as(i32, a[1]) * b[0]);
    return if (numerator % denumerator != 0) 0 else numerator / denumerator;
}

// Then in the caller, skip if the return value is 0.
if (count_tokens(a, b, p) == 0) continue;

This works, but it’s easy to forget to check for those values, or worse, to accidentally treat them as valid results. Zig improves on this with optional types. If a function might not return a value, you can return ?T instead of T. This also forces the caller to handle the null case. Unlike C, null isn’t a pointer but a more general concept. Zig treats null as the absence of a value for any type, just like Rust’s Option<T>.

The count_tokens function can be refactored to:

src/days/day13.zig

// Return null instead if there's no valid result.
fn count_tokens(a: [2]u8, b: [2]u8, p: [2]i64) ?u64 {
    const numerator = @abs(p[0] * b[1] - p[1] * b[0]);
    const denumerator = @abs(@as(i32, a[0]) * b[1] - @as(i32, a[1]) * b[0]);
    return if (numerator % denumerator != 0) null else numerator / denumerator;
}

// The caller is now forced to handle the null case.
if (count_tokens(a, b, p)) |n_tokens| {
    // logic only runs when n_tokens is not null.
}

Zig also has a concept of error unions, where a function can return either a value or an error. In Rust, this is Result<T>. You could also use error unions instead of optionals for count_tokens; Zig doesn’t force a single approach. I come from Clojure, where returning nil for an error or missing value is common.

Grid Padding

This year has a lot of 2D grid puzzles (arguably too many). A common feature of grid-based algorithms is the out-of-bounds check. Here’s what it usually looks like:

example.zig

fn dfs(map: [][]u8, position: [2]i8) u32 {
    const x, const y = position;
    
    // Bounds check here.
    if (x < 0 or y < 0 or x >= map.len or y >= map[0].len) return 0;

    if (map[x][y] == .visited) return 0;
    map[x][y] = .visited;

    var result: u32 = 1;
    for (directions) | direction| {
        result += dfs(map, position + direction);
    }
    return result;
}

This is a typical recursive DFS function. After doing a lot of this, I discovered a nice trick that not only improves code readability, but also its performance. The trick here is to pad the grid with sentinel characters that mark out-of-bounds areas, i.e. add a border to the grid.

Here’s an example from day 6:

Original map:               With borders added:
                            ************
....#.....                  *....#.....*
.........#                  *.........#*
..........                  *..........*
..#.......                  *..#.......*
.......#..        ->        *.......#..*
..........                  *..........*
.#..^.....                  *.#..^.....*
........#.                  *........#.*
#.........                  *#.........*
......#...                  *......#...*
                            ************

You can use any value for the border, as long as it doesn’t conflict with valid values in the grid. With the border in place, the bounds check becomes a simple equality comparison:

example.zig

const border = '*';

fn dfs(map: [][]u8, position: [2]i8) u32 {
    const x, const y = position;
    if (map[x][y] == border) { // We are out of bounds
        return 0;
    }
    // ...
}

This is much more readable than the previous code. Plus, it’s also faster since we’re only doing one equality check instead of four range checks.

That said, this isn’t a one-size-fits-all solution. This only works for algorithms that traverse the grid one step at a time. If your logic jumps multiple tiles, it can still go out of bounds (except if you increase the width of the border to account for this). This approach also uses a bit more memory than the regular approach as you have to store more characters.

SIMD Vectors

This could also go in the performance section, but I’m including it here because the biggest benefit I get from using SIMD in Zig is the improved code readability. Because Zig has first-class support for vector types, you can write elegant and readable code that also happens to be faster.

If you’re not familiar with vectors, they are a special collection type used for Single instruction, multiple data (SIMD) operations. SIMD allows you to perform computation on multiple values in parallel using only a single CPU instruction, which often leads to some performance boosts.^{[6]
[6]}

I mostly use vectors to represent positions and directions, e.g. for traversing a grid. Instead of writing code like this:

example.zig

next_position = .{ position[0] + direction[0], position[1] + direction[1] };

You can represent position and direction as 2-element vectors and write code like this:

example.zig

next_position = position + direction;

This is much nicer than the previous version!

Day 25 is another good example of a problem that can be solved elegantly using vectors:

src/days/day25.zig

var result: u64 = 0;
for (self.locks.items) |lock| { // lock is a vector
    for (self.keys.items) |key| { // key is also a vector
        const fitted = lock + key > @as(@Vector(5, u8), @splat(5));
        const is_overlap = @reduce(.Or, fitted);
        result += @intFromBool(!is_overlap);
    }
}

Expressing the logic as vector operations makes the code cleaner since you don’t have to write loops and conditionals as you typically would in a traditional approach.

Performance Tips

The tips below are general performance techniques that often help, but like most things in software engineering, “it depends”. These might work 80% of the time, but performance is often highly context-specific. You should benchmark your code instead of blindly following what other people say.

This section would’ve been more fun with concrete examples, step-by-step optimisations, and benchmarks, but that would’ve made the post way too long. Hopefully, I’ll get to write something like that in the future.^{[7]
[7]}

Minimise Allocations

Whenever possible, prefer static allocation. Static allocation is cheaper since it just involves moving the stack pointer vs dynamic allocation which has more overhead from the allocator machinery. That said, it’s not always the right choice since it has some limitations, e.g. stack size is limited, memory size must be compile-time known, its lifetime is tied to the current stack frame, etc.

If you need to do dynamic allocations, try to reduce the number of times you call the allocator. The number of allocations you do matters more than the amount of memory you allocate. More allocations mean more bookkeeping, synchronisation, and sometimes syscalls.

A simple but effective way to reduce allocations is to reuse buffers, whether they’re statically or dynamically allocated. Here’s an example from day 10. For each trail head, we want to create a set of trail ends reachable from it. The naive approach is to allocate a new set every iteration:

src/days/day10.zig

for (self.trail_heads.items) |trail_head| {
    var trail_ends = std.AutoHashMap([2]u8, void).init(self.allocator);
    defer trail_ends.deinit();
    
    // Set building logic...
}

What you can do instead is to allocate the set once before the loop. Then, each iteration, you reuse the set by emptying it without freeing the memory. For Zig’s std.AutoHashMap, this can be done using the clearRetainingCapacity method:

src/days/day10.zig

var trail_ends = std.AutoHashMap([2]u8, void).init(self.allocator);
defer trail_ends.deinit();

for (self.trail_heads.items) |trail_head| {
    trail_ends.clearRetainingCapacity();
    
    // Set building logic...
}

If you use static arrays, you can also just overwrite existing data instead of clearing it.

A step up from this is to reuse multiple buffers. The simplest form of this is to reuse two buffers, i.e. double buffering. Here’s an example from day 11:

src/days/day11.zig

// Initialise two hash maps that we'll alternate between.
var frequencies: [2]std.AutoHashMap(u64, u64) = undefined;
for (0..2) |i| frequencies[i] = std.AutoHashMap(u64, u64).init(self.allocator);
defer for (0..2) |i| frequencies[i].deinit();

var id: usize = 0;
for (self.stones) |stone| try frequencies[id].put(stone, 1);

for (0..n_blinks) |_| {
    var old_frequencies = &frequencies[id % 2];
    var new_frequencies = &frequencies[(id + 1) % 2];
    id += 1;

    defer old_frequencies.clearRetainingCapacity();

    // Do stuff with both maps...
}

Here we have two maps to count the frequencies of stones across iterations. Each iteration will build up new_frequencies with the values from old_frequencies. Doing this reduces the number of allocations to just 2 (the number of buffers). The tradeoff here is that it makes the code slightly more complex.

Make Your Data Smaller

A performance tip people say is to have “mechanical sympathy”. Understand how your code is processed by your computer. An example of this is to structure your data so it works better with your CPU. For example, keep related data close in memory to take advantage of cache locality.

Reducing the size of your data helps with this. Smaller data means more of it can fit in cache. One way to shrink your data is through bit packing. This depends heavily on your specific data, so you’ll need to use your judgement to tell whether this would work for you. I’ll just share some examples that worked for me.

The first example is in day 6 part two, where you have to detect a loop, which happens when you revisit a tile from the same direction as before. To track this, you could use a map or a set to store the tiles and visited directions. A more efficient option is to store this direction metadata in the tile itself.

There are only four tile types, which means you only need two bits to represent the tile types as an enum. If the enum size is one byte, here’s what the tiles look like in memory:

.obstacle -> 00000000
.path     -> 00000001
.visited  -> 00000010
.path     -> 00000011

As you can see, the upper six bits are unused. We can store the direction metadata in the upper four bits. One bit for each direction. If a bit is set, it means that we’ve already visited the tile in this direction. Here’s an illustration of the memory layout:

        direction metadata   tile type
           ┌─────┴─────┐   ┌─────┴─────┐
┌────────┬─┴─┬───┬───┬─┴─┬─┴─┬───┬───┬─┴─┐
│ Tile:  │ 1 │ 0 │ 0 │ 0 │ 0 │ 0 │ 1 │ 0 │
└────────┴─┬─┴─┬─┴─┬─┴─┬─┴───┴───┴───┴───┘
   up bit ─┘   │   │   └─ left bit
    right bit ─┘ down bit

If your language supports struct packing, you can express this layout directly:^{[8]
[8]}

src/days/day06.zig

const Tile = packed struct(u8) {
    const TileType = enum(u4) { obstacle, path, visited, exit };

    up: u1 = 0,
    right: u1 = 0,
    down: u1 = 0,
    left: u1 = 0,
    tile: TileType,

    // ...
}

Doing this avoids extra allocations and improves cache locality. Since the directions metadata is colocated with the tile type, all of them can fit together in cache. Accessing the directions just requires some bitwise operations instead of having to fetch them from another region of memory.

Another way to do this is to represent your data using alternate number bases. Here’s an example from day 23. Computers are represented as two-character strings made up of only lowercase letters, e.g. "bc", "xy", etc. Instead of storing this as a [2]u8 array, you can convert it into a base-26 number and store it as a u16.^{[9]
[9]}

Here’s the idea: map 'a' to 0, 'b' to 1, up to 'z' as 25. Each character in the string becomes a digit in the base-26 number. For example, "bc" ( [2]u8{ 'b', 'c' }) becomes the base-10 number 28 ( $1 \times 26 + 2 = 28$ ). If we represent this using the base-64 character set, it becomes 12 ('b' = 1, 'c' = 2).

While they take the same amount of space (2 bytes), a u16 has some benefits over a [2]u8:

It fits in a single register, whereas you need two for the array.
Comparison is faster as there is only a single value to compare.

Reduce Branching

I won’t explain branchless programming here; Algorithmica explains it way better than I can. While modern compilers are often smart enough to compile away branches, they don’t catch everything. I still recommend writing branchless code whenever it makes sense. It also has the added benefit of reducing the number of codepaths in your program.

Again, since performance is very context-dependent, I’ll just show you some patterns I use. Here’s one that comes up often:

src/days/day02.zig

if (is_valid_report(report)) {
    result += 1;
}

Instead of the branch, cast the bool into an integer directly:

src/days/day02.zig

result += @intFromBool(is_valid_report(report))

Another example is from day 6 (again!). Recall that to know if a tile has been visited from a certain direction, we have to check its direction bit. Here’s one way to do it:

src/days/day06.zig

fn has_visited(tile: Tile, direction: Direction) bool {
    switch (direction) {
        .up => return self.up == 1,
        .right => return self.right == 1,
        .down => return self.down == 1,
        .left => return self.left == 1,
    }
}

This works, but it introduces a few branches. We can make it branchless using bitwise operations:

src/days/day06.zig

fn has_visited(tile: Tile, direction: Direction) bool {
    const int_tile = std.mem.nativeToBig(u8, @bitCast(tile));
    const mask = direction.mask();
    const bits = int_tile & 0xff; // Get only the direction bits
    return bits & mask == mask;
}

While this is arguably cryptic and less readable, it does perform better than the switch version.

Avoid Recursion

The final performance tip is to prefer iterative code over recursion. Recursive functions bring the overhead of allocating stack frames. While recursive code is more elegant, it’s also often slower unless your language’s compiler can optimise it away, e.g. via tail-call optimisation. As far as I know, Zig doesn’t have this, though I might be wrong.

Recursion also has the risk of causing a stack overflow if the execution isn’t bounded. This is why code that is mission- or safety-critical avoids recursion entirely. It’s in TigerBeetle’s TIGERSTYLE and also NASA’s Power of Ten.

Iterative code can be harder to write in some cases, e.g. DFS maps naturally to recursion, but most of the time it is significantly faster, more predictable, and safer than the recursive alternative.

Benchmarks

I ran benchmarks for all 25 solutions in each of Zig’s optimisation modes. You can find the full results and the benchmark script in my GitHub repository. All benchmarks were done on an Apple M3 Pro.

As expected, ReleaseFast produced the best result with a total runtime of 85.1 ms. I’m quite happy with this, considering the two constraints that limited the number of optimisations I can do to the code:

Parts should be solved separately - Some days can be solved in a single go, e.g. day 10 and day 13, which could’ve saved a few milliseconds.
No concurrency or parallelism - My slowest days are the compute-heavy days that are very easily parallelisable, e.g. day 6, day 19, and day 22. Without this constraint, I can probably reach sub-20 milliseconds total(?), but that’s for another time.

You can see the full benchmarks for ReleaseFast in the table below:

Day	Title	Parsing (µs)	Part 1 (µs)	Part 2 (µs)	Total (µs)
1	Historian Hysteria	23.5	15.5	2.8	41.8
2	Red-Nosed Reports	42.9	0.0	11.5	54.4
3	Mull it Over	0.0	7.2	16.0	23.2
4	Ceres Search	5.9	0.0	0.0	5.9
5	Print Queue	22.3	0.0	4.6	26.9
6	Guard Gallivant	14.0	25.2	24,331.5	24,370.7
7	Bridge Repair	72.6	321.4	9,620.7	10,014.7
8	Resonant Collinearity	2.7	3.3	13.4	19.4
9	Disk Fragmenter	0.8	12.9	137.9	151.7
10	Hoof It	2.2	29.9	27.8	59.9
11	Plutonian Pebbles	0.1	43.8	2,115.2	2,159.1
12	Garden Groups	6.8	164.4	249.0	420.3
13	Claw Contraption	14.7	0.0	0.0	14.7
14	Restroom Redoubt	13.7	0.0	0.0	13.7
15	Warehouse Woes	14.6	228.5	458.3	701.5
16	Reindeer Maze	12.6	2,480.8	9,010.7	11,504.1
17	Chronospatial Computer	0.1	0.2	44.5	44.8
18	RAM Run	35.6	15.8	33.8	85.2
19	Linen Layout	10.7	11,890.8	11,908.7	23,810.2
20	Race Condition	48.7	54.5	54.2	157.4
21	Keypad Conundrum	0.0	1.7	22.4	24.2
22	Monkey Market	20.7	0.0	11,227.7	11,248.4
23	LAN Party	13.6	22.0	2.5	38.2
24	Crossed Wires	5.0	41.3	14.3	60.7
25	Code Chronicle	24.9	0.0	0.0	24.9

A weird thing I found when benchmarking is that for day 6 part two, ReleaseSafe actually ran faster than ReleaseFast (13,189.0 µs vs 24,370.7 µs). Their outputs are the same, but for some reason, ReleaseSafe is faster even with the safety checks still intact.

The Zig compiler is still very much a moving target, so I don’t want to dig too deep into this, as I’m guessing this might be a bug in the compiler. This weird behaviour might just disappear after a few compiler version updates.

Reflections

Looking back, I’m really glad I decided to do Advent of Code and followed through to the end. I learned a lot of things. Some are useful in my professional work, some are more like random bits of trivia. Going with Zig was a good choice too. The language is small, simple, and gets out of your way. I learned more about algorithms and concepts than the language itself.

Besides what I’ve already mentioned earlier, here are some examples of the things I learned:

The concept of Manhattan distance.
Cliques and the Bron-Kerbosch algorithm.
The ripple-carry adder circuit.
LaTeX math syntax with MathJax, which I used in my write-ups.
In Python, you can use complex numbers to traverse 2D grids!

Some of my self-imposed constraints and rules ended up being helpful. I can still (mostly) understand the code I wrote a few months ago. Putting all of the code in a single file made it easier to read since I don’t have to context switch to other files all the time.

However, some of them did backfire a bit, e.g. the two constraints that limit how I can optimise my code. Another one is the “hardcoding allowed” rule. I used a lot of magic numbers, which helped to improve performance, but I didn’t document them, so after a while, I don’t even remember how I got them. I’ve since gone back and added explanations in my write-ups, but next time I’ll remember to at least leave comments.

One constraint I’ll probably remove next time is the no concurrency rule. It’s the biggest contributor to the total runtime of my solutions. I don’t do a lot of concurrent programming, even though my main language at work is Go, so next time it might be a good idea to use Advent of Code to level up my concurrency skills.

I also spent way more time on these puzzles than I originally expected. I optimised and rewrote my code multiple times. I also rewrote my write-ups a few times to make them easier to read. This is by far my longest side project yet. It’s a lot of fun, but it also takes a lot of time and effort. I almost gave up on the write-ups (and this blog post) because I don’t want to explain my awful day 15 and day 16 code. I ended up taking a break for a few months before finishing it, which is why this post is published in August lol.

Just for fun, here’s a photo of some of my notebook sketches that helped me visualise my solutions. See if you can guess which days these are from:

What’s Next?

So… would I do it again? Probably, though I’m not making any promises. If I do join this year, I’ll probably stick with Zig. I had my eyes on Zig since the start of 2024, so Advent of Code was the perfect excuse to learn it. This year, there aren’t any languages in particular that caught my eye, so I’ll just keep using Zig, especially since I have a proper setup ready.

If you haven’t tried Advent of Code, I highly recommend checking it out this year. It’s a great excuse to learn a new language, improve your problem-solving skills, or just learn something new. If you’re eager, you can also do the previous years’ puzzles as they’re still available.

One of the best aspects of Advent of Code is the community. The Advent of Code subreddit is a great place for discussion. You can ask questions and also see other people’s solutions. Some people also post really cool visualisations like this one. They also have memes!

I failed my first attempt horribly with Clojure during Advent of Code 2023. Once I reached the later half of the event, I just couldn’t solve the problems with a purely functional style. I could’ve pushed through using imperative code, but I stubbornly chose not to and gave up… ↩︎
The original constraint was that each solution must run in under one second. As it turned out, the code was faster than I expected, so I increased the difficulty. ↩︎
TigerBeetle’s code quality and engineering principles are just wonderful. ↩︎
You can implement this function without any allocation by mutating the string in place or by iterating over it twice, which is probably faster than my current implementation. I kept it as-is as a reminder of what comptime can do. ↩︎
As a bonus, I was curious as to what this looks like compiled, so I listed all the functions in this binary in GDB and found:
```
72: static bool day04.Day04(140).matches__anon_19741;
72: static bool day04.Day04(140).matches__anon_19750;
```
It does generate separate functions! ↩︎
Well, not always. The number of SIMD instructions depends on the machine’s native SIMD size. If the length of the vector exceeds it, Zig will compile it into multiple SIMD instructions. ↩︎
Here’s a nice post on optimising day 9’s solution with Rust. It’s a good read if you’re into performance engineering or Rust techniques. ↩︎
One thing about packed structs is that their layout is dependent on the system endianness. Most modern systems are little-endian, so the memory layout I showed is actually reversed. Thankfully, Zig has some useful functions to convert between endianness like std.mem.nativeToBig, which makes working with packed structs easier. ↩︎
Technically, you can store 2-digit base 26 numbers in a u10, as there are only $26^{2}$ possible numbers. Most systems usually pad values by byte size, so u10 will still be stored as u16, which is why I just went straight for it. ↩︎