Let me tell you a story about the beautiful possibilities and the mundane realities of code. I lived this story. And it taught me that a good library doesn’t have to make code sing. It just needs to get the job done without making a mess.

I was working as a contractor at a company that was using Om Next, the sequel to the original Om. Both libraries were wrappers around React. While Om was a minimalist take on how to structure a UI, it revealed a problem: Often your UI’s hierarchy is very different from how your data is structured. Om Next solved this problem by introducing the idea of parsers, which let you convert a declarative expression of what a component needed into a function of the data. This indirection allowed you to shape your data and UI in different ways.

I had used the original Om, but I had not used Om Next in anger. At this job, I needed to create a few new components in the UI, so I got to writing a parser. I made a big mess. In order to express the information I needed (the query), I had to invent a new language. That’s not easy. Then you have to write code to interpret that new language, dig into the central data store for what the query needs, and return data in a new format that your component will consume. The code looked terrible (many nested ifs) and I just couldn’t see how to improve it. The existing components’ parsers looked just as bad. It looked like all of the complexity of the data model was condensed into these parsers so that the data store could remain normalized, the components could be simple, and the query could be concise.

I couldn’t help comparing this to Reagent and Reframe, other wrappers around React, both of which I was more familiar with. Reframe also had a centralized data store, but it didn’t have parsers and queries. Instead, you built subscriptions, which were “reactive” functions of the data store. If you wanted the data for a component, you would write a subscription which gave it to you. It was more direct than writing a query (new language) and a parser (new interpreter). But it was still a layer of indirection.

I asked one of the more senior engineers at the company whether I was crazy. I had never used Om Next before, so maybe I was missing something. Was all of that gnarly code worth it? Why is this better than Reframe? His answer inspired this post. He said, “It can get gnarly, but it gives you the tools such that when it’s done right, it’s really sublime.”

Being a fan of sublime code, I wondered why his answer troubled me. And then I realized the reason: If the choice is between sublime when everything goes right and horrible if anything goes wrong, that’s not a great choice. I would much rather have mostly readable, even when I’m not making perfect decisions, with less extremes on the good and bad. Put another way, mundane but workable beats sublime with herculean effort. Reframe is mostly workable. Om Next is mostly gnarly.

Since that conversation I’ve been pondering this as a principle. It’s a tradeoff between tradeoffs. When it comes to code beauty, I would rather have a fat average of the bell curve than a lot of outliers. I’d rather have base hits than home runs, because the players swinging for home runs strike out just as often. And, listen, I’m just a working programmer trying to get the cart icon to show the count of items inside. I’m not building a Rolex.

I believe my experience also inspired me to create my Reframe course. I learned to appreciate the framework. Reframe gives you the tools to build understandable UIs of medium to high complexity. And after my experience with Om Next, I saw how Reframe’s tools made decisions straightforward. They never reach the heights of sublimity that legends tell of Om Next. But the everyday code of Reframe is fine. Messes are contained and can be split. There aren’t any difficult design decisions to make, like designing a language and interpreter.

So that’s my story. I became a fan of Reframe because of how bad the default code in Om Next was. And I decided that the promise of beautiful code if you do it right is not enough. What does the code look like when I’m in a rush? What does it look like when I don’t really care that day? What does it look like when a junior programmer writes it? What does it look like when you’ve made the wrong decision but you don’t have time to rework it? On most days, things are not at their best. Making it easy to write decent code is way more important than making it possible to write beautiful code. It’s a principle I keep in my back pocket for when I or a colleague wants to make a tool to make beautiful things at work. I pull it out to dissuade them. It’s the everyday code that matters.

Denotational vs operational

I follow a number of people’s work, and one of them is Conal Elliott. I like that he’s very deliberate with his use of words. It makes me think a little harder. At some point, I heard him say that he didn’t think declarative was the right word. People tend to toss the word around and explain it as “saying what you want, not how to get it.” It’s the opposite of imperative, where you say how to get it. But Elliot is right. It’s not the word we want.

Like many problematic words, people use it in different senses. But the worst offense is that it’s not objective. It’s vibes. It’s relative. I’ve heard people say that map is declarative while a for loop is imperative. “With map, I just say what I want. But with a for loop, I have to say how to build the resulting array.” That may be true, but a for loop is very declarative relative to an if statement and a goto. Look at the anatomy of a for loop. You specify four things:

1. Initialization

2. Loop condition

3. Loop progress

4. Body of loop

And it magically executes the body of the loop until the loop condition is met. How does it do it? You never specified how. Which means it’s declarative. And I’ve implemented map many times over the years. I know how it’s working its wizardry. Which means it’s imperative.

These words just are not what we’re looking for. The main problem I have with the term declarative as we use it is that it’s all about ignorance. People say SQL is declarative because they don’t know how the query engine works. Once you learn how it works, you begin to write SQL to control how it runs.

Elliot suggests we instead use the term denotational, borrowed from the practice of Denotational Semantics. Denotational Semantics is a method for formally specifying the semantics of a programming language or construct in terms of lambda calculus. It contrasts with Operational Semantics, which is more like how a compiler might translate a language to machine code. If you squint, we want to replace declarative with denotational and replace imperative with operational.

In denotational programming, we specify how to derive the meaning of one expression by combining the meanings of the subexpressions. Even if we know what’s happening under the hood, the derivation of meaning is clear. Reasoning can be done locally. In operational semantics, we may get a good idea of how a feature can be implemented in assembly, but we have trouble simulating it in our head because of the many possible states the machine can be in. It is much harder to formalize and prove properties about it.

When we’re doing good functional programming, it’s much more on the denotational side. We’re defining one function in terms of other functions. We’re embedded in a lambda calculus, so it’s very comfortable to do.

I think there’s another correspondence, which is between operational/denotational and implementation/specification. I think one of the important skills that differentiates of senior from a junior engineer is the ability to separate implementation from specification. A junior is just trying to get something working. But a senior engineer thinks about the meanings they want to represent apart from how to represent them.

This brings me to the working title of my book, Runnable Specifications. The idea is one I borrowed from Alan Kay. He says the ultimate goal for a programming system is to be able to run the specification, not translate it into an implementation. You want to write down a question and run it to get the answer. He’s talking about a denotational way of programming where you write programs directly in the language of the domain.

Denotational differs from declarative in one other way. Denotational programming deals with how meanings are combined to form new meanings. Declarative can but often does not deal with combining meanings. I look at SQL again as an example of something that doesn’t make composition easy.

Well, I don’t think I really destroyed the term declarative as much I imagined I would at the beginning of this essay. That’s okay. I think denotational is a strong enough replacement that it doesn’t really need to be destroyed. The biggest downside to the word is that people don’t know what it means. But I do!

What are formal systems?

When I’m drafting my book, I often get a glimpse of the underlying structure the ideas I’m presenting are built on. These ideas seem so important. But alas, they’re often very abstract and not directly relevant to the skills I’m trying to teach to a particular audience. This newsletter and my podcast are a way to write/talk about those ideas to a more sympathetic audience.

My book is about domain modeling. One ideal to strive for when building a domain model in code is to construct it as a formal system. When I first got that idea, I do what I typically do, which is to start writing about it. But soon I realized that I didn’t really know what formal meant in that term. I did some research. And it’s a really important idea, so I’m going to share my understanding with you.

Before my research, if you asked me what formal system meant, I would tell you that it’s about being rigorously defined, as in “let’s get a formal definition.” But that’s not it, really. Yes, formal system are rigorously defined, but they’re a lot more like programming languages than just “good definition.” Let’s look at an example:

In predicate logic, a kind of formal logic, we can assign variables to a few concepts:

• Jim - the protagonist of this story

• Pizza - the plot device

• Likes - the character development

• Eats - the action

Then we can talk about Jim, like this:

This means that if Jim likes a food, he eats it. Now what happens if we also throw this into the mix?

Jim likes pizza. We can then imply that:

Jim eats pizza. Surprise ending.

Well, okay, it’s not a very interesting story. But it does give us everything we need to define formal system, the real plot of the story. Mathematicians will define it like this:

A formal system consists of:

• symbols (valid characters/tokens) — Jim, Likes, (, ⇒, …

• statements (valid arrangements of symbols) — Likes(Jim, Pizza)

• mechanistic rules (for deriving new statements from existing statements) — the rule that lets us derive our ending

• axioms (statements that are true implicitly)

I think of it in programming terms. The first two items (symbols and statements) are the syntax. The last two (mechanistic rules and axioms) define the semantics.

There’s something about formal systems that is powerful and paradoxical. There’s a strange combination of meaning, meaninglessness, and machine-like coldness that somehow makes them more powerful. And that’s why I find them fascinating, just like programming itself. It’s just electrons moving through a circuit, yet somehow it does meaningful work. Where does the meaning come from? (Hint: It’s us.)

First, the syntax defines a structure, like an Abstract Syntax Tree. An important part of the syntax is that there are symbols that are neutral. These are the variables. We defined variables like Jim and Pizza, but as far as logic is concerned, they could be A and B. Logic doesn’t care about Jim or pizza. Likewise, your compiler doesn’t care what you name your variables, nor does the chip it runs on. And this is important. The variables are only linked to human meaning through their names and other connections to real-world things.

However, some parts of the syntax are meaningful to the system. The structure itself is meaningful (as in F(A, B) indicates a predicate called F with arguments A and B). And some of the symbols have meaning (⇒ means implication). The meaning comes from their use in the mechanistic rules, somewhat how a syntax-driven interpreter would work. The structure of the statements is pattern-matched in the derivation rules, which leads to new statements. The term formal refers to the importance of the structure in defining meaning.

I find this mix of meaningless symbols and meaningful symbols to be a beautiful puzzle. Yes, there is a certain elegance to formal systems like propositional logic. A handful of meaningful symbols (logical connectives) and a few inference rules can ensure you only derive valid arguments. But there’s something more than just elegance at work. There’s something to their limited meaning that makes them practical. Logic wouldn’t work if you tried to include all humanly felt meaning, so you only include a tiny subset that you trust to work. Yet it can find new, if simplistic, results.

One interesting exercise is to find the minimal set of meaningful symbols that can express all the others. In propositional logic, this is the NAND connective (NOR works, too). The SK combinator calculus is Turing complete. And you can derive both S and K from the iota combinator! So iota, just one symbol (and using the lambda calculus rule of application) is Turing complete.

Similarly, Lisp is very minimal (at least the original Lisp). It is defined with three kinds of symbols:

1. Atoms (alphanumeric sequences)

2. (

3. )

The syntax also requires spaces between atoms to distinguish them. The syntax defines two kinds of values:

1. Atoms

2. Lists

Super minimal! The inference rules are also simple:

1. An atom’s meaning is either a number (if it’s all digits) or it’s a symbol, in which case its value is looked up.

2. A list’s meaning is determined by the first value in it.

   1. There are certain symbols with special meanings, like lambda or cond.

   2. Other symbols are looked up and applied like functions to the arguments.

This is such a simple formal system, yet it builds up an entire programming language. I’m leaving out that you probably have to define a lot of the functions yourself in some other language (like machine code), but even that number is small. In other words, Lisps are bootstrapped. Lisps today are more complicated, but they still maintain some taste of the simplicity of the original.

What I think is cool is that programming languages are essentially formal systems. They have all the ingredients. Their semantics are defined by the compiler, which defines them as a translation into another formal system, the machine code. Strangely, machine code does not have any non-meaningful parts. Anything you can express in machine code is meaningful to the semantics of the chip. The only hint of a meaningless thing is that it operates on numbers that themselves are meaningless to it, except in as much as it knows how to do arithmetic on them and fetch memory at a numeric address. They are abstract quantities that represent values meaningful to us. So we can say 1 means Jim and 2 means pizza.

I feel compelled to mention the book Gödel, Escher, Bach right now, for some reason. I have read only a few pages of it, but it’s probably a more playful and complete exploration of the strangeness of formal systems than I can possibly do. Just wait until you see what self-reference in a formal system does to it!

The challenge with using formal systems is in modeling a real-world scenario. It’s an art to translate Jim and his tastes into logical statements such that you can derive new meaningful statements from it. The derivation, once the statements are written, is quite trivial. This challenge of encoding the meaning is one reason why proofs are hard even when something seems obvious. By the way, the art of encoding a set of real-world scenario into a program is known as domain modeling. What are the meaningful parts you want to capture from the domain, and what parts are you leaving up to the humans for meaning?

There’s also a cool balance between minimalism of the system and human habitability. Sure, it’s very elegant that NAND is complete. But just look at the statements you have to write to do anything! They’re unreadable. It’s very hard, as a human, to wrap your intellect around them. Structure dominates. Perhaps you can eventually learn to read it, with time, but you’re mostly just counting parentheses. Check out this expression using the iota combinator:

So clear! (not)

There’s a tradeoff between number of symbols and length/structure of statements. We want to select a set of symbols we can easily juggle in our minds (not too big) that are, at least metaphorically, meaningful to the concrete world we live in. Making the story about Jim and pizza is something we can relate to with almost no effort. Jim eats foods he likes, he likes pizza, so he eats pizza. It’s obvious (I believe) because it taps into the social parts of our brains. We want symbols like ⇒, ∧, and ∨. We can understand them. Software design is often about this tradeoff: Do we inline the code or do we extract a function and give it a meaningful name? Do we want longer code or more symbols?

The last cool thing I want to mention is that this is the key to stratified design and metalinguistic abstraction. In stratified design, each layer defines new meaningful symbols for the layer atop it to use (relying on the underlying lambda calculus rules of inference). But in metalinguistic abstraction, you define an entirely new formal system, with new rules of inference, that hopefully lets you express your problem more easily than lambda calculus. Think a rules engine or SAT solver.

Well, it’s time to conclude. Formal systems are a neat mix of meaning and meaningless parts. The meaningless parts are like holes where we can fill in our own, human meaning. We want to find a set of meanings we can use to derive the answers to our questions. That is not easy. And finding the balance between the number of meaningful symbols and the structure is an art. These are some of the underlying questions we face every day when we program. The questions never go away, but the practice of domain modeling points you to a reliable source of answers: The domain itself.

It was great to go to the Conj this year. While it’s kind of a blur, I got what I was looking for, which was basically a boost in the perceived social value of Clojure. Let me put that less obtusely: Most of my friends are not programmers. They don’t know what Clojure is. I can’t really geek out with them about Clojure. Basically, my interest is private. Over time, it becomes harder and harder to maintain interest in something few people I encounter face-to-face care about. Am I crazy liking this thing? Am I out of touch with reality? Does it matter at all?

The Conj gives me that boost. Hundreds of smart people talking about Clojure, validating my interest in it, all in one small space. There are so many people that there are sub-groups. AI vs. anti-AI. Protocols vs. no protocols. Even REPL vs. no REPL (yes! they exist!). And it’s glorious.

The other benefit is that after three days together blabbing on about Clojure, it starts to get personal. You get to know people beyond their code and their jobs. It turns out that even Clojure programmers have lives.

I want to touch on a couple of things.

My workshop went well. The ideal is that I blow people’s minds with skills they couldn’t have learned any other way that they can immediately apply at their work. I hit somewhat short of that. The second best is missing that but learning how to do better next time. That’s where it was. Luckily, I can translate that learning directly into my book. Thanks to all the participants who played along.

People don’t get to see the magic of Rich Hickey outside of the Conj. Seriously, he’s amazing. As far as I can tell, he watches every talk (unless there are two tracks, obviously). He will ask questions during the Q&A. And he’ll approach the speaker after the talk and share opinions, critiques, and encouragement. Despite not being as active online as he once was, he is committed to the community when it comes to in-person gatherings. He’s super engaged with the design of new features and is still thinking through tough problems, even after his retirement.

Speaking of community, it was a big focus this year. Christophe and Jordan (and other volunteers) hosted a community-building event (where we all got to share our love within small groups guided by Clojure luminaries). Just their presence and energy made it clear that this conference is important.

And I’m super excited about the Clojure documentary. Rich Hickey has already been interviewed out at his cabin. Many Clojure folks have also been filmed, including yours truly. I hope my footage doesn’t wind up on the floor. We’re very lucky that the same company that made the Python, Rails, and Node documentaries are making one for Clojure. We’re much smaller. But they’ve told me they think the influence of Clojure is greater than what’s apparent in the number of programmers. I have to agree, but it’s nice that the filmmakers can see that.

Alright, folks! Be good to each other.

Here’s a series of books that give an introduction to the majesty and variety of functional programming.

Pure and higher-order functions

Grokking Simplicity, my own book, provides the necessary groundwork for functional programming. It begins way before most books begin by encouraging the distinction between pure and impure functions. Once that’s mastered, higher-order functions are explored. It has gotten high praise. I don’t feel weird hyping my own book because I wrote it because no book was taking up the task. However, be careful with chapter 3. Every time I open it, I dream of rewriting it for a second edition.

Separating data and behavior

Data-Oriented Programming by Yehonathan Sharvit, introduces the idea of programming with data and not objects. Since most people will be familiar with the standard object-oriented methodology of combining data with behavior, this book presents a very different perspective: Maybe data can be treated separately from behavior.

It is a decidedly Clojure-centered view of functional programming, but even a die-hard Haskeller will find some similarity to how they program, albeit using types instead of run-time schema checks. I think the idea of writing functions over data is such a foreign idea that it’s important to show how the other side lives. However, if there’s one thing from data-oriented programming that I wish was treated more in the book, it’s that much of the time, we don’t need to model our data quite so much. In other words, we’re not always building entities and defining their schemas. Sometimes, the best part of data-oriented programming is just leaving the data as it is and making a program that is data in, data out.

Functional domain modeling

So, I am writing a book on this right now, but it’s not done and what’s been published online will certainly change. It’s meant to be a hands-on guide to the intermediate skills of the programmer. Once you know how to write functions and for loops, where do you turn to write good functions and for loops? If you’re interested, go read the 200 pages that are already online for free. But I’m rewriting all of it to be more focused on learnability and objective. The premise is that software design does not need so much hand waving.

While you wait, you can read Domain-Modeling Made Functional by Scott Wlaschin. It’s good if a little hand-waving, like most software design books. I daresay that it’s the best book I’ve read explaining Domain-Driven Design.

Functional algorithms

Thinking Functionally with Haskell is one I enjoy. If you want something rigorous from someone who takes functional programming seriously, this is a great book. It reads a little like a textbook. But at the same time, it doesn’t have the 3-page code listings some industry books have. The algorithms are chosen to gradually show you the range of techniques you’ll need to solve any problem using functional, as opposed to imperative, style.

Types and category theory

I’ve read several books on these two areas, but I can’t say I’ve found a book that really captures the magic enough for me to recommend it. However, I haven’t read every book. Functional Programming in Scala, by Paul Chiusano and Runar Bjarnason, is extremely popular. It’s so popular, it’s referred to by its color (the red book). Still, I doubt it’s giving me what I’m looking for. I want a book that expresses the beauty and elegance of types and Haskell-style category theory beyond “preventing errors” and reinventing imperative programming.

Linguistic abstraction

This topic isn’t learned quite enough today. We tend to talk about simple indirections like runtime dispatch to solve problems. Runtime dispatch, however, is just a fancy conditional. Sometimes your indirection needs a totally new semantics. My two favorite books for these are Principles of Artificial Intelligence Programming (PAIP) by Peter Norvig and Software Design for Flexibility by Chris Hanson and Gerald Jay Sussman.

PAIP is a classic from the time when AI meant “good programming”. The examples are in Common Lisp and it shows how to build a logic programming system (similar to Prolog), rules engines, and equation simplifiers in a data-driven style.

Software Design for Flexibility is written using Scheme and shows other techniques for developing powerful abstractions for conquering complex problems. The premise is that your semantics need to have more degrees of freedom than they problem you’re solving. Both books show techniques we rarely reach for, even when conditionals and recursion don’t cut it. If I had to pick one, it would be PAIP.

Conclusion

If someone read these books and tried to apply them, they’d be well on their way to mastering functional programming. I know there are a lot of books I’m not familiar with, so please don’t take this as an exhaustive list. It’s just a list of good books that fill niches in the FP journey. I’d love to hear your takes on required reading. Just hit reply and send me a message. Please share what you like about your recommendations along with the titles.

Smalltalk exposed a simple syntax for writing software. I won’t go into too many details, but suffice it to say that it was designed for kids. Building things for children is a constraint which, when overcome, leads to a system that could be used by adults as well. The simple syntax and semantics opened the world of programming to new people. It was democratizing. And it was powerful. In 180 pages of code and 50 classes, the team built an OS, a word processor, a paint application, animation studio, and more. And many of the ideas, usually oversimplified, became the dominant way people program today: with classes and methods.

The thing I’m most afraid of is that we won’t see another leap forward in programming language design. As AI generates a greater portion of code, there won’t be any need for new ways to express programs. The AI doesn’t care how it’s expressed (or at least it doesn’t care about the same things humans do). Programming won’t get easier for humans.

One counterargument to this is that we have not seen any new ways of programming in about 50 years anyway. It’s not the AI’s fault. OOP seems to have been the last major “paradigm.” Maybe there isn’t any more to do? I admit that it’s possible, but it breaks my heart to think there isn’t something better yet to be invented. And it seems so myopic and arrogant to think we’ve figured it all out. I err on the side of hope and humility.

Another argument is that AI is the new paradigm. Instead of typing JavaScript, you type English and the AI types JavaScript. That doesn’t quite make sense, yet, because we do still read the generated code. We have to. AI doesn’t get it right all the time, so you have to review the code, even if you get the AI to generate all of it. I accept that as part of an AI workflow, but it doesn’t support the point. It’s not a new programming paradigm, as in FP, OOP, Procedural, or Logic. Those paradigms provide basic constructs that are composed and named, allowing you to create new constructs. AI is much more like a new way to edit code. The code, though, is the same.

It’s doubtful that a new language, tailored to AI’s strengths, would be economically viable. AI is trained currently on human-written programs. Humans have written billions of lines of JavaScript and the other popular languages. These are the ones that the AI “understands” the best, so these are the ones that it will prefer to generate and make fewer errors with. But at best, it will generate code like the average industry programmer, not like the ones who invented new paradigms. So we won’t see AI being able to invent its own new paradigm.

The one hope for this is something similar to self-play that we see in AlphaGo Zero. Perhaps an AI could be given the syntax of the language and be asked to solve increasingly hard problems. It would learn how to generate good code without human-created examples. But a human would probably have to invent the language the AI was learning. What would that language look like?

Unfortunately, I don’t think it would be very friendly for humans. AIs might as well generate machine code! I’d rather have the AI learn the human thing than humans have to learn the AI thing.

But perhaps more advanced AIs will learn new human-centered languages quickly. Perhaps you could feed it the docs and some example programs, and, along with a REPL, it could figure it out. And so there’s nothing to worry about. Except the new language has bigger challenges to face than it did before. Before, if a new language was 10x more productive than JavaScript (or the most popular language), it had a shot. But if you’re already 10x faster with the AI using JavaScript, your language will have to be 10x that combination, using an AI not trained on it.

So that’s my biggest fear: The use of AI to write code is going to stunt the development of new programming paradigms. In the worst case, this is the end of paradigms as we know them, at least in industry. I hope it doesn’t happen. I still hold out for finding new and better ways to express ideas. I just don’t expect AI to help with it.

Apropos is now on break for the Summer. Enjoy!

LLMs’ surprise teaches us about ourselves

I studied AI in grad school. My master’s thesis was about combining Natural Language Processing (NLP) techniques with Machine Learning (ML) to answer very specific information from a large collection of text documents. That was back in 2008. When ChatGPT first came out, I was skeptical at first. But I gave it a shot and was very surprised. In this essay, I’d like to explore what was surprising to me before my memory of life before LLMs fades.

When I studied AI, NLP was a very different field than Knowledge Representation (KR) and Information Extraction (IE). NLP was about grammars, understanding sentence structure, and manipulating the structure of text, such as turning a declarative into an interrogative. Knowledge Representation was basically data modeling but for AI algorithms to be able to use. And IE was taking unstructured text and turning it into structured data.

Each of those parts had its own literature. Sure, any particular project was going to touch on multiple fields, but each had its own difficulties and hence its own literature. LLMs shatter these distinctions.

Take a basic statistical Markov Chain. It builds a statistical model of the probability distribution of the next word, given the last n words. It was clear, even back in 2008, that if you increased n and gave it more text to train on, the text that the Markov Chain generates sounds more and more human. It is more grammatically correct. And it sounds less and less like the source material, as differences of author start to average out. But also the text is meaningless. Your mind tries to make sense of it because it does sound like language. But if it does mean something, it’s more by coincidence than anything. Part of the joke was that you’d train a Markov Chain on a politician’s speeches and you could generate babble that sounds like them, often hitting the right topics, but never makes any sense. Get it? Politicians talk a lot but don’t say anything of substance.

The success of Markov Chains to produce language, but without any reliability of the content, was a kind of proof that you needed to integrate knowledge representation with the language generation. You needed a way to generate text under the constraint that it say something from the database.

LLMs ignored that problem and just kept increasing n (now called the window) and the size of the training set. They also moved from a statistical to a neural network model, which allows for much better generalization of learning. And there were some structural additions to classical neural nets. Transformers was one of them. It allows the network to attend to different parts of the window—which is now thousands or even millions of words long.

As I write this, LLMs can output flawless English text, meaning they don’t make grammatical mistakes. They’ve done so for a long time. In fact, they’re so good at language itself that they’re used to do fun linguistic things like converting an essay into pirate speak or a rap in the style of Shakespeare. Or even practical things like summarizing text.

What’s most surprising is that the models have improved their factual correctness. I know it may sound odd given how famous LLMs are for hallucination, but it is the most prominent improvement over classical Markov Chains. Whereas statistical Markov Chains said something factually true 1% of the time, LLMs are correct close to 100% of the time. They’ve gone from babbling, senile uncle to overconfident but smart uncle. Or maybe it’s more like a dancing dog. When a dog dances for a few seconds, we clap. But the dance isn’t very good by human standards. We clap for the fact that it danced at all. But a human dancing, that’s a different matter. We start to judge its faults. It missed a beat. There’s some awkward movements. LLMs have crossed over into the level where we notice its hallucinations more than the fact that it works at all.

LLMs began as NLP and mastered it, but in order to keep improving, had to get good at Knowledge Representation and Information Extraction. That’s the surprise. And, like all AI surprises, teaches us something about ourselves.

The lines between language and knowledge and reason have always been hard to draw. If you draw up a simple English grammar, then generate sentences using it, most of the sentences will be meaningless. But some will be awkward. It will say things like “Colorless green ideas sleep furiously.” That one is classically nonsensical. But it will also say things like “I ate three rices.” The trouble is that rice is a mass noun, not a collective noun. It can’t take a number as a modifier. It’s a grammatical distinction that we can incorporate into the simple grammar. But it’s also a very simple aspect of a model of the world. As we add more and more of these aspects to the grammar, it becomes more and more an understanding of the world. And there are countless fine distinctions in grammar that linguists have teased apart in English. LLMs, in my opinion, have creeped into knowledge representation through the kind of world knowledge embedded in grammar.

And they’ve moved far beyond it. However, grammar doesn’t know what the capital cities of countries are. They’re historical accidents. And that’s why it might hallucinate those kinds of facts. It’s still a wonder that they can know any of them at all.

But here’s where the information extraction comes in. In my thesis, I built a system to learn from text. I gave it all of the country names and all of the capital city names. Then I gave it all of the sentences in Wikipedia that contained both a country name and a city name. I parsed the sentences into a graph, then plucked out the path between the city and country names. Then I used good, old-fashioned graph matching to figure out which ones were correlated with capital cities.

The software broke about 80% accuracy with thousands of sentences. But LLMs are different. You can give it one sentence, like “Paris is the capital of France”, and then ask it: “What is the capital of France?” It will tell you. Or you can ask the other way, “What country is Paris the capital of?” Even if it doesn’t know what the capital of France is without you telling it (it probably does), it can use the information in the sentence you gave it to answer questions. What’s more is that you can give it a 10 page document full of information (including the sentence “Paris is the capital of France.”) and ask it those questions. It will still find the answer. Again, it’s approaching Information Extraction from an NLP doorway.

And this also tells us something about ourselves. The knowledge of capital cities is declarative knowledge. It’s the kind of knowledge that is easy to quiz for. I say UK, you say London. But imagine all of the knowledge you have to have to even begin answering the question. What is a country? What is a capital? What if I ask it more subtly, like “Where is the seat of government in the largest country by area?” How could it get that right without a deeper model of the world? Capitals are typically the seat of government. Cities are places. What is the largest country? What does area mean? In my opinion, this has been the key to unlocking Information Extraction—a critical mass of facts filling in an adequate model. And I don’t know how to argue for this, but I believe it: What we call understanding is largely linguistic, captured in our feelings of correct usage. It’s what Wittgenstein was on about. So as the LLM gets better at correct usage, it gets better at interpreting the meaning from text.

There’s one more thing that’s perhaps more subtle. When I first used ChatGPT, it was version 3.5. It hallucinated a lot. But more than that, you’d ask it a difficult question and it would pick a side. For instance, what is the best way to learn to a new language?It might say immersion. Or it might say study from books. Or take a class. All plausible. But when 4 came out, the same question was answered qualitatively differently. What is the best way to learn a new language? “There are differing opinions about what is the most efficient way to learn a language. Some believe immersion is best. Others believe personal study from books. But it depends largely on your goals with learning the language…” In short, it had a theory of knowledge, a kind of epistemology. It didn’t just know facts. It knew that different groups of people believed different facts. In fact, a better way to understand a topic is to understand the controversies of the field—and to be able to argue all sides.

Again, this is surprising. Simply by making the LLM bigger (bigger window, bigger network, bigger training) it has transcended basic models of truth and found a higher version of expertise. This also tells us about ourselves. As we become experts, we transcend any particular point of view and take on a meta point of view. We can hold multiple, often conflicting models in mind at once, and choose between them as we please, never really believing in them any more than we have to. Expertise is built on conflicting models.

Well, thanks for coming on this tour with me. I think the main point is that language is a doorway into our intelligence. It’s hard to know where language ends and understanding begins. And this has allowed LLMs to do many skillful things. Not everything, mind you. LLMs are not able to walk or drive. But they can do a lot. And like all interesting AI, they teach us about ourselves.

The Clojure/conj CFP is open! If you’re looking to speak, go apply! I’ll be there giving a workshop on domain modeling in Clojure.

The economic inevitability of AI

The current hype around AI triggers our cognitive biases and makes it hard to take an objective look. It’s just hard to be certain about anything. On what timeline will AI take our jobs? What threats does AI pose to culture and civilization? Will AI help us manage and mitigate climate change or will it exacerbate it? There are a lot of difficult questions. But there’s one thing I’m fairly confident about, and I think it’s worth taking the time to clear the air about it. I’m certain of the economic inevitability of the AI industry.

The future is clear. It has replayed itself many times in the history of business, and certainly in the history of the computing industry. The future is this: Some company (probably OpenAI but we can’t be sure) will capture a significant percentage of the revenue of white-collar labor. Just like Amazon now captures a significant portion of online commerce, just like AWS takes a cut of most startup’s revenue, the winner of the AI race will extract a fee from most every white collar worker. Like licenses for Microsoft Office or Windows, companies will pay for access to AIs they believe will make their employees more productive.

Companies will gladly ask their employees to use AI if it speeds up their work. Businesses want to be competitive, so as other companies gain efficiencies from their use of AI, they’ll need to do it, too, to keep up. It’s a rational choice. I’d ask my company to do it, too, if I were the CTO. Eventually, most employees doing knowledge work will be using AI. And the employers will be paying licenses to OpenAI (or whoever wins).

We live in a world where winner-takes-all is the natural course of things. Slight advantages compound. The big company can acquire, undercut, or outlast the small company. Eventually, the market sorts out the winner with ~80% market share. The rest of the companies fight over the remaining 20%.

In short, one company will have 80% market share, and the market is AI accelerators for knowledge work. Microsoft had “A computer on every desk.” OpenAI will have “An AI on every desk.”

Once the 80% winner is found, the enshittification can begin. These companies are fueled by massive investments. They can sell services well below cost for years. OpenAI forecasts losing $44 billion in the next four years. But in 2029, they can start experimenting with how to turn all those customers, now dependent on their services, into profitable clients, likely by raising prices, degrading service, or inserting ads. OpenAI will create a few trillionaires, many billionaires, and uncountable millionaires on their ascent.

If you’re looking to make money, hitch yourself to the rise of these AI giants. You could be one of the millionaires by being one of the companies renting the shovels during this gold rush.

History is repeating. Personal computers were supposed to liberate us. But what we didn’t realize when the digital dream of personal freedom was still alive was that the companies who amassed more computers and more data would be more free than others. The computer giants donated computers to school.

Similarly, AI is currently sold as a liberating force. It is “democratizing” programming. It’s giving everyone a cheap therapist. They are more reliable companions. But we know that the productivity gains and the huge profits will accrue to the employers and to the AI companies most of all. OpenAI and the others are making products that are sure to become the next essential tool for thought work. Employers will demand that we use them. OpenAI will take rent on all of the work we do. And it will become enshittified. With that inevitability out of the way, we can talk about the more nuanced ideas in the next emails, including some positive ones!

Our next guest is Peter Strömberg. Peter is known as PEZ online. He is the creator of Calva, the Clojure plugin for VS Code. He’s going to demo some REPL-Driven Development with Calva.

AI, Lisp, and Programming

I have a Masters of Science in Computer Science, with a specialty in Artificial Intelligence. With the way AI salaries are these days, you’d think I pull seven figures. Alas, my degree is from 2008. At that time, people wondered why I wanted to go into a field that had been in a sorry state since the 1980s. Wouldn’t I enjoy something more lucrative like security or databases?

But I liked the project of AI. AI, to me, was an exploration of our own intelligence—how we thought, solved problems, and perceived the world—to better understand how we can get a machine to do it. It was a kind of reverse-engineering of the human mind. By building our own minds, we could understand ourselves.

It’s clear that Lisp and AI have been linked since very early in the two fields’ existences. John McCarthy, the inventor of Lisp, also coined and defined the term artificial intelligence. Lisp was traditionally closely associated with the study of AI. Lisp has been a generator of programming language ideas that might seem normal now, but were considered weird or costly at the time. Here’s a partial list:

1. Garbage collection

2. Structured conditionals

3. First-class and higher-order functions

4. Lexical scoping

While it’s clear that Lisp has been influential on programming languages (even after being ridiculed for the same features languages borrow), what is not so clear is how much AI has been an influence on programming practice. Until recently, it was kind of a joke that AI had been around since the 1950s but hadn’t produced any real results. The other side of the joke is that once something works, it’s just considered programming and not artificial intelligence.

Artificial intelligence has produced so many riches in the programming world. Here is a partial list:

1. Compilers (which used to be called “automatic programming”)

2. Tree search

3. Hash tables

4. Constraint satisfaction

5. Rules engines

6. Priority queues

Let me put it directly: Seeking to understand human thought has been fruitful for software engineering. My interest in Lisp is interlinked with my interest in AI. AI has always been synonymous with “powerful programming techniques.” And I have always seen myself as part of that thread, however small my contribution might be.

The link between AI, Lisp, and programming was so strong 30 years ago that Peter Norvig started the preface of Paradigms of Artificial Intelligence Programming with these words:

This book is concerned with three related topics: the field of artificial intelligence, or AI; the skill of computer programming; and the programming language Common Lisp.

source

In 2008, when I graduated, Google could barely categorize images. Identifying a cat in a photo was still considered a hard problem. Many attempts had been made, but none could get close to human-level. In 2014, I heard about a breakthrough called “deep learning”. It was using the scale of the internet and the vast parallelism of GPUs to make huge neural networks trained on millions of images to break accuracy records. It was working. And it was completely uninteresting to me.

Okay, not really completely uninteresting. It tickled my interest in building new things. I could see how being able to identify cats (or other objects) reliably could be useful. But I saw in this nothing of the project for understanding ourselves. Instead, it was much like what happened at Intel.

Nobody really likes the Intel architecture. It’s not that great. But once Intel got a slight lead in marketshare, it could ride Moore’s Law. Instead of looking for a better architecture, invest your time instead in scaling the transistor down and scaling the number of transistors up. Even the worst architecture will get faster. And Intel can cement their lead by investing in better manufacturing processes. Their dominance wound up lasting decades. But computer architecture has languished relative to the growth of the demand for computing.

The same effect is at play in neural networks: Instead of investing in understanding of how thought works, just throw more processing and more training at bigger networks. With enough money to fund the project, your existing architectures, scaled up, will do better.

These are oversimplifications. There were undoubtedly many minor and some major architectural breakthroughs that helped Intel keep pace with Moore’s Law. Likewise, there have been similar architectural breakthroughs in neural networks, including convolutions and transformers. But the neural network strategy is dominated by scale—more training data, more neurons, more FLOPS.

My whole point is that my research into the history of the field of AI has somewhat inoculated me against the current hype. I don’t think AI will “replace all humans”. And I don’t think AGI (artificial general intelligence) is defined well enough to be a real goal. So where does that leave us? How is AI going to transform programming? Where will all of this end up?

Artificial intelligence has always been a significant part of the leading edge of programming. And its promise has always been far ahead of its ability. In the next few issues, I want to explore what this current hype wave of AI means for us. I don’t like where I see AI going. But I also want to give apply some optimism to it because I think a lot of the consequences are inevitable. The world is being changed, and we will have to live in that new world.

I started perfecting the process of writing processes. At one point, even the meta-work was included in the process itself. Think of the process like a recipe. The recipe says you need 4 cups of flour, 1 cup of oil, etc. Then it has steps like mix the flour and oil in the bottom of a heavy pan, mix some other ingredients in a medium bowl. It seems complete. But it’s not. At some point I started even writing the little steps like gather a spoon, a heavy pan, and a medium bowl. All of the implied steps got written down.

It might seem like those steps are obvious and don’t need to be explicit. To people who believe that, I say that they just haven’t experienced the calming effect of a good list of steps. The steps become the artifact that records all of your learning. You can rearrange the steps and otherwise optimize the whole process in writing.

It’s such a relief when you start to do that. It’s very relaxing to know that every little detail is handled in the instructions. For example, you might not know you need a medium bowl until you read step three. But at that point, you might have your pan on the stove and need to stir constantly. Having all of the steps, including the meta-steps, listed out and in an order that works is freeing. Your mind can attend to the present step.

At work I’ve been working through this same thing with project planning. We’ve got a feature we want to build. We discussed the details of it. Now it’s time to make a project in Linear to track it. There are two questions that I’m wrestling with: How much granularity do you want? And how much of the meta-work needs to be written down?

Too much granularity and things get a little too prescriptive. In work like programming, breaking the work down into smaller chunks and writing a description of those chunks is actually part of the work. And if your chunks are too small, you’re not taking into account the realistic uncertainty every programming project has about exactly how it should work. However, don’t break it down enough and you’ve got a single task: Implement the feature. Finding that middle path is part of the art of project planning.

But should planning the project be written down as the first step in the project? Sometimes I think yes. If the planning is not obvious (where you can just do it then and there), yes, you can write the first step is “Break the project into steps.” And if your planning process is more structured with multiple steps (like getting approval, etc.), write those down, too.

The default setup in Linear is to have multiple statuses for each task. Backlog→In-progress→In review→Done. But I find that code review where I work is hefty enough to merit its own step. So I put it. Meta-work is work when it requires effort. For instance, I wouldn’t put a step in the project saying “change status of step 2 to Done.” That seems weird and a little too meta. But asking for a review, following up until they do it, addressing the comments, then following up again until it’s approved is real work. That gets its own step.

It reminds me very much of the Getting Things Done methodology where they recommend writing down the “next action” to take. It should be precise enough to actually accomplish. For instance, if you need to call a plumber, but you don’t know a plumber, the next action might be “ask your friends for plumber recommendations.” Better yet would be to list the friends.

In the same way, we want each step to be clear. If the project requires research before you know what steps to take, write “Research x.” You can add more steps as things become clear. The point is to capture the steps you do know and feel a sense of progress when you check them off.

It’s all work, meta or not. The real trick is that planning is a way of manipulating the future. You write down steps, you change them, you rearrange them, you erase them, all before you take the first step. Some steps are known and fixed. Merging to main comes last. It’s pegged at the end, while some are fluid and can be rearranged, done in parallel, or even eliminated if they’re not necessary. The question is whether the meta-task warrants a place in the future among all the other tasks. I say we should default to saying yes more often. It helps us take the meta-tasks too seriously. For example, some teams may consider testing as a meta-task. By making it a task and writing it down, we honor its place among other tasks. And we honor ourselves when we complete it, giving us the satisfaction of a job done.

The talk I was watching was You’re Doing Exceptions Wrong by Matt Burke. In it, Burke presents some rather compelling advice on how to design your exception throwing and handling to make your code more robust and easier to work with. I quite agree with a lot of what he said, though I think there’s plenty of room for subtleties and context-dependent decisions. You can’t cover everything in a one-hour talk.

What struck me, though, was how easy exceptions are to get wrong. He goes over many examples where poor use of exceptions led to problems. Sometimes people caught exceptions and ignored them. Sometimes they threw the wrong exception. Etc., etc. There was a lot of advice to digest and apply. And you could still get it wrong.

It got me thinking about how many degrees of freedom there are when dealing with exceptions. What class of exception you choose? Should you throw? How much code you wrap your try/catch in. Do you catch at all? How specific do you make your catch statement’s selector? How do you rethrow? Do you need a finally? All of these were subtle and non-mechanical decisions. That’s a lot of cognitive work going on.

It reminds me a lot of the advice in Java Concurrency in Practice. It’s a great book. The advice is solid. I learned a ton about Java when reading that book. But it’s impossible to apply correctly all the time. It is 432 pages of dos and don’ts. Java’s design defaults to a sequential model of computation—one thread. To switch to a concurrent model, you have to start using a whole set of new conventions. You need to properly use the volatile and synchronized keywords, choose “thread safe” classes, and master a whole set of concurrency primitives. There are too many degrees of freedom.

Clojure solved this by reversing the situation: Make concurrency the default. What does that look like? Immutable data structures and concurrency primitives with simple contracts. You can still do sequential programming for those inner loops where you need it. But it’s more cumbersome. In short, Clojure chose the more general case (multiple threads) and constrained the degrees of freedom by choosing the right defaults.

I wonder if there isn’t something like that for exceptions. What would that look like? It would be about choosing the right, general-case default, reduce the number of decisions that have to be made, and making the few decisions left easy to get right. (Aside: This is something Clojure does particularly well.)

If we assume the default is the general case (every function is broken :), then we

The first thing I would do is to come up with a short list of scenarios that the exception can capture. Here’s a first stab:

• IllegalArgumentException — somebody passed me an invalid argument

• UnexpectedReturnValueException — I called a function and got something back I wasn’t ready for

• NoPossibleAnswerException — I was called with valid arguments but I can’t fulfill the contract

The main idea behind the first two is try to distinguish the source of bad values. The ubiquitous NullPointerException tells you you got a null, but not whether it was from an argument or a return value. I’m not as concerned about whether it was a null or some other invalid value (such as 0 for division) as much as where the bad value came from. These exceptions would require you to provide the value.

NoPossibleAnswerException is for situations where you want to throw an exception because you can’t answer. For example, if I ask for the configuration key “HTTP_PORT” but it’s not set, I may want to throw an exception instead of returning null. This is the exception for that case.

There are still other exceptions that might come up. I’m thinking about FileNotFound or TimeoutExceptions. I wonder where those go.

There’s another dimension by which we should slice the exceptions: Whether or not they should be retried. For example, if there’s a timeout on an http get request, go ahead and retry. But if you divide by zero, retrying is useless. If there’s one piece of information I’d love to have available programmatically, it’s whether the thrower thinks retrying is a good idea.

With these few pieces, you can build some primitives that help localize the errors:

(defn divide [num denom]
  (assert-argument denom (not (zero? denom)))
  (/ num denom))

assert-argument is a macro that throws IllegalArgumentExceptions if the condition is not true. It can use the form to report good error messages, including the name of the argument, its value, and the condition that failed. It helps document your assumptions about the arguments you will receive.

Similarly, you can document the assumptions about the return values you will get with assert-return.

Unfortunately, most existing libraries (certainly those in Java) do not follow these conventions. What has worked best for me is to build something like an Anti-Corruption Layer (as in Domain Driven Design). The anti-corruption layer wraps the library and converts the library’s conventions into the conventions of my codebase. It’s a place to say “this is how we use this library” and “this is what we expect this library to do”. It seems redundant at first. Why wrap each library function you call in another function that just calls the library function? Well, it’s to constrain and centralize the assumptions you’re making about the library. Instead of having workarounds for the library spread throughout the codebase, you centralize them and standardize them to the anti-corruption layer.

Before I conclude, I want to mention a different yet similar approach. Erlang is famous for its “let it crash and retry” strategy for handling errors. It’s similar in that Erlang assumes anything can fail (the default is something will go wrong). And the default strategy is to reset the state and try again. This often works, especially with stateful systems. I think it’s a great default.

The problem I have encountered with it is that you do need a certain level of correctness for retries to be effective. If your code doesn’t handle the null you got, retrying it is not going to fix that. You would like to surface bugs early (hopefully during development) so you can fix them. In those cases, you don’t want to hide the problem with a retry. However, we all know that bugs are inevitable. So being protective and having a decent default strategy is prudent. Retry seems to be pretty good to me as a default. So it seems we may want different behavior in production from during development.

In development, crash early so we can surface the errors. In production, do your best to continue without errors. For instance, if the product recommendations don’t load on your product page for a new tea kettle, you should still load the page, just without recommendations. That way, the user can still buy their tea kettle. But don’t do that during development because if you break the recommendation engine, you want to know.

The tension between these two behaviors shows that there are still significant decision points a programmer will need to make. The key is to make the decision easy to get right. That means reducing the number of options and making each option easier to do right. We want to standardize and reduce boilerplate.

Let’s conclude. Exception throwing and handling has a lot of degrees of freedom—too many to get right all the time. It’s ironic because the error reporting and handling will have errors! We can apply Clojure’s design principles of using the general case as the default. In this case, it’s that the function you’re calling can’t be trusted fully, and the caller of the current function can’t be trusted. You have to check your arguments and the return values. But checking them is a lot of work and easy to get wrong. We need better primitives to make it easy to get right. I proposed a systematic evaluation of the kinds of errors you want to throw, and making those easy to throw. I talked about an anti-corruption layer to enforce conventions when using third-party libraries. And I talked about wanting different behavior in development and production. I don’t think I got it all right. So please let me know what your policies are in your software. And I’m also interested in knowing in what other areas (besides exceptions) you apply the policy of “general case is the default”.

REPL-Driven Development and Learning Velocity

The main advantage of Lisps (including Clojure) over other languages is the REPL (Read-Eval-Print Loop). Lisp used to have a bunch of advantages (if statements, garbage collection, built-in data structures, first-class closures, etc.), but these are common now. The last holdout is the REPL.

The term REPL has diluted, so I should define it: A REPL is a way to interactively inspect and modify running, partially correct software. My typical workflow is to open my editor, start the REPL, and start the application server from within it. I can make requests from my browser (to the running server), recompile functions, run functions, add new libraries, and inspect variables.

The REPL accelerates learning by increasing the speed and information richness of feedback. While programming, you learn about:

• Your problem domain

• The languages and libraries you’re using

• The existing codebase and its behavior

The REPL improves the latency and bandwidth of feedback. Faster and richer feedback helps you learn. It lets you ask more questions, check your assumptions, and learn from and correct your mistakes. Fast, rich feedback is essential to achieving a flow state.

The obvious contrast with REPLs is with the mainstream edit-compile-run (ECR) loop that most languages enable. You edit your source code, run the compiler, and run the code. Let’s look at the main differences between REPL and ECR:

• In ECR, your state starts from scratch. In REPL, your state is maintained throughout. All of the variables you set up are still there. Web sessions are still open.

• In ECR, your compiler may reject your program, forcing you back into the Edit phase. Nothing is running, so you must fix it before continuing. In REPL, when the compiler rejects your change, the system is still running with the old code, so you can use runtime information.

• In ECR, if you want to try something out, you have to write an entire program to compile and run. In REPL, trying something out means typing the expression and hitting a keystroke.

The end result is that the ECR loop is much slower than the REPL. One of the benefits of modern incremental testing practices (like TDD) is that it approximates the fast feedback you get from the REPL:

• With testing, you do not maintain your state. However, you write the code to initialize the state for each test.

• With testing, you make small changes to the code before rerunning the tests so you are usually not far from a running program.

• With testing, it is easy to add a new test and run just that.

The advantage of testing is that you have a regression suite, which you don’t get from the REPL. But feedback in testing is poorer. I haven’t heard of anyone writing a test just to see what the result of an expression is. And, by the way, doing incremental testing with the REPL is a breeze. It’s like the best of both worlds.

The REPL gives you fast, rich feedback in three main ways:

• Maintaining state — Your running system is still running, with all in-memory stuff still loaded. This means that after editing a function and hitting a keystroke, you can inspect the result of your change with a delay that is below human perception.

• Running small expressions — Within your running system, you can understand the return value from any expression you can write, including expressions calling your code or libraries you use. The cost of running these expressions is below a psychological threshold, so they feel almost free compared to having to scaffold a test or a public static void main(String[] argv).

• Ad hoc inspection of the running system — This is a big one you gain skill with over time. You can do anything you can imagine, from running your partially completed function (just to make sure it does what you expect) to printing out the value of a global variable (that you saved values to from your last web request). The flexibility makes tools like debuggers feel rigid.

However, other languages have chipped away at the advantages of REPL-Driven Development. I already talked about incremental testing approaches (like TDD) and how they approximate the feedback of the REPL. But there are more technologies that provide good feedback in the mainstream ECR (Edit-Compile-Run) paradigm:

• Static analysis — You can get feedback on problems without leaving the edit phase with tools like LSP and squiggles under your code.

• Static types — If you’ve got good types and you know how to use them, the Edit-Compile cycle can also give you rich feedback. The question is whether your compiler is fast enough to keep up.

• IDEs with run buttons — Many IDEs for compiled languages use their own, incremental compiler. The code is constantly being compiled as you edit. When you hit the Run button, you’re essentially cutting out the Compile phase (which can often be very lengthy). If you can set it up to run a small expression at a keystroke, you’re very close.

• Autocomplete — Autocomplete speeds up the Edit phase. Autocomplete with the REPL is a cinch. You can inspect what variables are available in the environment dynamically. However, modern IDEs can use static analysis to aid autocomplete.

• Incremental testing — Incremental testing (like TDD) speeds up the Edit and Run phases. Added here just for completeness.

But don’t tell anyone: we can use these in addition to the REPL. In fact, Clojure has an excellent LSP and a great testing story. The only thing we don’t have is a great story about static typing.

Many languages claim that they have a REPL. But what they really have is an interactive prompt where you can type in expressions and get the result. This is great! But it doesn’t fully capture the full possibility of the REPL.

People often ask what’s missing, especially from a very dynamic language like JavaScript. I’ve tried to set up a REPL-Driven Development workflow in JavaScript, but I encountered these roadblocks. There are probably more:

• Redefining const: In Clojure, we pride ourselves on immutability. However, global variables are mutable so that we can redefine them during development. Unfortunately, JavaScript engines are strict about global variables defined with const. I found no way to redefine them once they were defined.

• Reloading code: It’s not clear how to reload code after it has changed. Let’s say I have a module called catfarm.js and I modify one of the functions in it. My other module old-macdonald.js imports catfarm. How do I get old-macdonald to run the new code? The engines I tried did not re-read the imported modules, instead opting for a cached version. In addition, even if they did, the old-macdonald code needs to be recompiled. Clojure’s global definitions have a mechanism to allow them to be redefined, and any accesses to them after redefinition are immediately available. If I recompile a function a in Clojure, the next time I call b (which calls a), it calls the new version of a.

• Calling functions from other modules: When you import a module, it’s basically a compiler directive where you say what you’re importing. But how do you call things from that aren’t imported? Why would you do that? Because you want to try them out! This, combined with not being able to re-import them, makes it really hard to incrementally work on your code. In Clojure, require is a function that loads new modules. And in-ns is a function that lets you navigate through the modules like in a directory structure from your terminal. require() in JavaScript (the old way of doing modules) worked more like that.

Hot module reloading is an attempt to address these limitations. It also seems like a major project with a lot of pitfalls. And it still doesn’t maintain state. Maybe it will get good enough one day to be close to REPL-Driven Development. Or maybe Node should have a “REPL” mode where const variables can be redefined and modules can be reloaded and navigated.

The biggest problem with REPLs is that they require an enormous amount of skill to operate effectively. To know what you need to recompile, you must understand how Clojure loads code and how Vars work. You need to navigate namespaces. Most importantly, you need to develop the habit of using the REPL, which is not built-in. It’s common for someone in a beginner’s chat to ask “what happens when I call foo on a nil?” My first reaction is: “Why are you asking me? Instead of typing it here, type it in the REPL!” People need to be indoctrinated.

Here are some ways to improve your use of the Clojure REPL:

• Next time you’re writing a function, use a rich comment block ( (comment …) ) to call that function. After the tiniest of modifications, call the function to inspect the return value. This is useful when writing long chains of functions.

• The next time you’re tempted to look up documentation for a function, do a little experiment to see what happens at the REPL. Some common questions to answer:

  • What type does a function return?

  • Can I call that with an empty value?

  • What does the zero-argument version do?

• Learn the keystrokes in your editor to evaluate:

  • The whole file

  • The current top-level form (often referred to as defn)

  • The expression just before the cursor

• Learn the keystrokes to run the tests from your editor. Run your tests. Edit and compile functions (see previous bullet). Run the tests again. This combines the best of incremental testing and RDD.

I love the REPL. I miss it when I go to other languages. The REPL is also forgiving. It can make up for a lot of missing tooling (like autocomplete and debuggers). The REPL also requires a lot of skill. Squiggles and LSP can immediately give you hints for making your code better. But the REPL requires a deep understanding of the language and lots of practice. This is ultimately the biggest barrier to its adoption. People who haven’t learned how to use the REPL don’t even know what they are missing.

PS: You can learn the art of REPL-Driven Development in my Beginner Clojure signature course.

Beginner Clojure, my video course, is better than ever. I recently completely rebuilt the Introduction to Clojure module. It’s the fastest way to get from zero to a deep, functional programming and data-driven programming experience. Go check it out to see why hundreds of people have enjoyed it. If you buy, you get lifetime access plus all updates. Planned future updates include VS Code with Calva, The Clojure Command-Line, and Shell Scripting with Babashka. The course already contains eight great modules, including JVM, Repl-Driven Development, and functional programming. Buy it today.

Crafting your environment

My Emacs setup is boring. So is my shell prompt. I’ve usually left things vanilla. Sometimes I look at other programmers and their awesome tooling setup and their colorful zsh prompts and feel superior. I tell myself that I am serious. I don’t waste time with silly things. Instead, I get to work. I’m very conservative.

But over time, I’ve heard of enough arguments for extreme configuration and seen enough examples of powerful IDE setups and missed enough tools that are now standard that I wonder if I’m wrong. Is it a waste of time? Or is it time invested? I’m starting to like the other side, though I’m still very bad at it.

Let’s start with the argument for why it’s not really worth your time to improve your tools.

Being slightly faster does not pay off

Sometimes I see a programmer obsessing over their setup. They’re spending hours reading forums and getting their Emacs config just the way they like it. They get into building their own keyboard. They figure out the best chair to sit in. It’s not so much that it doesn’t matter at all, it’s that it seems like a huge distraction. You want to be better at programming, so you spend hours 3D printing your keycaps? It seems to me like a big waste of time.

Let’s calculate it. If you spend 10 hours making a keyboard (which is very conservative), and it makes you 1% faster at typing, and we assume you spend 1 hour per day typing, you will need to type for 1,000 hours before you’ve made up the difference. That’s 2.7 years. That doesn’t seem worthwhile to me.

Typing speed is not the bottleneck

One of my friends put it this way: Typing speed is not the bottleneck in programming. They’re evoking a principle from the Theory of Constraints which is very popular in manufacturing circles. The principle says that every system has a bottleneck, and improvements to anything besides the bottleneck will not improve the overall productivity of the system. The bottleneck is the limiting factor. According to my friend, typing speed is not the the limiting factor to your effectiveness as a programmer.

All that time spent learning another vim command won’t really make your programming faster. You’re still limited by the bottleneck, which I agree is not typing speed.

Customization is not shareable

And just one more issue with ultra-customization. One time I was pairing with my coworker on his laptop. I was having trouble communicating something he should try, so he handed me the laptop. I put my hands on the keys. And every key I pressed surprised me. I was skilled at Emacs. And this was Emacs. But this was not the Emacs I knew.

He had customized the basic Emacs commands to different keys. And the official keys were remapped to other commands. They made sense to him, but I was useless on his keyboard. That really made me double down on leaving keys standard. With non-standard keys, sure, you may be faster, but nobody else can use your computer.

Do you agree with these? Are there other arguments?

I once agreed with these, but not any more. These three arguments are missing something important.

Being slightly faster is not the point

The reason the “efficiency argument” doesn’t hold water is that it’s not really about being faster. If all you’re looking at is productivity such as lines of code per second, you’re missing the main point. I talk about this a lot in The Surprising Benefits of Automation. We improve our tools and skills for agency and efficient use of cognitive resources, which we’ll talk about soon.

Typing is on the critical path

My friend evoked the concept of the bottleneck from manufacturing (note the efficiency mindset). Another concept from manufacturing and project management is the critical path. This is the path of sequential steps that affect the timeline of the project if they take more or less time. Typing might not be the bottleneck, but it is on the critical path, especially of iterative development practices like REPL-Driven Development (RDD).

I think what my friend meant when he said that typing is not the bottleneck is that programming is mostly about thinking. You might spend two hours thinking and ten minutes typing it in. Taking eleven minutes to type it in isn’t going to seriously change the timeline.

However, when I’m doing RDD, the speed I can type a piece of code, eval it, and understand its effect is part of the thinking process. With RDD, we extend our mental capacity with the REPL, just like a geometer extends their mental capacity with a compass and straightedge. The faster you can go through the loop, the better integrated your thinking and the REPL can be. Slowing down your typing can significantly affect that integration. And speeding it up with better configuration can help it, too.

And typing speed becomes more important the more iterative and incremental your process is. If you’re doing waterfall, your process is dominated by thinking, planning, design, and redesign. But if you’re working iteratively, you are typing a lot and learning from the result. The faster you type, the faster you learn.

Customization is not shareable

Well, this one I think is still huge. But as Juan Monetta opined on our recent episode of Apropos, that is much less important when we’re all programming on our own machines. We share a lot of development tooling: Linter configs, formatter configs, and tests. But do we need to share editor configurations? I’ve heard it both ways. I’ve heard people who love pairing and process argue that the team should be discovering as a group the best way to work as a team—meaning they should use the same editor configuration. But I’ve also worked at places that took an individualistic approach, so much so that the variance of productivity and understanding was huge. Some people knew how to run the code at the REPL, and others thought it was impossible. More on this later.

Now that we see that it’s not about speed but about cognitive resources, what are some arguments for configuration?

Energy and joy

Let’s say you open up a drawer in your kitchen. It’s filled with kitchen tools, but it’s a cluttered mess. It’s frustrating. You dig around, find the spoon you need, and close it. You get on with your cooking task. Or you could stop what you’re doing and organize the drawer. Toss out broken tools. Create sections and sort the tools that you still use. How do you feel afterwards? I bet you feel energized. The frustration has turned to joy.

The fact is, feeling control over your environment is important psychologically. That frustration is a signal that you want something to change. Ignoring that signal takes energy. Resigning yourself to live with frustration is a sure path to despair and burnout.

Energy is important. You can’t measure everything based on time efficiency. Time efficiency could be important, but it’s not everything. A programmer’s good energy improves their agency and their productivity. If a programmer feels like they can’t improve their environment, they’re more likely to zone out at work. You might as well use that zone-out time to make some improvements.

Cognitive resources are a bottleneck

My friend was correct that thinking is the bottleneck. Each of us is endowed with a certain amount of “cognitive resources” to work with. Research shows that this is a limited and shared resource—there’s only one pool of it. So if your roof is leaking and you’re worried about it, that will affect your programming at work. The bottleneck is the cognitive resource pool.

Your resources are also drained by tasks that take effort. Imagine if you had to search for the { key on the keyboard each time you needed to type it. Sure, it would slow you down. But worse than the slowdown is that you would use up precious cognitive resources. You would have fewer resources to think about the problem. If it took you long enough, you might even forget what you were trying to do. In short, it’s not slow typing that is the problem, but typing that takes effort.

How many of your subskills take effort because you haven’t mastered them yet? Maybe you can type { without thinking about it. But what about figuring out which braces to close? An easy solution is to use paredit—but that requires configuration and learning new keystrokes. Well worth it in my opinion. But what other packages could unlock similar improvements?

Experimentation and waste

I try to read a lot of books on programming. Learning from others is a great way to get better. Right now, I’m focused on books on software design. Unfortunately, most of the books on software design have bad advice. Does that mean it’s a waste of time to even look? Hour-for-hour, I don’t think it’s worth it. My software design skills could not have improved by that much. I have spent thousands of hours over my career reading programming advice. But it probably has not led me to ship everything thousands of hours faster (even in aggregate).

But how can we know? I certainly feel like the time spent reading was worth it, even considering that a lot of the advice was bad. When I look at my skill vs someone who doesn’t read, I can see the difference. When I look at my skill vs myself 10 years ago, having read fewer books, I can see the difference. But I can’t attribute any improvement to any particular piece of advice.

While you’re reading, you are absorbing other people’s experience. You’re organizing your own experiences. Even when you read something you don’t agree with, it could give you a new perspective on that idea. Reading alongside direct experience helps you learn.

Likewise, when I see someone building their fifth custom keyboard, it feels like a waste of time. Do you really think you’re going to get that much better this time? But they think it’s possible. And even if the keyboard doesn’t make them better, there’s something else going on: Engagement in the process. Building your own keyboard is a physical way to engage your brain. What keystrokes are important to my process? How do I want that key to feel when I press it? What fingers do I want to use? The learning transfers to configuration and typing.

You cannot program 100% of the time

Another misconception is that configuration time is eating away at programming time. If you’d give up that time spent configuring, you’d have more time to program. But that’s not how people work. We need breaks from the productive work. The productive work makes a mess, and we need to clean up. The productive work gives us ideas for doing our work better, and we can use those ideas to improve our work.

Permission to make small improvements

If you want to engender resentment and passivity in your programmers, tell them they can’t configure their tools. They’ll probably say they agree with you that it’s a waste of time. But you’ve also tapped into the efficiency-seeking part of the brain. That efficiency-seeking part of the brain isn’t looking to get more done with less. No, it’s about doing as little as they can get away with.

If you want to activate someone’s brain to look for ways to improve, encourage (or require!) them to make small changes daily. Some companies require each employee to make a change that could save them 2 seconds. Setting the bar low makes it achievable. But it also gets the mental balls rolling. With each small change, they develop a sense of ownership over their process and that engenders agency—which is probably what you want in an employee.

Improvements compound

Some tweak to your process might seem small. But over time, the work with each other to improve non-linearly.

Working at the right cognitive level

Again, typing speed is not the bottleneck, but thought is. If you are moving a cursor around, backspacing, and typing, you’re thinking about the cursor and characters. Another person, however, could be issuing higher-level commands, like rename symbol, if they had the right configuration and practice with the keystrokes. There is less distance between what they want done and the command to do it. It’s not about speed. It’s about the cognitive processing it takes to translate the desired action into editor commands.

Improvements reveal underlying problems

As workers at Toyota eliminated inventory between steps, the assembly line would start to break. They would say: “We told you that we need that inventory. Otherwise the process doesn’t run smoothly.”

Taiichi Ohno didn’t agree: “We are lowering the level of the water in the river (the inventory) to reveal the stones (the problems). The stones were always there, we just couldn’t see them.”

Taiichi Ohno knew that inventory tends to pile up before the bottleneck. Since the bottleneck is the slowest step in the process, the fast steps before it produce more than the bottleneck can handle. But when he looked around, there was inventory everywhere. By eliminating the easy piles of inventory, he could see where the real problems were.

We can do that in our tools, too. Our tools are full of frustration. We can start anywhere, fixing problems as we see them. This will make the process smoother up to a point. But the better our configuration is, the easier it is to see the parts that still frustrate us. Focusing on the bottleneck is good in theory, but it’s hard to recognize the bottleneck when everything feels frustrating.

Fixing our frustrations leads to new projects

If we don’t fix our frustrations, we have a tendency to become blind to them. We get used to them and work around them. But these frustrations can be very valuable. Frustration with existing debugging approaches (like println) fueled the development of FlowStorm. And Tailwind CSS’s creator Adam Wathan credits his sensitivity to frustration for the development of Tailwind. If you’re feeling frustrated, chances are, other programmers do, too. Your solution could be valuable to them.

Discontinuous improvement

There’s a phrase “continuous improvement”, which sounds great. You’re making improvements all the time. But I also think there’s something like discontinuous improvement, too. You see, if our cognitive resources are the bottleneck, there’s a discontinuity at the edge of our cognitive capacity. Jobs just bigger than our capacity need some kind of process and externalization (like a todo list) to manage it. But jobs small enough to fit in our minds don’t. If you can cross that boundary with better skills and tooling, you’ve got a stepwise improvement.

Aesthetics are important

When I see people geeking out over their keyboards, it’s not just about function. People choose colors and lights and other design elements that please them. The same for a lovely shell prompt or a nice font in their editor. The aesthetics of your environment plays a noticeable role in your productivity. If your office is a dump, you’ll probably make trash software. But if you surround yourself with beautiful things, you’ll be inspired to do beautiful work.

Now let’s talk about some principles that can help make the most of this learning.

Manage resources

The most important principle is that we are trying to manage limited cognitive resources. You can:

• Make a tool more pleasant so that it gives you more resources (e.g., make it have nice UX)

• Use tools that operate with the right concepts (e.g., paredit)

• Learn a skill better so that it takes fewer resources to perform (e.g., practicing paredit commands so they are muscle memory)

Choose your battles

Alan Kay talks about the problem with a good engineer is they are so dissatisfied with everything that they want to build it all themselves. It takes great discernment to choose what you build on top of. Likewise, improvements do have a cost. Are you really going to hand-craft your keyboard? Might you want to try one of the existing commercial ones first? It might get you 99% of the way there.

Configure away, but use standard keystrokes

Emacs has a set of keystrokes that come standard out of the box. They have been the same for decades. C-a goes to the beginning of the line. C-e goes to the end. C-x-f opens a file. And, believe it or not, there are many packages for opening files. So while you can change the way you open files, most packages keep the same keystroke. Even though the behavior is different, someone familiar with the keystrokes can still use it. I recommend that approach.

Share when you can

I’m not as extreme as my friend who wanted everyone at the company to program with the same tools. But I also think most companies don’t encourage enough sharing of tooling and process. Pairing and watching people work can help spread the great tools and skills that people have set up. But you can also just talk about it. Have a monthly meeting where someone shows off their editor.

AI can help you configure

I don’t use AI to generate production code. But I’m leaning on AI now to help me configure my tools. Even though I’ve used Emacs for years, I’m terrible at programming it. I have to admit that it’s one of the reasons I haven’t done more configuration. Likewise for packages. There are just so many out there. I don’t have the patience to try them all. But with AI, I can ask for package recommendations and for tips to configure it how I like it. It’s really helping.

If you want to learn Clojure, there’s no better way than Beginner Clojure, my signature course for those new to the language. I’ve recently rebuilt the introduction module to teach better, so there’s never been a better time to buy. The point of that module is to give you a deep experience with Clojure (and FP+Data-Driven Programming) as quickly as possible.

Models are messy

I am not a Platonic Idealist. I do not believe that the round table I’m working at is some pale shadow of the ideal circle. At best, circle is a mathematical abstraction that’s easy to define and analyze which happens to approximate the real table well enough to be useful. It is the abstract circle that is the pale shadow of the real table. Circles are just ideas.

So I guess that makes me a Materialist. But as I write this, I realize how much of a process it was for me to arrive at that conclusion. And even describing it took some work. And by learning that about myself, I realized a profound truth: Many people are idealists unconsciously.

One problem is that materialism is often conflated with anti-intellectualism. If the material world is primary, maybe it calls into question studying the idealized world of geometry and algebra. Maybe math isn’t important. But, no, it doesn’t. Materialism acknowledges that geometry and algebra are simply useful mental tools—not the ultimate essence or the source of all truth.

I sometimes wonder, as one does, about the marriage between empirical experiment and mathematical abstraction that has led to the recent explosion of scientific understanding. Why is it that mathematics has been so fruitful at describing the world we discover through experiments? Mustn’t mathematics be a fundamental part of the universe? Is mathematics discovered? Or is it invented by us?

Those are fun questions, but I’m not going to explore them deeply here. I will simply say that it was Buckminster Fuller who got me thinking about all of this stuff. He said there are no zero-dimensional points in the real-world. There are no one-dimensional lines. There are no two-dimensional planes. Everything we can experience is three-dimensional. So he didn’t base his geometry on points and lines and planes. Instead, he always used real objects to explore the nature of space. I still think about that. He went back to first principles and built up an understanding based on empirical exploration instead of abstractions.

The world is messy. Everything is complicated. When I talk about creating a domain model, one of the common pushbacks I get is from people who worry I’m advocating for creating an over-idealized model. One that sounds good but misses all the rich detail of the messy world.

Nothing is further from the truth. In fact, your domain model should enable all of the messy details that you need to capture. The level of abstraction most mathematics achieves is too clean. However, abstraction is the basis for how software works. Software for managing a farm does not operate on real sheep. Yet the sheep are represented.

No, what I’m suggesting is that we must represent the structures we encounter in the world in the software. The structure is often messy. But there is often a deeper, underlying order to it, as in my example of combining simple Boolean predicates to build up a representation of the messy voting laws.

Many real world “laws” are actually rules that apply most of the time but they have a number of exceptions. That itself is the structure we must capture—general rules with exceptions. Many real-world processes are mostly continuous—they’re continuous with a known discontinuity. That’s fine. Those discontinuities can often be handled at a higher level. It’s kind of the complement of finding order at a lower level. Move the messiness to a higher level.

It reminds me a lot of the way I recommend people deal with algebraic properties in property-based testing. People often get caught up looking for extremely regular properties, like the commutativity of addition.

What I recommend is using the formula of commutativity as a starting point—a kind of inspiration.

Ask about whether the order of arguments matter, how they matter, when they matter. You could find some rich patterns there that are worth capturing as properties for your domain.

For instance, merge for maps is commutative if you know the two maps don’t share any keys. So you start with the basic formula and you adorn it with the real-world detail you’re looking for.

Likewise, the order of checks arriving at the bank doesn’t matter—as long as you have enough balance for all checks to clear:

let bill be the $x check to for the power bill (debit to your account)
let paycheck be the $y weekly paycheck (credit to your account)

That is, if you’ve got the money to pay the bill, it doesn’t matter which order the checks are cleared. But if you don’t have the money and the bill clears first, you might get an overdraft fee.

Remember, commutativity is just a nice idea that corresponds to some things we see in the world. We can adapt it, like any mathematical notion, to make it useful to our purposes. Putting the simpler definition of commutativity ahead of what we encounter in the real world is latent Platonic idealism. We have to reject that idealism. The real world is where the action happens. And its rich detail is what makes it interesting enough to model in software. When you’re building a model, avoid the temptation to make a perfect gem. Instead, aim to honor the imperfections. But don’t be an anti-intellectual and reject modeling outright. Modeling is what enables the magic we wield. So are you an idealist of a materialist—or an anti-intellectual?

I have finally released the new version of Introduction to Clojure, my flagship module in Beginner Clojure, my signature video course. This update is long overdue, but it makes up for its tardiness with fully updated content, modernized for current idioms and better teaching.

If you have the previous edition, you can already find the new edition in your dashboard. You get the upgrade for free as my thanks for being a part of this crazy journey.

If you buy Beginner Clojure now, you’ll also get the new version. Because it’s such a major upgrade, I’m going to raise the prices soon. If you want it, now is the time to buy. It will never be this cheap again.

Small modular parts

I’ve been seeping in the rich conceptual stews of Patterns of Software. In it, Richard Gabriel explores how the ideas of Christopher Alexander apply to software engineering (long before the GoF Design Patterns book). One of the early ideas in the book is that of habitability, the characteristic of a building to support human life. Architecture needs to provide the kinds of spaces humans need, and also be adaptable to changing life circumstances. A house must support time together, time alone, and time alone together (sharing a space but doing different things). But it also must allow adding an extra bedroom as your family grows.

Habitable software is analogous. Programmers live in the code. They must feel comfortable navigating around, finding what they need, and making changes as requirements change. Christopher Alexander says that it is impossible to create truly living structures out of modular parts. They simply don’t adapt enough to the circumstances.

However, we know that’s not entirely true. Bricks are modular parts, and many of the living examples he gives are buildings made of bricks. It must be that the modules need to be small enough to permit habitability. You can’t adjust the size of a wall if the wall is prefabricated. But you can adjust the size of the wall if the bricks are prefabricated to a resolution that is just right.

This is true in software as well. Large modules are not as reusable as small ones. Take classical Java. I think the language gets the size of the abstractions wrong. The affordances of the language are the for/if/… statements, arithmetic expressions, and method calls, plus a way to compose those up into a new class. It goes from the lowest level (basically C) to a very high level, with very little in-between.

Contrast that with Clojure, which gives you many general-purpose abstractions at a higher level than Java (map/filter/reduce, first-class functions, generic data structures), and then almost nothing above it. Just the humble function to parameterize and name a thing. Lambda calculus (basically first-class functions) goes a long way. Java’s methods and classes give you a way to build procedural abstractions over data storage and algorithms, but the language offers torturous facilities for control flow abstractions. First-class functions can abstract control flow. I think Clojure got the level right.

Except maybe Clojure’s standard library overdoes it. I’m a big fan of map/filter/reduce. You can do a lot with them. But then there are others. For instance, there’s keep, which is like `map` but it rejects `nil`s. And there’s `remove`, which is the opposite of `filter`. Any call to keep could be rewritten:

(keep {:a 1 :b 2 :c 3} [:a :b :c :d])

(filter some? (map {:a 1 :b 2 :c 3} [:a :b :c :d]))

Those two are equivalent. `remove` can also be rewritten:

(remove #{:a :b :c} [:a :b :c :d])

(filter (comp #{:a :b :c}) [:a :b :c :d])

I do use keep and remove sometimes, when I think about them. But how much do they really add? Is the cost of learning these worth it? How often do you have to switch it back to map and filter anyway to make the change you want?

Here’s what I think: keep is just slightly too big. It’s a modular part that does just a tad too much. map is like a standard brick. keep is like an L-shaped brick that’s only useful at the end of a wall or on a corner. Useful but not that useful, and certainly not necessary. The same is true of remove. It’s not useful enough.

I think Clojure did a remarkably good job of finding the right size of module. They feel human-scale, ready for composition in an understandable way. It makes programs of medium to large size feel more habitable. I see this about what little Smalltalk code I’ve read: Smalltalk’s classes are small, highly general modular units, like Point and Rectangle, not UserPictureDrawingManager.

One aspect of habitability is maintainability—the de moda holy grail of software design. Back in 1996, when Patterns of Software was published, Gabriel felt the need to argue against efficiency as the reason for software design. Somewhere between efficiency’s reign and today’s maintainability, code size and then complexity ruled.

Long-time readers may guess where I’m going: These characteristics all focus on the code. Abstraction gets talked about in terms of its (excuse me) abstract qualities. An abstraction is too big or small, too high- or low-level, too shallow or deep. At best, we’re talking about something measurable in the code, at worst, some mental structures only in the mind of the guru designer who talks about it.

I want to posit domain fit as a better measure—one that leads to habitability—and that is also objective. Domain fit asks: “How good is the mapping between what your code represents and the meanings available in the domain?” That mapping goes both ways. We ask both “How easily can I express a domain situation in the code?” and “How easily does the code express the domain situation it represents?” Fit covers both directions of expressivity.

I believe that domain misfit causes the most difficulties for code habitability. If your code doesn’t fit well with the domain, you’ll need many workarounds. Using reusable modules is only a problem because they don’t adapt well to the needs of your domain—not because they’re too big. It just so happens that bigger modules are harder to adapt. It’s not that a wall module is bad, per se, just that it’s almost never exactly the right size, and so you make compromises.

It’s not that the components in Clojure are the right size. It’s that Clojure’s domain—data-oriented programming—is the right size for many problems. It allows you, the programmer, to compose a solution out of parts—like bricks in a wall. And Clojure’s code fits the domain very well. Tangentially: It makes me wonder what the domain of Java is. I guess what I’m saying is that using a vector graphics API to do raster graphics is going to feel uninhabitable. But you can’t say it’s because vector graphics is a bigger abstraction than raster. It’s more about having the right model.

Now, Alexander might disagree that a pre-fab wall of exactly the right size is okay. He believes that there’s something in the handmadeness of things, too. It’s not just that the wall is the wrong or right size. Even if it were perfect, the perfection itself doesn’t lend itself to beauty. Geometrically precisely laid tiles cross some threshold where you don’t feel comfortable anymore. Ragged symmetry is better. We want bricks but they shouldn’t be platonic prisms.

So this is where I conclude and tease the next issue. I started this essay thinking size was important. I thought that Clojure got it right by finding a size of composable module that was a sweet spot. But now, I think it’s not about size. I don’t even know what size means anymore. It’s more about domain fit than ever. Perhaps I’m digging in deeper to my own biases (and please, I’m relying on you the reader to help me realize if I am). But this is what my reading is leading me to—the importance of building a domain model. When we talk about domain models, we often think of these jewel-like abstractions with perfect geometry. But this is a pipe dream. Our domains are too messy for that. In the next issue, I want to explore the dichotomy of geometric and organic adaptation.

Have you seen Grokking Simplicity, my book for beginners to functional programming? Please check it out or recommend it to a friend. You can also get it from Manning. Use coupon code TSSIMPLICITY for 50% off.

Skyscrapers or mud huts

From 2005-2007, I lived in the West African country of Guinea. I loved the people, but their country had a sad story. After rejecting their French colonizers, they made a series of bad deals with other countries. While they started out as the hope of West Africa due to their wealth of natural resources, after 45 years of kleptocracy, the infrastructure had decayed. While Guinea’s neighbors saw increasing prosperity, theirs was declining.

One of the symptoms of their decline was that they were de-urbanizing. In other countries in the region, people were leaving the village, looking for a better life in the cities. But in Guinea, apparently the better life was in the village.

I would often ponder this on my infrequent trips to the capital, Conakry. Large portions of the streets were broken or missing, revealing mud that could swallow a small car during the rainy season. Utility poles were a rats-nest of spliced wires. Probably most of the wires were dead. Roofs leaked. Many inhabited ten-story highrises had no electricity or running water.

Contrast this to the village where I lived. Yes, they were poor, but I never saw squalor. What I came to discover was that the village was simply easier to maintain by people with few resources. You could make a house out of mud bricks and thatch it with straw you harvested from your field. If the roof leaked, you could climb up and patch it. If a wall crumbled, you could repair it.

In the city, the tall buildings require a huge number of resources to maintain. You need cranes and bulldozers and other diesel machines. You need cement and iron forges and other high-input materials. You need a functional supply chain to get the diesel and other inputs to you. And you need the highly skilled labor to do the repairs. Don’t forget a functioning economy to pay for those workers. In short, modern urban life, which is very attractive when it works, is hard to maintain.

Even if you aren’t talking about repairs, the cost of keeping the building working is expensive. You need to pump water up to the top floors. If the building is tall enough, you need to pump clean air up there as well. And you need to deal with all of the waste products of life. Things have to keep working or the top floors aren’t usable anymore.

When programming, sometimes I wonder if our high-power tools, like Clojure, cause us a similar issue. One person can create a wondrous indirection mechanism that solves the problem at hand. It’s like erecting a skyscraper. Sure, it might be easy to write the code—easier than construction. But there is a certain mental cost. The complexity of understanding it requires the mental equivalent of cranes.

Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it? — Brian Kernighan

This quote from Kernighan (though I believe the sentiment comes from before his time). It expresses succinctly the idea I’m trying to convey. Clojure gives us the power to build things that are too clever to debug, just like a surge of political will can create a skyscraper that you don’t have the ability to maintain. Is maintaining a skyscraper twice as hard as building it?

I’ve worked on codebases where the cleverness was too much to maintain. The history of the project was a super smart programmer being brought onto the team to solve a tough problem. They built a brilliant solution. Then they left to use their brain somewhere more worthy of their mental capacity. The team enjoyed their skyscraper while it slowly decayed.

Richard Gabriel wrote a wonderful essay in Patterns of Software about the habitability of our code. In it, he argues that habitability is the important quality of good software. Programmers work inside the code, making adjustments, improvements, and expansions. We need to feel at home there—and like we can take care of it ourselves.

What is important is that it be easy for programmers to come up to speed with the code, to be able to navigate through it effectively, to be able to understand what changes to make, and to be able to make them safely and correctly. If the beauty of the code gets in the way, the program is not well written, just as an office building designed to win design awards is not well designed when the building later must undergo changes but those changes are too hard to make. — Richard Gabriel

A skyscraper has low habitability. It’s hard to adapt it to new purposes. It’s hard to adapt it as your needs change. But a mud hut is easy to change and expand. Our software needs that because we can’t plan it out ahead of time.

The genius’s plan for the software might have been good. But it was a master plan that only they could understand. They were the smartest programmer at the company and they used all of their capacity. How could anyone else keep up? Further, you can never fully communicate the “theory” behind the code. Just like Peter Naur recounted in Programming as Theory Building, it’s not just the raw intelligence. There’s also quite a lot of knowledge for how to build and extend the program:

In several major cases it turned out that the solutions suggested by group B [the extenders of the software] were found by group A [the original builders] to make no use of the facilities that were not only inherent in the structure of the existing compiler but were discussed at length in its documentation, and to be based instead on additions to that structure in the form of patches that effectively destroyed its power and simplicity. The members of group A were able to spot these cases instantly and could propose simple and effective solutions, framed entirely within the ex1 isting structure. This is an example of how the full program text and additional documentation is insufficient in conveying to even the highly motivated group B the deeper insight into the design, that theory which is immediately present to the members of group A. — Peter Naur

You have to learn to live in a skyscraper. And you have to learn to extend the software.

So we’ve got this triple whammy:

1. The mental capacity of the original “hero” developer.

2. The knowledge and plan they build while developing it (the “theory”).

3. The high-leverage Clojure programming language.

But here’s the thing: We can’t consider the power of Clojure to be a bad thing. That can’t make sense. If it were true, we’d want to move toward a much poorer programming language, maybe all the way back to assembly. I can’t believe that. Neither can we consider intelligence bad. Where would that lead us?

It seems to me that we have a problem of balancing power with wisdom. What I mean by wisdom is the skill of using half of one’s mental capacity to build the original software so that you have enough intelligence to debug it. It’s the ability to see that our plan is too ambitious, we need to grow the software more organically. It’s the ability to keep the theory simple so you can teach it to the rest of the team. Software can get complicated. The only solution seems to be humility.

Have you seen Grokking Simplicity, my book for beginners to functional programming? Please check it out or recommend it to a friend. You can also get it from Manning. Use coupon code TSSIMPLICITY for 50% off.

Anti-entropic functions

There’s a rule in internet protocols called Postel’s Law. It states that you should accept input liberally (doing your best to decipher faulty messages you receive) but you should be very strict in your output. If every node in a network follows this principle, then we would expect the network to operate more robustly than one that did not. The receiver’s flexibility allows even buggy implementations of the sender’s protocol to operate effectively.

There is an analogous principle in electronics. In electronic components, noise is a major concern. So you build modules to tolerate some noise on the input but to do their best to minimize noise on the output. If every component does this, the system will work better. Noise is a fact of life, but if every component aims to reduce it, then the system will work better.

I’m not a huge fan of Postel’s Law. I’ve had to work with some horrendous HTML as input to my software. It was only horrendous because the browser is so tolerant that it rendered fine visually. But if you tried to parse it with the intent of making sense of it, good luck. In short, tolerance of input allows all sorts of deviant behavior with no consequences.

But Postel’s Law does work for the user of the browser. Instead of showing just an error message, the browser shows something. The page is readable. The links are clickable. And the user is happy. It’s not nothing. If you imagine the world where browsers weren’t tolerant, one stray < character could mess up the parse, which would cause the browser to crash, which would make the user lose their job, then they lose their house, then they’re living on the street. All because of a <.

If Postel’s Law is about protocol errors and the electronics version is about noise, programmers need a law about complexity. Our systems are so complex. We complain about it all the time as the main source of difficulty. So I thought we could formulate a principle like this:

Aim to reduce complexity at every step.

We already do this a little bit. We receive text from an HTTP request and parse it as JSON. This vastly reduces the space of input by structuring it. Then we normalize the JSON into a form that we want to work with, often translating it to a different, less complex data structure, like say a Document. Then we call “accessors” on these data structures; for example, getStatus(). The accessors boil down the complex data structure we pass it to a small set of possible states.

If you combine it with the idea I discussed in my last issue, that of extending the spectrum from partial function through total function to a function that is very forgiving of the input but manages to reduce the output to a small set of states.

In other words, they accept more complexity than needed and output as little complexity as needed. getStatus() takes a whole Document, which includes all the information relevant to status plus all the non-relevant information, and it spits out one of just four valid values for status. That’s a big reduction in complexity. A word for this might be anti-entropic.

When you’re writing a function, ask yourself: Does this function reduce complexity or add complexity? Does it make the problem easier to solve? Does it introduce other problems that will need to be solved?

For example, in How variants can reduce complexity, I talk about how returning different types from a function and attaching meaning to those types actually makes the problem harder. It’s too common in Clojure that we return a string or an integer from a function, with the idea that if it’s a string, it’s the URL of an external resource, but if it’s an integer, it’s the id of an entity from our database.

Returning different types forces any code that receives that value to interpret it. That means the code:

1. Is coupled to the meanings of the types given elsewhere. Action at a distance.

2. Needs to add an if statement to do the interpretation. Added code complexity.

Instead, aim to return the smallest structure that solves the problem. There are many options that are super dependent on context. But here are several distinct approaches. Here’s our original function:

function getDocument(e: Person): string | number;

One improvement would be to use more precise types:

function getDocument(e: Person): URL | DocumentID

That already looks much better. But we can improve it:

type DocumentLocator = {
  locatorType: "url";
  url: URL;
} | {
  locatorType: "document-id";
  url: DocumentID;
};

function getDocument(e: Person): DocumentLocator;

In this code, we’re defining a new type which captures the meaning we intend. You still have the if, but you don’t have the same kind of coupling.

But, more and more, I’m seeing the value of separating these two functions:

function getURLDocument(e: Person): URL | null;
function getDBDocument(e: Person): DocumentID | null;

You need an if statement to determine between them anyway. You might as well just ask for the one you want. I’ve added the null in there because in my imaginary domain, you have none or one or both.

A final approach is to forget the data altogether. Instead of returning a way to get a resource (by URL or by DB id), we return a function you call to get it:

function getDocumentFetcher(e: Person): () => (Document | null);

This one requires no ifs where it is used.

On the other hand, if you’re looking at an existing function and it’s a mess, the way to clean it up is to look for opportunities to introduce steps that reduce complexity. For instance, I once helped someone wrangle some really gnarly code. It was a big nested if statement, where each test was itself a complex boolean expression. We could have sat there and tried to detangle it, looking for some better way to express it.

Instead, I asked the person: What does this code do? After some back-and-forth, it was clear that the if statements were managing the states and transitions of a state machine. They were ushering a complex entity through a process. So we introduced a function that named what state the entity was in. While the if statements were still there, it detangled choosing what state you were in from choosing what to do, and that helped. The code had fewer nested ifs and the logic was clearer.

We could have attacked the if statements themselves head-on. I do that, too, sometimes. I do simple refactorings like eliminating double negatives, rearranging the tests to make them less nested, extracting a function, etc. That would be addressing the style. But in this case, we didn’t. The purpose was to understand what it was doing—to understand the substance. I believe that the substance (the domain model) of our code is a much richer source for finding and eliminating code complexity.

So when I’m talking about reducing complexity at every step, I’m actually referring to model complexity. Reducing model complexity will reduce your code complexity. Code complexity is the biggest thing we complain about as programmers. We must look beyond the surface to bring clarity to the model—make the code more expressive of the model. And with this “principle” of reducing complexity at every step, we can ask ourselves not if the code in the body of a function is simple, but if the function returns something that is less complex than what it receives as arguments—and how perhaps it could reduce complexity even more. I want to believe that this could be a principle, but I’m jaded so I don’t trust it. What do you think? Do you see any holes in this idea? Is it yet another principle that only applies sometimes?

Have you seen Grokking Simplicity, my book for beginners to functional programming? Please check it out or recommend it to a friend. You can also get it from Manning. Use coupon code TSSIMPLICITY for 50% off.

Total functions and beyond

What is a total function?

A total function returns a valid value for all valid argument values. The opposite is a partial function. These terms come from mathematics.

A classic example of a partial function is division. Let’s look at it from a programming perspective:

function divide(a: number, b: number): number;

This says that for any two numbers a and b, divide() will return a number. But this is a lie: If b is zero, there is no answer. It’s undefined in mathematical division.

In JavaScript (and TypeScript), division by zero is possible and well-defined:

 1/0 => Infinity
-1/0 => -Infinity
 0/0 => NaN

All these values are valid in the type number—but in my experience, they’re useless, so I don’t count them as valid return values. Some other languages don’t return such values. Instead, they throw an error:

(/ 1 0) ; throws ArithmeticError "Divide by zero"

Division is a partial function due to one lousy value that breaks the type signature, whether the function throws an exception or returns a useless value.

Why are total functions important?

I think total functions are the next important concept from functional programming that needs to escape to the mainstream. Total functions make our code more reliable and easier to reason about. With a total function, we don’t need to worry about edge cases or unexpected behavior. Every input has a well-defined output, so we won’t accidentally send weird inputs as we build on them.

What if we saw the binary (partial/total) as a spectrum?

I often try to turn a binary into a spectrum. I did that for pure functions in Grokking Simplicity. To make an impure function pure, convert the implicit inputs to parameters and include the implicit outputs in the return value. Zero implicit inputs and outputs means it’s a pure function.

Just because you can’t get the implicit inputs and outputs to zero, it doesn’t mean reducing them isn’t beneficial. If your function reads from 3 global mutable variables and you reduce that to 2 or 1, your function is better off, even without all the benefits of it being pure.

The benefits of being pure are important. A pure function can be tested easily. It can be easily combined, moved, and erased by the compiler. These benefits are discontinuous since you only get them when you become fully pure. But we often forget that the other benefits, like how easy it is to reason about the function, are continuous. Getting rid of each implicit input makes your function easier to understand.

To make a partial function total, we must return a correct value for all valid parameter combinations. But making one more valid parameter return a valid output helps. So let’s consider it a continuous spectrum.

What lies beyond total functions?

Another question I ask is “What if I extend the trend?” Consider total function an interesting point on the line beyond which there are other interesting points. To the left of “total function”, there are various degrees of partiality, with more valid inputs causing errors. To the right, we could accept more “invalid” inputs (as long as we know how to handle them).

For example, let’s say we had a function called parse-int.

;; String -> Long (throws if can't parse)
(defn parse-int [s]
  (Long/parseLong s))

It’s partial. Not all strings can be parsed. We make it total by returning nil if there’s any trouble parsing:

;; String -> Long | nil
(defn parse-int [s]
  (try
    (Long/parseLong s)
    (catch Exception e 
      nil)))

Great! It’s now total. But there’s more we can do. What about a string that’s almost parseable, like it has some whitespace at the beginning? Could we handle that? I can imagine software wanting that, so let’s pretend ours does and add it:

;; String -> Long | nil
(defn parse-int [s]
  (try
    (Long/parseLong (clojure.string/trim s))
    (catch Exception e 
      nil)))

We could handle more “invalid” inputs to this function. We could call it to-int instead of parse-int. It would convert a value to a long. What values could we convert to a long?

• Strings

• A Long itself (idempotent)

• A Double

;; String | Long | Double -> Long | nil
(defn to-int [v]
  (cond 
    (string? v)
    (try
      (Long/parseLong (clojure.string/trim s))
      (catch Exception e 
        nil))

    (int? v)
    v

    (double? v)
    (long v)))

We could go further with it.

As an example of a real function that does this, check out clojure.core/slurp. Here’s a list of the kinds of inputs it will take:

• Reader

• BufferedReader

• InputStream

• File

• URI

  • If it’s a file:// URL, convert it to a File first

• URL

• Socket

• byte arrays

• character arrays

• String

  • Try to parse it as a URL

  • Failing that, try to parse it as a local filename

I usually think this kind of function, which does a lot of magic to “do the right thing,” doesn’t have much place. I prefer to know the type I’ve got and call the right function to convert it to what I want. But sometimes I think they feel right—and this is one of those times. I’m trying to figure out when they feel right and when they don’t. In this case, I believe it feels right because the function:

• reduces the system complexity by converging several different (potentially unknown) types down to one known type

• remains a total function

Do you have ideas?

The other side of the coin: Avoiding invalid answers

I want to discuss the other side of the coin. In JavaScript, even with TypeScript’s protection, you still run into issues:

function divide(a: number, b: number): number {
  return a/b;
}

That could return a “real” number, or it could return NaN, Infinity, or -Infinity. Those last three values suck because they throw a monkey wrench in other arithmetic operations. They’re just waiting to cause trouble and this function will silently return them.

Let’s prevent that!

function divide(a: number, b: number): number {
  if(b === 0) throw new Error("Divide by zero");
  return a/b;
}

Although this function is now partial, it does a better job at protecting our code from NaN and other pesky values. I think we do similar things in Clojure.

I see this behavior as the flip side of total functions. Total functions ask you to work for all valid inputs. This technique guarantees that a function will only return valid outputs. Do you have a name for this?

By the way, I think we need to use these in type-safe languages a lot as well, just perhaps not for the types of cases I’ve shown. For instance, there are values of numberOfMegabytes that will result in problems and you probably want to limit it:

function allocate(numberOfMegabytes: number): Array {
  if(numberOfMegabytes > 1000) throw new Error("Can't allocate more than one gig");
...
}

You could, of course, return undefined instead of throwing an error to keep it total.

Review and conclusion

I like the concept of total and partial functions. I hope that, just like pure/impure functions, it escapes the functional programming niche and enters the mainstream. I explored how we could turn the binary into a spectrum because a partial function being closer to total is better than not. But that opened up a world of possible inputs we could make valid past the special point on the spectrum called “total”. We saw what that could look like. I also brought up the idea that we often want to protect the return value. It’s a dual concern to the idea of total functions.

What techniques do you use to avoid functions returning bad values? How do you ensure that you only pass good values to your functions?

I speculated about why I thought that to-int was an okay function, even though it borders on “magical” type conversion. I think it has to do with its output being less complex than its input—and also being a total function. In the next issue, I want to expand on this idea, bring in “Postel’s Law”, complexity, and understandability.

Have you seen Grokking Simplicity, my book for beginners to functional programming? Please check it out or recommend it to a friend. You can also get it from Manning. Use coupon code TSSIMPLICITY for 50% off.

The presentation was chaotic. My slides weren’t working. My daughter was sitting on my lap. And, what’s worse, the audience did not believe what I had to say. My main point was that you didn’t have to live in a nest of if statements. You could find a model in any domain. But they weren’t buying it.

That presentation was my first step toward the idea of domain modeling as the heart of software design. That was years ago. And it seems obvious now (so obvious that it seems weird to need to say it), but here’s where I eventually arrived: A domain model is judged by how well it corresponds to the domain. The model and the domain must have similar (ideally: identical) structure. The closeness of correspondence between model and domain is where the power of abstraction comes from. How do we get computers to do useful work? Correspondence.

But correspondence doesn’t sound great when you’re looking at a messy domain. In the intimate audience of that presentation, someone mentioned their domain had lots of government regulation. He believed there was no way to find a clean model. The only choice was building up spaghetti code to match the spaghetti of laws that made up their domain. And here I was trying to sell an answer to complexity by finding a better model.

I empathized with him. I’ve faced similar domains. But I’ve also been surprised. I came away from that presentation with a new awareness of the difficulty of the challenge I had taken on. You see, many times in my career, when we were struggling to find a simple model, we’ve eventually found a one (though sometimes too late). Well, not just a simple model—a much better model than what we currently had. The new model was so good, in fact, that when we did find them in time to build them into our code, they drastically reduced the complexity of our codebase. At the same time, they added dramatically to the business value.

I’ll tell a story that I’ve told before. But it’s very relevant. I worked at a company that helped Americans register to vote. In the US, each state determines its own voting registration requirements. A co-founder of the company had studied all of the laws of all the states, and she presented us with a big mess of overlapping categories, showing how crazy the logic was. It was clear she’d tried to organize the chaos but came up short.

We all struggled with the rules. Some states let you register the day of. Some you didn’t need to register. Some you had to register two weeks before. Some required you to live in the state for two years, unless the state you moved from didn’t allow you to vote there. Some just wanted you to vote where you lived. Ugh. It really was a mess. There were no clear seams that we could pull apart—which, in retrospect, makes sense when each state is acting mostly independently.

Eventually, I gave up. Let’s just do a big if statement! Why not? The first level of nesting will be by state. If you’re in Nebraska, then follow this logic. If you’re in New Jersey, follow this logic. We divide it up, and the task is 50x easier. We just have 50 new if statements to write, one for each state.

It’s sometimes useful to look at that worst case scenario. The worst case is that you have 50 small messes instead of one giant mess. Ask yourself: How would you do it in a straightforward way? It doesn’t mean you have to commit to going that way, but it helps jog the mind. And jog my mind it did.

Once it was a set of conditionals, I saw something we had overlooked: The question we were trying to answer was really simple: Can person x register in state y? That can be written in a nice function signature:

function canRegister(x: Person, y: State): boolean;

And each of those 50 branches of the conditional was then a question of this signature:

function canRegisterInNebraska(x: Person): boolean;

One of those functions for each state. That’s just a simple predicate on Person. And when you look at the rules of Nebraska (or any state), they’re of a similar form:

function isMajorityAge(x: Person): boolean;
function isResident(x: Person): boolean;
function isCitizen(x: Person): boolean;

We can easily combine those smaller, easy to implement questions into one bigger question with some boolean operations:

function and(f1: (x:Person)=>boolean, 
             f2: (x:Person)=>boolean): (x:Person)=>boolean;
function or (f1: (x:Person)=>boolean, 
             f2: (x:Person)=>boolean): (x:Person)=>boolean;

It turns out that isMajorityAge() is very reusable. Most states have that rule. So we were getting plenty of reuse. And we were able to build up rules that had a very similar structure to how the laws worked in each state. Each state was custom, but it was built out of modular, reusable parts.

In essence, we looked beyond the mess to find structure underneath. The structure was at a level of higher generality than we were looking at. We wanted to find a hierarchy to fit the states into. Instead, we needed to accept that each state was complicated and build a platform of atomized decisions that could be composed to mirror the structure of the state’s laws. We build a platform on which to model the states’ laws. I sometimes call such a platform a core abstraction.

I think of that story every time someone claims their domain has no structure. There’s always a structure. Humans build things with structure, even if it’s very messy and complicated. It’s never random. You can find the structure.

However, I don’t think it’s merely a mindset shift. I don’t really believe mindset is enough. Over the years, I’ve realized that it takes a very trained eye to spot that structure. How did we see that there was Boolean logic at the base? Lots of practice? How did I come up with that function signature? Years of working in Haskell. A part of my brain is always asking “what is the type?” And the composing functions? Luckily, this was a Clojure company, so we were used to thinking about higher-order functions. Would the average Java programmer have thought of that? I would guess not because they’re not used to thinking that way.

We should judge a model by how well it corresponds to the domain. The better it corresponds, the simpler our code that uses that model. But many domains are messy. Their structure is entangled. Does that mean that our model should be messy as well? I say no. It needn’t be messy, but you have to look for structure at a different level of generality than you’re currently looking.

But this a bittersweet synthesis. On the one hand, it’s hopeful. There is a core model that structures the mess. I’ve always found one, eventually. On the other hand, it’s not easy. I think of a passage from The Early History of Smalltalk (emphasis mine):

It started to hit home in the Spring of '74 after I taught Smalltalk to 20 PARC nonprogrammer adults. They were able to get through the initial material faster than the children, but just as it looked like an overwhelming success was at hand, they started to crash on problems that didn't look to me to be much harder than the ones they had just been doing well on. One of them was a project thought up by one of the adults, which was to make a little database system that could act like a card file or rolodex. They couldn't even come close to programming it. I was very surprised because I "knew" that such a project was well below the mythical "two pages" for end-users we were working within. That night I wrote it out, and the next day I showed all of them how to do it. Still, none of them were able to do it by themselves. Later, I sat in the room pondering the board from my talk. Finally, I counted the number of nonobvious ideas in this little program. They came to 17. And some of them were like the concept of the arch in building design: very hard to discover, if you don't already know them.

The connection to literacy was painfully clear. It isn't enough to just learn to read and write. There is also a literature that renders ideas. Language is used to read and write about them, but at some point the organization of ideas starts to dominate mere language abilities. And it helps greatly to have some powerful ideas under one's belt to better acquire more powerful ideas.

The challenge I’ve taken on in my book is to give people a handful of powerful ideas that help people model. I think the biggest idea is to judge code complexity not by how hard it is to read, but on how well it expresses the model. Then there are others, like to use total functions. And total functions is what we’ll talk about next week.

Do you have examples of finding core abstractions in an otherwise intractable domain? Is there a domain you gave up on and resorted to spaghetti code?

Have you seen Grokking Simplicity, my book for beginners to functional programming? Please check it out or recommend it to a friend. You can also get it from Manning. Use coupon code TSSIMPLICITY for 50% off.

Bad data models lead to code complexity

There I was, staring at a complex, nested if statement with hard-to-follow logic in each condition. What’s worse, similar logic was repeated in several places in the codebase. That was when I realized how much complexity a bad data model can cause.

An example of a bad domain model

The domain was simple. I’ll describe a similar one that is an easier example so I don’t have to explain the whole domain. Imagine a content management system for a magazine. Articles go through several stages as they are prepared for publication:

• Drafting

• Editing

• Approval

• Publication

We modeled it with several Maybe<DateTime> type. As we finished a stage, we added the time. If the time didn’t exist, it meant that we had not finished that stage yet.

type Document = {
  drafted?  : Date;
  edited?   : Date;
  approved? : Date;
  published?: Date;
};

This worked for a while, but eventually we found problems.

The first problem was that our logic was complicated and duplicated. If we wanted to know if something was in the approval stage, we had logic like this:

if(document.drafted && document.edited && !document.approved)

The second problem was that as we increased the number of documents in the database, with many releases to our software (often with bugs), we eventually got data that had impossible states. For instance, it was edited without being drafted:

{
  drafted: null,
  edited: new Date("2010-01-02")
}

How could this happen? Well, despite our checks, something slipped through. Maybe it was a bug in a prior release. Maybe it was user error. Whatever the case, it existed, and our code needed to make sense of it. That meant more if statements.

This is a bad data model. I mean that in an objective way. We can actually measure how bad it is. We can count the number of states in the domain and the number of states in the model, and see how far off they are from each other. Here is the timeline of a document workflow:

The horizontal lines represent events that divide the timeline into sections. Each section is a distinct state that the document can be in. We see clearly that there are five states a document can be in in this domain.

Our model is different. It contains for optional values, where the presence of the value is meaningful. The date is also meaningful, but in a different way. I’m going to ignore it for this analysis, even though it also could present difficulties such as if a document is published before it is drafted. We’ll just look at the presence or absence of the value.

In that situation, there are four values with two options each (present/absent), which is 2^4 = 16. So our data can represent 16 states. That means that 11 of our states are redundant, ambiguous, or invalid, because we only needed 5.

That sucks. Those unneeded states caused a lot of code complexity because our code had to map them to the domain concept. It had to deal with the ambiguity of the situation. That was the second problem I talked about (finding impossible data in the database). We could have seen this coming with some simple math.

The first problem was still there. Remember, the first problem was about the complex, duplicated logic we found everywhere. It wasn’t really because there were ambiguous states. It was more that what we stored didn’t correspond well to what we needed.

Remember, we were storing the time that we crossed each of the horizontal lines. But what we wanted to know was which green box we were in. We had to write compound conditionals to convert the lines into boxes.

Adapting the existing data model to a better one

Luckily, we can adapt this model and solve both problems with in one stroke. Remember, domain modeling is about mapping. So let’s create a function that maps the data model we have to the data model we wish we had:

function documentStatus(document) {
  if(document.published) return "published";
  if(document.approved) return "ready";
  if(document.edited) return "approval";
  if(document.drafted) return "editing";
  else return "drafting";
}

We did double-check to make sure that this does correctly map the ambiguous states to the correct states. Luckily, it was simple. We could imagine it being much more complex than this. For instance, we might have a check somewhere in the code that a document with a published date but no content is actually in the drafting stage. I have a hunch that this is where the really hairy complexity comes from—different behavior in different places in the codebase because of inconsistent logic. Either way, having a single place that defines the mapping to our desired model is a good thing. The biggest trouble with it is making sure to use it consistently.

We could then use the human-readable string representing the status in place of a bunch of booleans anded together:

if(documentStatus(document) === "approval")

instead of:

if(document.drafted && document.edited && !document.approved)

Here are the things we achieved:

• Centralized the logic for determining the status

• Eliminated inconsistencies in status logic across the codebase

• Made status checks more human-readable

• Eliminated complex conditional logic

I’d say it was a major win.

Is this a major win? Or a waste of time?

Even still, I have encountered people who think this is a waste of time. To put a fine point on it, this change was both small (in the number of lines of code it saved) and extensive (in the number of lines of code and the number of files it touched). Further, it was risky. Changing so many lines of working code could introduce bugs we couldn’t anticipate. In short, it’s high risk and it doesn’t eliminate much complexity.

When I ask them about where code complexity comes from, they say it’s from other sources—mainly business logic, user requirements, and system interactions. Bad domain models do introduce complexity, but it’s a rounding error compared to those sources.

My response is always the same: Those sources of complexity can be modeled, too. Nesting if statements is one simple tool we have to address a corner case. I’ve done it myself thousands of times. It’s extremely useful when I don’t have the time to fully understand what is happening in the codebase. Surgically adding an if where I’m certain it will be useful is a versatile tool.

But nesting ifs is a tool that breeds more complexity. The next time I or someone else needs to handle a new case, it becomes that much harder to understand, so that much more likely that we’ll add yet another if.

Modeling is the only way to rein in complexity

Look, I’m getting deep into the weeds here, and I need to bring this thread back to my intended point: There’s no way to clean up those ifs without understanding the domain. All a domain model is is an understanding of the domain expressed in code. When you express it in code, its an understand that is available to anyone who cares to read it. Domain modeling is software design. Trying to eliminate complexity without modeling is a fool’s errand. Domain modeling is a necessary skill for designing software. So we might as well get good at it.

When we talk about modeling business logic, often people hear “oversimplifying”. That is not what I’m talking about at all. Domain modeling means embracing the complexity, but identifying the structure of it, then writing that structure down in useful code. Maybe I’m preaching to the choir, so I’ll just leave it at that. If you’ve got an understanding, you can write it down. I also hope to show that there are objective measures of data model quality in my upcoming book. Modeling is a skill, and you can get better at it.

Conclusion

Code complexity is a major cost of software development. Bad data models add significantly to code complexity—when models don’t correspond to the domain, we require extra code to handle the redundant and missing cases. Cleaning up the data model can significantly contribute to reducing complexity. Even if you think that data models aren’t a major source of complexity, the sources of complexity can also be modeled. Chances are, the current models of things like business logic, user requirements, and system interactions are poor models. They’re adding way more complexity than they actually need. You can clean those up, too.

I hope to bring this awareness of poor models to the industry. Currently we’re stuck thinking about code in isolation instead of how it relates to the domain. I’d like to help fix that, which is why it’s a major theme of my book. I don’t want domain modeling to be just another abstract skill. Instead, it’s fundamental to software design. The more I research it, the more I see that it’s the real key to explaining a lot of software design advice.

Do you have any interesting stories involving domain modeling? How do you apply domain modeling in your work?