journal.stuffwithstuff.com

The Value of Things

Sat, 24 Jan 2026 08:00:00 GMT

One of the reasons I write is to help me organize my own mind. I have a compulsive need to figure things out and I’ll lay awake at night shuffling sentences around in my head until it hangs together. Then I just have to try to get it all down in Markdown before it dissolves back into chaos.

Like a lot of people these days, I am losing a lot of sleep over LLMs and generative AI. I mean that literally. I’ve had nights where I tossed and turned worrying about whether AI is going to destroy a career I love or wreck modern society until the sun comes up.

I keep hoping I’ll sort it out in my head enough to reach some inner peace. I’m not saying I will literally figure it out, like, for the world. But I keep hoping I can at least figure out my own relationship with AI.

I’m still not there, but I was able to pull one thread out of the tangled discourse around AI and get it to make sense in my head. I hope it will help you too. Or maybe this is all nonsense and you’ll point out the myriad ways in which I’m wrong. That’s a possibility, but I’m trying to be more courageous in my writing, so here I am.

Anyway. Generative AI is about using machine learning to produce digital things. For AI to be a net good for the world, at the very least, the things it produces must have value. But what does it mean for a thing to have “value”?

Utility#utility

The obvious answer is that a thing has value if it’s useful. If it solves some material problem in the world or at least creates some joy, that’s value. In the UX world, they call this “utility”: the thing a thing actually does.

An apple has utility. You can eat it and it will give you some calories, some hydration, a little fiber. If it’s not an oxymoronically named Red Delicious, it might even bring you some delight. Material objects have a natural sort of utility from their physicality, but digital objects can have utility too.

A while back, I took an interest in audio programming. I read a lot of articles online to learn some of the math and algorithms behind digital signal processing. That information is useful. Before I read those articles, I couldn’t program an FM synth. Now I can. That utility is just as real whether I got it from reading a classic article written by Julius O. Smith III, or from asking ChatGPT.

In both the physical and digital worlds, utility really does matter. One possible anthropological definition of “technology” is anything that enables humans to more efficiently generate utility. It’s central to human progress.

The reason I can go to a store and buy a bag of apples is because thousands of years of agriculture technology development means humans can very efficiently produce and transport food. And, you know, we need food to live. Food is cool.

My career as a programming language designer is only possible because of all of the material I found online to learn how compilers and languages work. Hobbies like knitting, making music, and cooking are deeply important to me. They’re possible in large part from learning through YouTube videos. Those articles and videos were truly useful to me.

Generative AI can help improve this. Here’s a concrete example: I got curious about how AI is affecting software jobs outside of my weird tech company bubble so I started poking around job listing sites. I stumbled onto a software engineer position working for the Washington Department of Ecology. The listing says:

We’re looking for two journey-level developers who care about clean architecture, thrive in Agile teams, and see modern tooling—including AI-assisted development—as a way to work smarter, not riskier. You’ll collaborate with product, architecture, security, and platform engineering partners to design secure, accessible, scalable applications that replace decades of legacy complexity.

…

Use AI-Assisted Tools Responsibly and Effectively

Leverage AI tools for boilerplate, test generation, and safe refactoring.

Validate AI-generated code for accuracy, security, and maintainability — no blind trust, no shortcuts.

Share best practices with teammates adopting new AI workflows.

It’s hard for me to imagine a better morally justified use of AI for software development. This is a government office whose charter is to “protect and sustain healthy land, air, water, and climate in harmony with a strong economy”. They are literally working to make the world better for nature and humans.

But as a government department, they are also really limited in their resources. If AI can make those two software engineers more productive, it can take the same tax dollars as input and produce a healthier world as output. That’s hard to argue against.

Meaning#meaning

When people get excited about AI and productivity, I think utility is what they have in mind. But utility is not the only kind of value a thing can have.

A couple of years ago, I hand-knitted a scarf for my mother-in-law. The scarf does have utility. It’s a rectangle of fabric that keeps her neck warm. The Olympic Peninsula is a lovely corner of the country, but it’s often a chilly one. A scarf made out of wet tissue paper would not have the same utility.

I could have just gone to the store and bought her a scarf. Were I the kind of robotic rational actor who thinks only about maximizing their utility function when not sipping pour-overs in some culty rationalist commune outside of SF, then I would know that the optimal strategy is to use my software engineer skills to maximally turn my labor into cash and then use that cash to buy a scarf.

The scarf I hand-made took dozens of hours to make. At even a modest West Coast software engineer salary, that number of hours would buy an extremely nice scarf. Maybe not vicuña, but at least cashmere. Had I dropped a few hundred bucks on a scarf, would it have more value? Realistically, it would be a better scarf. Softer material, finer stitches. Certainly more fashionable.

You and I know the answer is “no”. As any parent who lovingly clips their kid’s objectively terrible art to the fridge knows, the value of a made thing is not entirely based on its merits. That scarf has value not because it’s a great scarf but because I chose to spend an irreplaceable fraction of my life making it. It’s a symbol of how much I care about her.

In short, it has meaning. Meaning doesn’t make an object more useful. It’s not even transferrable. If my mother-in-law gave the scarf to you, its meaning would evaporate.

But I bet that if you think about the objects that have the most value to you, your S-tier will be populated by things that excel mostly in meaning, not utility. The things that you would grab before running out of a burning house are the things with sentimental, not functional value.

Source of meaning#source-of-meaning

You kind of know the answer intuitively, but I’m trying to be clear in my thinking, so when I talk about objects having meaning… where does that meaning come from? Why do some objects have more meaning than others?

I’ve spent hundreds of hours knitting over the past couple of years, mostly making things for friends and family. I’ve thought about this question a lot. I suspect that it comes down to spending time in service of another.

Humans are sentient actors with agency and entire cognitive universes in our heads. But we are also animals made of meat and bones, and that flesh doesn’t last forever. Every one of us has a finite amount of things we will ever do. “Spend time” is one of those stock phrases that has been sucked dry of semantic flavor over the years, but there is a profound metaphor there. When you choose to use some of your time making a thing for someone, you don’t get that time back. It is spent.

Choosing to spend a fraction of our most precious resource for someone else is the strongest signal we can send that that person matters to us.

I’m in my late 40s. I hope I still have a lot of living left to do, but the odds are very good that I’m past the mid-point of my lifespan. An increasing fraction of conversations with my friends revolve around our various medical tribulations. I can feel the finiteness of life in my bones, quite literally. When I think about what I want to do with my remaining allotment of life, the answer that resonates is taking care of and making things for the people I love.

Generating meaning#generating-meaning

Generative AI, when wielded deftly, can be an amazing tool for creating things with utility faster and more easily than you ever could before. But it can’t generate meaning. The giant matrix of floating point numbers in a rack of GPUs in some data center does not love you.

Another story: When my brother and I were growing up, we were really into movies. We made short videos (hilariously bad), learned how to do special effects make-up (actually tolerably good), and all sorts of stuff like that. We dreamed about growing up and becoming another pair of Hollywood brothers like the Zuckers or Coens.

Many years later, as a birthday present, I wrote my brother a screenplay for a short horror film about a mythological siren. I toiled on it every night after the kids went to bed for weeks. It’s one of my favorite gifts.

I don’t know if we’ll ever get a chance to shoot it. We live on opposite sides of the country and he can’t handle the gloom of Seattle any more than I can handle the politics of the South. It’s likely this screenplay has zero utility. But it still has a ton of meaning because I sweated every single word in that stack of 12-point Courier pages.

Today, with the help of ChatGPT, I could probably put together a feature-length screenplay in a tenth of the time. It might even be an objectively better screenplay for a better movie. But because I made the screenplay in a tenth of the time thanks to ChatGPT’s help, it would hold only a tenth of the meaning for my brother. If my hypothesis that meaning comes from time sacrifice is true, then by making us more productive, AI eliminates meaning.

Both kinds of value#both-kinds-of-value

Most objects, digital or material, have a mixture of both kinds of meaning. My favorite gifts to give or receive are those rare ones that are both thoughtful and useful.

An object carries with it the weight of time that was put into making it and has a certain heft from what you can do with it. I think of AI as a tool that can transmute that former kind of value into the latter. It drains some of the meaning out, but in return lets you make more things with utility for the same effort.

There is nothing unique to AI about this. Any tool that increases efficiency has the same property. They make little knitting machines that make rows of stitches as easily as turning a crank. I could have made a scarf for my mother-in-law using that in a fraction of the time.

Efficiency—the multiplier that determines how much effort is required to make an object—works sort of like a slider that lets us choose the ratio of utility and meaning the resulting thing has. Generative AI, assuming the claims are true, has the capability to radically move that slider. It can produce huge volumes of stuff with real utility. But that exact same ease implies that the results are all devoid of a sense of personal meaning.

Two tangents#two-tangents

I don’t want to take the idea that efficiency is a slider between meaningful-but-scarce and meaningless-plenty too far. It’s a rough model. There are at least two problems with it:

First, it suggests that you can make an object infinitely meaningful by just making the process arbitrarily hard. I could have knitted that scarf using two leftover take-out Chinese bamboo chopsticks. In the dark. One-handed. That would certainly have required a greater sacrifice of time (and sanity) on my part. But there’s obviously a point of diminishing returns where people value the effort we put into making things but less so when we elect to be masochists.

Also, when making things, we often derive personal joy from the process itself. Yes, I did sacrifice time to knit a scarf but… I like knitting. It wasn’t an entirely miserable toil or something. I got to play with yarn.

The high level point is just that the more we automate the process of making a thing, the less of ourselves we put into it. And an object with less of ourselves in it is often valued less by the person who receives it. That’s all I’m saying.

What kind of value do you want?#what-kind-of-value-do-you-want

The tools we use to make things and the resulting productivity gives us a lever to control how much we prioritize getting a useful object out fast versus getting an object suffused with meaning. That suggests a way to decide when we should and shouldn’t use AI.

If you just need a thing that does a thing, then its utility is what matters. I want the Washington Department of Ecology to use my very limited tax dollars as effectively as they can to keep the Pacific Northwest the enchanting natural marvel that it is. If they can do that better using AI, then by all means they should do so.

If you want to make a thing that has some emotional resonance or some connection to other people—whether that be a loved one, an audience, or humanity as a whole—then you don’t want to suck out any of its meaning.

I listen to a lot of electronic music while I work. In some sense, it is “utility music”. I just want a vibe to help me tune out the world and focus. I can’t even handle lyrics while I’m programming or writing.

I have to admit that the genres I listen to are the ones most amenable to being automatically generated by machines. I literally do listen to generative ambient which has been a little corner of the music world since well before “generative” glommed onto the word “AI”.

Even so, I still find my listening experience more gratifying when the music was made by someone who actually gave a shit about it. Even though the relationship between artist and listener is vague and indirect, that connection matters to me. I know that when I make music, the thing that helps me push through all of the many frustrations of producing is the hope—however tentative—that at the end of it all, one day someone might put on a track of mine and find their day improved by it being the temporary soundtrack to their life.

Those moments of connection between artist and audience are what I live for as a creative person. For those, I can’t see generative AI as anything but harmful.

Sometimes yes and sometimes no#sometimes-yes-and-sometimes-no

Thinking about the things I make and consume and the kinds of value they contain helps me sort out some of my extremely conflicted feelings around AI.

I want to emphasize that I’m only looking at this from the perspective of a single person’s individual use of AI. The global effects of AI are just as if not more important. But I haven’t sorted out my feelings around that to have any idea what to say here. Maybe I never will.

But I can set that aside and imagine a world where I can run a generative AI completely locally on my machine. A world where I was the only person on Earth who had an AI. In that timeline, there would be almost no externalities to consider. Even then, I still want to be mindful of the consequences of using AI to help me make.

In my own uses, I aim to focus on the places where I think AI can improve the efficiency of making utilitarian goods. And when I’m making something where the human connection and meaning are important, I will try to put as much of myself and as little of the machine into it as I can.

Setting Up an SDL3 Mac App in XCode 16

Sun, 13 Jul 2025 07:00:00 GMT

This is mainly a note for my future self, but making it a blog post in case it helps anyone else. This weekend, I decided to try making a little SDL3 app on my new M4 MacBook Pro. It took me a few hours to figure out how to get XCode to play nice with the SDL3 framework given MacOS’s paranoia around unsigned code and malware.

Here’s the summary for anyone whose Google searching takes them here. The instructions are fairly similar to the SDL3 README for MacOS:

Start up XCode. Choose “File > New > Project…”.
Select “macOS”, then under “Application” choose “App”. This step is important. Choosing “Command Line Tool” will not work. Click “Next”.
Give your app a name. For “Interface”, choose “XIB”. For “Language”, choose “Objective-C”. Click “Next”. Choose a place to save your project.

Now you have a vanilla “Hello World” Mac app. The next step is integrating SDL:

Download a release of SDL. As I’m writing this, the latest is 3.2.16. Download the corresponding .dmg file. Open it in Finder.
Open SDL3.xcframework/macos-arm64_x86_64 and copy SDL3.framework from there to somewhere in your XCode project’s source tree. (I made a lib folder and put it there in mine.)
In XCode, in the file explorer on the left, click the top project icon to open the settings for it. In the main panel on the right, click the icon for your app target under “Targets” to open the target settings.
Under “Frameworks, Libraries, and Embedded Content”, click the “+” to add a new framework. In the popup that appears, click “Add Other… > Add Files…”. Navigate to wherever you copied “SDL3.framework” and choose it.

Then let’s use SDL and verify everything is working.

In main.m, comment out the entire main() function.
Make a new main.c file and copy the contents of hello.c from the SDL docs into it.
Compile and run.

If that all works, you should see a delightful pixelated “Hello World!” fill your screen. Press any key to quit.

Let’s make sure the full iteration loop works. (Read below to see why I feel the need to verify this.):

In main.c inside SDL_AppIterate(), change the message string.
Hit Command-R to run. You should see your updated message on screen.

There you go, you now have a working shell of a Mac app with using a local install of SDL3. You can probably delete the main.m file entirely and maybe some of the other generated files that XCode put in the new project, but I haven’t figured that out yet.

Now here’s all the wrong stuff I tried first before I got that working…

Attempt 1: Download a release and use the xcframework#attempt-1-download-a-release-and-use-the-xcframework

In XCode, I created a new “CLI Command” C app. That got me a “Hello, world!” up and running without any problems. Then I copied the contents of this SDL hello example into my source file.

As expected, that failed to compile because it couldn’t find SDL3/SDL.h. So next is getting SDL linked in.

The first thing I did was download the 3.2.16 stable release installer. I downloaded the DMG file and opened that. Then following the instructions, I dragged the entire SDL3.xcframework directory into the XCode project.

That correctly added a reference to the Framework and got XCode to a point where it could resolve the include and compile the program. Great.

But when I ran it, it failed at startup. The OS tried to dynamically load the SDL3 framework, but then balked because it thinks the framework isn’t correctly signed and safe to use. The OS gave me a popup error with “SDL3.framework Not Opened” because “macOS cannot verify that this app is free from malware”.

Crap.

Attempt 2: Embed the framework#attempt-2-embed-the-framework

Poking around online, some people say this error can be fixed by embedding the SDL framework in my app and having XCode sign it when it embeds it. In XCode, in the target settings under “Build Phases”, there is a panel for “Embed Frameworks”. I dragged SDL3.xcframework under there and checked “Code Sign On Copy”.

Alas, when I tried to compile, now I got a build error. It complained that two tasks in the build were trying to produce the same file. I’ve lost the exact error message but it looked like one of the tasks was a simple file copy and the other was a “ProcessXCFramework” invocation. I’m guessing the latter is the task that signs and embeds the framework.

I tried basically every combination of build settings to try to fix this: removing the framework from “Link Binary with Libraries”, checking “Copy only when installing”, and a bunch of other things I don’t remember. Nothing worked. Either I removed so much that the compiler couldn’t find the framework at all, or it got the duplicate build file error.

Attempt 3: Don’t use the entire “SDL3.xcframework”#attempt-3-dont-use-the-entire-sdl3xcframework

I started watching this video to see if it could help. He doesn’t use the entire SDL3.xcframework bundle. Instead, he opens that, goes into macos-arm64_x86_64 and copies the SDL3.framework framework out of there.

I removed SDL3.xcframework from my XCode project and dragged SDL3.framework into it instead. That gave me something that compiled, but when I tried to run, it failed at load time with:

dyld[15605]: Library not loaded: @rpath/SDL3.framework/Versions/A/SDL3
Referenced from: <...> /Users/...
Reason: tried: '/Users/.../Build/Products/Debug/SDL3.framework/
Versions/A/SDL3' (no such file)

In the target settings, I again added the framework (but this time SDL3.framework not SDL3.xcframework to “Embed frameworks”). For reasons that are entirely unclear to me, that worked. I hit Command-R and…

Hallelujah.

All done, right? Not so fast. I made a tiny tweak to the code and tried to run it again. The compile immediately failed with:

'SDL3/SDL.h' file not found

What? I literally just compiled and ran it. Just to make sure I wasn’t crazy, I cleaned the build directory and compiled again. That worked! “Hello World!” popped up on screen again.

Build again… same error.

Clean build and then build. Works!

So I can build once but then something in the build directly causes any subsequent compile to fail. The full error is:

Did not find header 'SDL.h' in framework 'SDL3' (loaded from
  '/Users/.../Index.noindex/Build/Products/Debug')

So what I think is happening is that in the first clean build, it locates the framework inside my app’s main directory. That framework does have the headers so the compiler can find what it needs.

Then after that first build, the compiler starts using the framework that it copied into the build directory. But when I look in there, Debug is completely empty.

Attempt 4: Project-relative DerivedData#attempt-4-project-relative-deriveddata

I hate when tools stuff built data in random system directories unrelated to the project I’m working on because those files end up sitting around forever. So the next thing I did was go under “Project Settings” and changed the “Derived Data” to be “Project-relative Location”.

I cleaned and built successfully. But, again, if I made a trivial code change and tried to build again, it failed. This time, though, there was actually something in DerivedData/my_app/Build/Products/Debug/.

I can see an SDL3.framework directory under there. It contains Versions/A/SDL3 (and some other stuff). So the library is there as expected. But there is no Headers directory under Versions/A/, nor is there a Headers symlink in SDL3.framework pointing to it.

In theory, that’s fine. In the compiled app, all that’s needed is the library itself. But for reasons I don’t understand, XCode is trying to use this semi-copied version of the framework for subsequent compiles instead of the original one in my project’s source tree.

Attempt 5: Explicit header search path#attempt-5-explicit-header-search-path

OK, if XCode can’t find the headers because it’s looking in the wrong place, maybe I can just tell it where to look. Under “Search Paths”, I added lib/SDL3.framework/Headers to “Header Search Paths”. (I put the framework under lib in my project tree, hence the lib/ part.) Didn’t help. Tried making it recursive. Nope.

Maybe just lib/SDL3.framework? No.

I’m guessing the problem here is that the include paths look like SDL3/SDL.h, not just SDL.h, so I need to get XCode to understand that the headers are inside a framework to resolve the SDL3/ part.

Attempt 6: Framework search path#attempt-6-framework-search-path

OK, so maybe it’s not a header search path. Instead, I tried adding lib/ to “Framework Search Paths”. No help. Making it recursive didn’t help. Neither did lib/SDL3.framework.

I do a bunch of Googling and eventually stumble onto this old StackOverflow post. He specifically mentions only getting the error after the first build, so this is my exact issue. And the author apparently found a solution!

…which they described only as “It works know after certain changes!”. So helpful.

I eventually found this Apple Developer forum thread, which also sounds like my problem.

Attempt 7: An App instead of a CLI Command#attempt-7-an-app-instead-of-a-cli-command

The forum thread suggests that the issue may have something to do with creating a “CLI Command” instead of an “App”. I’ve always done the former in XCode because that seemed like the simplest way to get a vanilla C/C++ app up and running. In older versions of XCode/macOS, it worked fine. But maybe it doesn’t play nice with frameworks in later versions?

I trash the entire XCode project and create a new one. This time I choose “App” instead of “CLI Command”. I set the language to “Objective-C” (because that’s the “closest” to C, I guess—I’m winging it here). I set “Interface” to “XIB” based on this README.

I run it and get an empty window, the “Hello World” of GUI apps. OK.

I copy SDL3.framework out of SDL3.xcframework/macos-arm64_x86-x64 into lib/ in my new app’s source tree. Then in XCode, in the “General” tab of the main target, under “Frameworks, Libraries, and Embedded Content”, I click the “+”. From there I choose “Add Other…”, “Add Files…”, and add lib/SDL3.framework.

I admit to feeling a moment of anticipation. Will this actually work?

I make a new .c file and paste in the hello.c SDL3 example code. I comment out main() in the main.m that XCode generated. Compile and… it runs! I’ve got an SDL3 “Hello World!” on screen again.

Now the real test… can I make a change and build again?

Yes! It works! Finally!

As far as I can tell, I am up and running now. I didn’t have to add SDL3 to my system framework path. (I didn’t want to do that because ultimately, I want an app that users can just run without having to install SDL themselves.)

So it looks like the failure mode was creating a “CLI Command” instead of an “App”. How on Earth would I be expected to know that?

Consider Knitting

Fri, 30 May 2025 07:00:00 GMT

Let’s say that, like me, you are a person who stares at a computer and writes code for a living. As a straight male who grew up in a time where knitting was very strongly female coded, it for the most part never occurred to me that knitting was a thing I could do and might enjoy. Regardless of your demographic categories and background, it’s possible that you have also not really considered knitting.

This article exists to get you to do so. Specifically, I’ll try to convince you, one software person to another, why it might be a good fit for your life and brain. This is a pitch for knitting, but—for better or worse—an extremely nerdily argued one.

Frog and Toad, from this delightful pattern. I made these as Christmas presents for my daughters.

Before I start, note that when I say “knitting”, you can read that as any of the various fiber arts, including crochet, weaving, macramé, cross-stitch, etc. I talk about knitting here because that’s the one closest to my heart and I strive to speak from the heart. You can make stuff out of string however you want. We are all fiber friends.

The sense of touch#the-sense-of-touch

I love the aesthetics of programming. Sitting in a cool quiet room, techno thumping in my headphones, coffee mug next to me, while a neatly arranged field of glowing monospace glyphs stream across my screen. But there’s one sense unmentioned in that sentence: touch.

The sense we devote more neurons to than any other is curiously underutilized while pumping out code. It’s no surprise that some programmers fetishize keyboards. It’s just about the only part of programming that has any physical sensation at all.

I got into knitting a few years after the pandemic. While I have a variety of hobbies, most still involve staring at a screen and maybe pushing some buttons and turning knobs. When I wasn’t doing those, I was staring at a screen for work, or staring at a screen for not-work.

I don’t know if I have a good way to explain how much my body craved tactile experience by the end of that. It’s like my fingers ached. A deep hunger, but not for taste. I’d wander around the house, driving my wife insane, unable to sit down and get comfortable. My body was just screaming at me to do something.

My youngest daughter had just picked up knitting, and she taught me. At first it was frustrating and annoying. But once I had the basics down, it was like a a deep sigh felt in my hands. Knitting is so touch-centered. Skilled knitters can knit without looking at their hands at all—touch alone is sufficient.

Starting to work on a slanting stripes scarf for myself.

There are so many different kinds of yarn to work with and they all feel different. Cotton is tough and firm, like twine. Wool is soft and springy, forgiving as you pull stitches open to work them. Superwash wool is smooth and glides off the needles. Non-superwash wool has this very slight stickiness to it that makes the resulting fabric feel solid and whole. Thin fingering weight yarn wraps tightly around your finger like a reminder knot. Working it is like performing delicate surgery. Squishy chunky wool spreads your fingers wide and works so quickly it’s like fabric is spooling out of your hands on its own accord.

The wool for this hat is a mixture of alpaca and sheep's wool. It's soft, fuzzy, and just the tiniest bit scratchy.

Even needles each have their own personality. Stiff grippy bamboo with its dull clack. Less worrisome to use because stitches don’t slide off as easily, but harder work to push against the friction. Polished stainless steel where the stitches fly off the needles—good when you are done with those stitches but not so much otherwise.

An inauspicious start for our friend Toad.

Once you’ve worked a few thousand stitches into your muscle memory, you can watch your fingers form stitches almost of their own accord, hypnotically. Right needle opens the stitch and slides in. Left finger wraps the yarn around the needle. Right finger grabs yarn. Right hand pulls the needle back out, a new stitch formed and transferred to the other needle. Over and over, like a meditation in the body.

Knitting feels good. It is an intimate, constant reminder that we are a tool-using species with thousands of years of evolution giving us incredible dexterity and the emotional wiring to make us want to use it.

An open world game with optimized skill curve#an-open-world-game-with-optimized-skill-curve

Of course, you could get much of that same tactile joy by driving to your nearest yarn store and wandering around the aisles jamming your fingers into every ball and skein of yarn they have. (An activity I certainly also do and highly recommend.) Knitting isn’t just about having a hedonistic tactile experience. It is a skilled art with an unbelievably deep lore.

I used to be a game programmer, and I tend to look at a lot of activities through the lens of game design. Games are interesting because they are user experience distilled to its essence. When you are, say, using a banking app to transfer some money into savings, there is a user experience in play. But there’s also a utility, a real effect you are trying to have in the material world. You may use your banking app and be satisfied that it helps you save money even if the UX is trash.

But a game, almost by definition, doesn’t do anything “out there”. Aside from leaderboards and stuff, the point of a game is to be a low-stakes sandbox where you can play without, you know, accidentally deleting money out of your bank account. Because of that, it’s a pure user experience. All experience and nothing else.

Game designers have to make a user experience so good that you don’t care that you aren’t actually accomplishing anything at all with it. That makes the tools they use to think about gameplay experiences useful lenses to evaluate just about any kind of human endeavor.

Two aspects (at least) are relevant to knitting.

Linearity#linearity

A game is “linear” if there’s only one path from beginning to end, one way to play. Don’t particularly enjoy the desert level? Tough shit, you gotta get through it to get to the end. “Nonlinear” games let players choose among multiple paths to reach the end. “Open world” or “sandbox” games blow the gameplay wide open and let players go where they want when they want. The game may not even have an “end”.

Knitting is an open world game. There are all sorts of objects you can make out of yarn, and all sorts of styles and techniques to make them. Don’t like socks? Fine, you don’t ever have to go through a “sock making phase” to graduate into what you really want to make. Find stranded colorwork too fiddly? There’s intarsia or just buy a ball of self-striping yarn and let the yarn change colors for you. Is a sweater too big of a commitment? You can make hats forever.

I've made a lot of hats. They're quick projects, only take a single skein of yarn, and almost always fit the wearer.

Once you get past the very basics of getting loops on the needles and making stitches, knitting very rarely forces you to slog through something you don’t want to do in order to reach some other goal. Except weaving in ends, I guess, which is kind of a chore. But, honestly, it’s not that bad.

Skill curve#skill-curve

A skill curve is sort of like a learning curve. It’s an imaginary graph of how much effort it takes to reach greater and more rewarding levels of skill. Some skill curves are steep at the beginning and then flatten out once you’re over the hump. When you first start playing guitar and don’t have the hand strength, callouses, or dexterity to form chords, it’s really hard. But after a few days you can get the basics down. Then it’s pretty easy to learn more and more chords after that.

Other skill curves start shallow and get steeper. You can learn chess in a day and have fun playing it, but as you get more serious about it, each incremental increase in skill requires a greater commitment to studying the theory of the game.

Knitting has a marvellously smooth, user-controllable skill curve. There’s a small hump at the beginning. It does take a little while to figure out how to hold the needles, control the tension of the yarn with your fingers, and get your hands to work in concert to make stitches. It feels like you’re making a shadow puppet of a sewing machine. Your initial experience will be frustrating.

But you can push through that in an hour or so. In a day, you can learn a basic way to cast stitches onto the needles, make knit and purl stitches, and then cast off to finish the work. With just those, you can make scarves and dishcloths. You are a knitter.

Working on a swimsuit for Toad. The legs are stockinette stitch, just simple knit stitches over and over, switching colors every few rows.

Then the world is your oyster. There are thousands of patterns out there, each listing the techniques required. You can pick ones well inside your comfort zone and grow your skills slowly. Or you can challenge yourself to learn a bunch of techniques at once. There are dozens of tiny little independent tricks to learn, each it’s own little merit badge serotonin hit: long-tail cast on, increases and decreases, cables, etc. So many fun different ways to form stitches. Each is one bite-sized lesson and no matter how many or few you want to chew on at a time, there is a project out there that will satisfy your appetite.

I learned how to do a picot bind-off to make these tiny frog fingers.

It’s not just the steepness of the curve, but also its height. Some skill curves top out early. I suspect the world’s greatest kazoo player is not profoundly better at kazoo than I am. Others seem to have no limit, like the world’s best violinists or Go players.

Friend, the knitting lore goes deep. People have been developing this artform for literally over a thousand years. Knitters have sat there, brain semi-idle, while their fingers worked yarn for millions of hours. They had plenty of time to invent all sorts of crazy ways to tangle yarn up. You could knit full-time for the rest of your life and never run out of new things to learn. Knitting will never stop rewarding you.

This pattern taught me short rows, wrap-and-turns, and cable cast-ons.

Structured but not a game#structured-but-not-a-game

To be clear, I’m talking about knitting and videogames as a metaphor. I don’t think of knitting as a game to win. Like a lot of programmers, I am prone to pointing my dumb optimizing brain at random activities and trying to min-max the shit out of it. Ask my wife how many ways to make coffee I have tried to find the optimal effort/reward ratio.

When I knit, I do try to knit efficiently. If it takes 10,000 stitches to finish a single scarf, it behooves one to put some thought into the process. At the same time, knitting isn’t just a pointless exercise to scratch my gamification itch. It is a real artform, and I am making real objects.

Thousands of stitches, spooled out from time into space.

For me, knitting strikes a good balance between structure and unstructure. I like making music and when it goes well, it’s very rewarding. But sometimes I sit down and every melody that comes out sounds like a knock-off NES game (and not in that cool chiptune way). Or I’ll spend an entire evening working on the drum mixing and at the end I can’t tell if I made things better or worse. It’s too unstructured.

When I sit down to knit, I might make a mistake that needs unwinding and fixing. But for the most part, I can be confident that an hour spent knitting will get me closer to a beautiful finished object. It provides a reliable serotonin hit of “I’m making progress”. But it’s not so rigidly structured that it triggers my optimizing brain into sucking the joy out of it by turning it into math homework.

A time and a space#a-time-and-a-space

A particularly nice property of knitting is that it is able to provide that reliable gratification while accommodating all of the other complexities in my life. It takes very little time to make progress knitting. Unlike, say, painting, there’s almost no set up at the beginning of a session or clean up at the end. When I pick up the kids from school, I can get ten minutes of knitting in while I wait for the bell.

Working on a blanket square while I wait for the kids.

Nor does it take up much space. A zip-loc bag with a ball of yarn and two needles is basically all you need, which is always in my backpack when I get on an airplane now.

Knitting expands and contracts around not just physical space, but headspace too. Had a long day and want something mindless to help you unwind? Slap together a garter stitch scarf and just do the same stitch over and over again. Stressed out by work or grieving a loss and need something consuming to take your mind off it? Start a lacework or cable knit project and the chart and counting won’t leave room to think about anything else.

Working on a little dishcloth while sipping hot cocoa.

Whatever logistical or mental capacities you have, there is a knitting project that will tuck neatly into it.

And then at the end#and-then-at-the-end

So far, I’ve been talking about knitting as an activity. A personal hobby to kill time for your own joy. Kill time it does, but knitting isn’t just about whiling away the hours. It’s not playing solitaire or binging a TV show for the fifth time.

As yarn spools through your fingers and the hands twirl around the clock, an actual physical, beautiful object emerges at the end. Well, your first couple of objects may not be so beautiful. But even the lumpiest knitted scarf is imbued with something increasingly elusive these days: care.

The first real thing I knitted was a scarf for my mother-in-law. In retrospect, I can’t say it’s a great scarf. Kinda cheap acrylic yarn. Not really her color. 4x4 rib was about all I could handle complexity-wise at the time, and it means the scarf tends to bunch up on itself. But when she opened the package on Christmas and saw it, her eyes teared up. Mine are tearing up now writing this.

The scarf.

Because regardless of how good the object itself is, it is an inarguable testament to the fact that I chose to spend dozens of quiet hours making stitch after stitch, all the while thinking about her and how much she means to me. A fraction of my life’s wick that I burned for her and no one else.

In a world where so many seem to want to get more and more out of less and less, to automate and AI-ify everything until an infinite content firehose is blasting into every orifice of every consumer, hand knitting to me is the antidote. An acknowledgement that all we really have is time and thus there is no gift more precious than spending it on someone.

Also, once you finish a project, you get to buy more yarn. Because, if I’m honest, a little consumption feels kinda nice too.

OK, I’m sold#ok-im-sold

Anyway, this is what knitting means to me. Which, now that I read all this, is a lot more than I realized. If that didn’t pique your interest, fine. It’s not for everyone. I do hope you find something out there to spend your time on that provides as much joy as knitting does to me. You deserve that.

If this did make you want to give knitting a try, you’re probably wondering what next. Fortunately, there are, like, a million “learn how to knit” tutorials out there. One of the actually marvellous things about living in the world today is good access to lots of videos, and knitting is an activity that’s really hard to convey in book form. You kind of need to watch someone’s hands. Learning from someone in person is best, but if you don’t have that, YouTube is a pretty good substitute.

Keep in mind that everyone’s hands are different! There are many ways (a handful, heh heh) to hold the needles and form stitches because our anatomy and the texture of our skin varies so much. Watch a few videos and don’t worry if what works for them doesn’t work for you. Eventually, you’ll find one that does.

Expect to be challenged and frustrated at first. There’s a lot to going on all at once: controlling the tension of the working yarn, keeping the stitches from the previous row on the left needle, keeping the new stitches on the right needle, forming new stitches. This may be one of the first times you’ve used this many of your fingers doing different things all at once.

I promise that if you’re patient with yourself and give it a few tries, you will get over the hump. Once you can knit a swatch of garter stitch, everything else will come naturally over time.

So go your local craft store, buy a cheap pair of size 7 needles, a ball of worsted (i.e. medium) weight acrylic or wool yarn in a color you think is pretty, and give it a try. The worst that can happen is you’ll waste a few bucks. If you’re lucky, you might end up making your mother-in-law cry (in a good way).

Access Control Syntax

Mon, 26 May 2025 07:00:00 GMT

I’m still tinkering on a scripting language for my hobby fantasy console project. I’m ashamed to admit this, but up to this point, the language had absolutely no notion of modules. Literally every source file is dumped into one big global namespace and compiled together.

I always planned to have some sort of module system. I just hadn’t figured it out yet because I had other, harder language design problems to solve. I assumed that the module system mostly didn’t interact with other language features, so I could kick it down the road for now.

That was true until it wasn’t. I’ve been beating my head against the wall around generics for… oh God I just checked the Git history and it’s three years now. I still don’t have that pinned down. Parametric types are hard.

Anyway, one of the approaches I’m exploring does get tangled up in modules and scoping so now I have to figure modules out. This post is about one little syntax design question I ran into: how do you distinguish public and private declarations?

A basic module system#a-basic-module-system

Since my language is a scripting language, my ambitions for the module system are pretty minimal. Think more like Python or Dart than Java or C#.

Every file has its own top-level scope that isn’t shared with others. If you want to access top-level declarations from another file, you import that file. That makes its top-level declarations available in the importing file.

Of course, a module might have some declarations that are only for its own internal use and should not be made available when you import it. A module should be able to encapsulate parts of its implementation. Thus, I need a way for users to indicate which declarations are private and which are public.

What other languages do#what-other-languages-do

Every language out there has some kind of module system and an ability to control access (though it did take JavaScript about 20 years to get there, bless its heart). In some sense, it’s a solved problem. But they don’t all solve it the same way, especially if you dig into some of the more obscure corners of the language world. Let’s go on a tour…

Modifier keywords#modifier-keywords

The approach you probably already have in mind is modifiers before declarations. In Java, C#, PHP and others, that’s public, private. Maybe also more specific ones like protected, and internal.

It’s clear, explicit, and gets the job done. It lets you support a large number of flavors of access control if you need. It’s also extremely common, so easy for users coming to a new language to pick up.

The flip side is that it’s quite verbose. Java is reviled for being too boilerplate-heavy and wordy, and I believe that having private and public scattered throughout every single file is a major contributor to that.

Picking the wrong defaults doesn’t help. I used to program in C# professionally for several years and in that time, I can’t recall ever wanting a member to have internal access, which is the default. So I had to write public or private on basically everything. I think the same is true in Java land.

Rust improves the situation by having a shorter modifier, pub, and picking what is arguably the right default for a language designed for Serious Programming: private.

Modifier sections#modifier-sections

C++ has a strange but perhaps underappreciated spin on modifiers. It has access modifier keywords, but they apply to all subsequent declarations. This lets you write a keyword once and apply it to a whole bunch of things. That really does cut down the verbosity.

The price to pay is that it also makes the language curiously context-sensitive. If you’re ever defined a preprocessor macro that inserts code in a class and had to be careful where in the class you called the macro, you’ve run into this.

Access control sections mean that you can’t look at a single declaration and know what its access is. You have to know what section contains the declaration.

There’s also the funny historical thing where members in struct default to public and members in classes default to private. C++ is weird. Like a house built on the back of some Eldritch being whose architecture and plumbing reveals the unholy foundation it is mated with.

For presumably less Eldritch reasons, I believe Ada takes a similar approach.

Sigils in the names#sigils-in-the-names

Modifiers are clear but verbose. If you want something more syntactically economical, why not encode the access control directly in the name of the declaration itself? That’s what Python, Go, and Dart do.

Python’s system is a mixture of informal and language supported. A leading underscore in a name doesn’t prohibit it from being used outside of the module but it sends a signal to the user that they shouldn’t use it. Sort of “velvet rope” security. If a class member starts with two leading underscores, then it really is private. The language will name mangle it to make it inaccessible.

In Go, if a declaration’s name starts with a capital letter, it’s public. Otherwise, it’s private. Because Go allows any Unicode letter in identifiers, the definition of uppercase is not trivial. Wikipedia tells me there are 1,858 uppercase letters and 2,258 lowercase letters that can be used in identifiers in Go. Fun!

Dart only allows ASCII in identifiers and follows Python: If an identifier starts with _, it’s private. Otherwise it’s public. Unlike Python, this is strictly enforced by the language. Private names are completely inaccessible outside of the library where the declaration appears.

I don’t have enough Python experience to have an opinion there aside from finding __ pretty long. I work full-time on Dart and have only tinkered in Go so I’m highly biased, but Go’s approach has always felt strange to me. I’m sure it’s mostly habit, but my monkey brain really wants case to be used to distinguish kinds of declarations: leading capitals for types and lowercase for functions. Seeing types with lowercase names or functions with capital names gives me the same squishy feeling in my gut as watching The Polar Express. Dumb subjective biases aside, though, it seems to work fine in practice.

I’m much more used to Dart’s style. I don’t love it, but, eh, it gets the job done.

These approaches both have the advantage of being very terse. No modifiers, no additional reserved words. There are two strikes against them:

They’re obscure. If you’re new to Java and you see the word private, you probably don’t know what it does, but you can guess it has something to do with “privacy”. With Go, if you see that some names are capitalized and some aren’t, that tells you absolutely nothing about what’s going on. Historical baggage? Weird personal preference? Maybe the author is German and prefers capitalizing nouns?

Likewise, if you’re skimming some Dart code, why do some of the names start with _? Do you pronounce that when you say the name out loud?

You just have to be told what’s going on because the language’s own syntax doesn’t guide you towards an explanation.
They show up at the use sites. Access control is a property of a declaration. So, in principle, any syntax for specifying it should only appear at that declaration. When you go to use a name, either you can access it (in which case no syntax is necessary) or you can’t (in which case there’s nothing you could say). But with Go and Dart, every place you mention the name also carries the access control.

In theory, this could be a major problem if you want to change the access control of a declaration. You have to fix the identifier at every single use site! In practice, this doesn’t actually cause much pain. If a name is going from private to public, then every existing use site is already confined to one file, so you can rename those without affecting the rest of the program. If an identifier is going from public to private, again all uses must already be confined to that file or you can’t make it private without actually breaking things.

I used to say that if I had a time machine, I would go back and change Dart to not use _ for privacy. But I have mellowed on that opinion over the years and now I’m not sure. I don’t like how it shows up at every use site, but it is terse, which is nice.

Export manifests#export-manifests

Modifers put access control at the declaration. Sigils put it at the declaration and the use sites. The third major approach puts it at neither.

Someone will tell me this generalization is wrong but I think most functional languages including the ML family and Lisps (including Scheme) have some kind of separate syntax for listing exports. Within a module, you declare all the functions and types you want without worrying about modularity. Then at the top of the file, the language gives you some dedicated syntax to list the things that should be made visible outside of the module.

For example, in Scheme, the syntax for defining a module looks like:

(library library-name
  (export list-of-names-to-export...)
  declarations-in-library...)

SML and its spawn have a notion of a signature which is declared separately from the module. You could also argue that header files in C and C++ are this pattern.

A cool thing about export manifests is that they keep all the access control logic in one place. When you’re writing declarations, you don’t have to worry about access control then. You just write declarations. As a user of a library, you can look at just the export section to see what the module lets you do without digging in to the implementation at all.

It does a good job of firmly separating interface from implementation.

On the other hand, it’s quite verbose. You end up saying the name of every exported declaration twice. Maybe even the full type signatures too. Since the export manifest is separate from the declarations, they have to be manually kept in sync. Rename an exported function and you have to remember to rename it in the manifest too.

I ragged on JavaScript for not having its act together, but I gotta say when they did add modules, they came up with a clever blended approach. You can use export as a modifier right on a declaration:

export function area(width, height) {
  return width * height;
}

But you can also use the export keyword to export a set of declarations that are already declared elsewhere:

export { area };

// Code...

function area(width, height) {
  return width * height;
}

Given that modules were added later to the language, I’m guessing this makes it easier to add modularity to a large existing JS library.

Sigils at the declaration#sigils-at-the-declaration

In poking around the annals of language history, I found one last approach in Niklaus Wirth’s magnum opus language Oberon.

In Oberon, top-level declarations are private by default and only accessible to the containing module. If you want to make a declaration public, you mark the declaration with an asterisk:

MODULE Hello;
  IMPORT Out;

  PROCEDURE SayHello*;
  BEGIN
    Out.String("Hello World!");
    Out.Ln;
  END SayHello;
END Hello.

Here, the * after PROCEDURE SayHello means that SayHello is public. Note that this marker is only at the declaration. It’s not part of the name. At callsites, the name is just HelloWorld.

This has the same inscrutability that Python, Go, and Dart have. If you’re a new Oberon programmer and you see *, you have no idea what it means. Fortunately, there are no new Oberon programmers, so this is not a problem in practice.

This approach has the advantage of only appearing at the declaration. You don’t have to repeat the sigil every time you use the thing.

Any syntax at all?#any-syntax-at-all

I’m sure there are interesting tweaks and refinements in various languages, but I think that mostly covers the approaches in use out there. (If you’re aware of any others, do tell.) Which approach is the right one for my little scripting language?

A deeply related question is the choice of defaults. Java has public for public stuff and private for private stuff, but it also has a syntax for package private declarations: nothing at all. If you don’t specify anything, C++ makes your struct members public and your class members private.

It’s hard to beat “zero syntax at all” when it comes to brevity, so the choice of what access control a declaration gets by default is an important one.

When I put on my rigorous software engineer hat—probably some kind of construction site hard hat—the obvious right default is private. No one can fit an entire huge program in their head, so code needs to be broken into smaller isolated pieces you can reason about locally. Defaulting to private encourages users to make smaller independent modules.

If I was making a language for big mission-critical infrastructure software, I would definitely do that. I think Rust made the right choice by defaulting to private and requiring you to opt in to public with pub.

But I am making, like, the opposite of that. A hobby project that may never see the light of day for people to make their own hobby games that, realistically, also may never see the light of day. If anything, I want to encourage (hypothetical) users of my (currently mostly vaporware) language to get things done. Access control can get in the way of that.

Class modifiers in Dart#class-modifiers-in-dart

A few years ago, we added a slew of class modifiers to Dart. These are not directly tied to public and private, but in the process of designing those, I spent a lot of time talking to users about how they use the language and what defaults they prefer.

Dart defaults to public. You have to opt in to private by prefixing a name with an underscore. It turns out that most users actually do prefer public being the default. And after looking at mountains of Dart code, what I see is that most Dart code is relatively small applications. Those apps are built on top of a set of frameworks and libraries, but compared to the volume of application code out there, those libraries make up a fairly small fraction of the entire Dart ecosystem.

These apps are the leaves of the dependency tree. Nothing depends on them or imports them. The authors of most of those are just trying to slap some UI together and ship a thing. And for most of those, defaulting to public keeps the language out of their way.

Syntax for privacy#syntax-for-privacy

That all leads me to feel that my scripting language should default to public and have a way to opt in to private. How do to that?

No export manifests#no-export-manifests

I know I don’t want export manifests. They are much too verbose. Also, there is some extra language complexity and error reporting that gets sucked in. Since the manifest is separate from the declaration, the compiler has to handle cases where you try to export a name that doesn’t exist. My language also has overloading, which means when exporting a function, you would need to a way to specify which of the overloads to export and which to leave private.

Export manifests are just way more machinery than fits for a scripting language. This approach is out.

No modifier sections#no-modifier-sections

The C++ approach is nice in that it’s clear while still being pretty terse. You only need to write a modifier and can apply it to a bunch of declarations in the same scope.

But at least in C++ (not sure about Ada), that only works within a type declaration. My language isn’t really object oriented. It’s more procedural. Sort of “structs plus functions” like C. Instead of classes, you have records. But a record declaration tends to be pretty small with only a handful of fields in it. There’s not enough stuff inside a single record to benefit from reusing a modifier for a bunch of nested declarations.

I could allow private at the top level and have it apply to everything after, but that feels like it errs too far in the other direction. It would be really confusing to have a thousand line file and not realize that half of it is private because some modifier way offscreen above flipped the access control.

Modifier sections are out.

No sigils in identifiers#no-sigils-in-identifiers

Sigils in identifiers do have a sort of terse, scripty feel. No extra keywords.

I can’t do Go’s approach because my language already makes identifier leading case significant. It supports destructuring pattern matching. Like some functional languages, it uses case to distinguish when a pattern is binding a new local variable (lowercase) versus matching against a type or constant (uppercase).

Now I’m not sure that corner of the language will stick, but it’s there right now.

I could do Python/Dart’s approach and use a leading underscore. But I’d really prefer to not have to mention the sigil at every single use site. After having written probably a million lines of Dart code, I know it’s not intolerable. But it feels like an annoying tax. And for a language I’m designing for my own joy, I’d like to eliminate as many annoyances as possible.

Maybe modifiers#maybe-modifiers

That leaves two approaches and is where I’m currently at. Approach one is the typical solution to use an access control modifier at the declaration. Unfortunately, defaulting to public makes this harder. In Rust, pub is a really nice little keyword to flip to public when the default is private, but there is no obvious converse. private would be the longest keyword in my language. pri doesn’t read like anything. priv is… strange. I suppose that pvt is the well-established abbreviation for “private” but that feels a tad militaristic.

I can’t find any other synonyms for “private” that admit reasonable abbreviations either. So I’m open to taking this path, but I’m simply failing to come up with a good keyword for it.

(A silly goblin part of my brain suggested I use shh to mean “private”, like the declaration is being whispered and can’t be heard outside of its module.)

Maybe sigils at the declaration#maybe-sigils-at-the-declaration

This leaves Oberon’s weird approach. There are several advantages here:

It is maximally terse. Zero syntax at the use site, and only a single character at the declaration site.
No additional reserved words which might get in the way of user identifiers.
By defaulting to public, the opacity of the syntax is less of an issue. Yes, the sigil won’t immediately convey what it does, but if you’re just starting to use the language, you don’t need to even know it exists at first. You can just make everything public.

So there is some appeal. I don’t think I can literally take Oberon’s syntax with * following the name. My language lets you define operators and it would be confusing to have * right after other punctuation:

def +*(left Vec, right Vec)
  Vec(left.x + right.x, left.y + right.y)
end

This is declaring a private + function. It would probably look equally weird to have * after the name in a record declaration:

rec Vec*
  val left Int
  val right Int
end

That led me to thinking the sigil should be by the declaration keyword. In my language, every declaration form does start with a leading keyword (unlike, say, C where function declarations don’t have one), so it’s feasible. Something like:

rec* Vec
  val left Int
  val right Int
end

def* sayHi()
  print("Hello!")
end

I don’t hate it. But using * feels weird for “private”. If anything, it seems to emphasize the declaration (which is what it does in Oberon where it means “public”).

Maybe alternate keywords#maybe-alternate-keywords

That led to my last idea which I can’t decide if I like or not. Given Python and Dart, it seems like underscore vaguely conveys “private” to some people. So maybe use that? It looks like:

rec_ Vec
  val left Int
  val right Int
end

def_ sayHi()
  print("Hello!")
end

It looks odd, but I don’t hate it. Or at least I don’t hate it any more than any novel programming language syntax usually triggers revulsion.

However, there’s sort of a problem. Underscore is a valid identifier character. So the lexer is not going to scan rec_ as a rec keyword token followed by a _ token. Instead of will treat rec_ as a single identifier.

But then… I could just let it do that. I’d then define a separate set of reserved words for all of the private declaration keywords: def_, rec_, var_, and val_.

So now instead of a modifier or sigil to control access, you explicitly choose one of two declaration keywords. One makes it public and one makes it private. This technically means adding a handful more reserved words, but they aren’t ones that are particularly useful for users anyway.

I keep trying to talk myself out of this approach because it’s so unusual but so far it seems to be lodged in my head better than any of the alternatives I’ve considered. What do you think?

Does Go Have Subtyping?

Thu, 19 Oct 2023 07:00:00 GMT

I’ve been noodling on a static type system for my current hobby language. To try to keep the language as simple as possible, I’m trying to see if I can live without subtyping. Since most of my programming experience is in object-oriented languages, I’ve been learning more about languages that lack—or at least claim to lack—subyping, to see how they work.

The most intriguing one to me is Go because the authors say it doesn’t have subtyping, but when you look at interfaces, it does seem to have something really close to subtyping. Is it subtyping just under another name, or is there really something different going on?

This post is the answer to that question as best as I can tell. The short answer is that no, Go doesn’t have subtyping. But also, yes, it sort of does.

What is subtyping?#what-is-subtyping

If you’re reading my blog, you probably already know what subtyping is, but let’s make sure we’re all starting from the same place. Subtyping defines a relation between two types. Given two types A and B, it might be the case that B is a subtype of A, or it might not be.

Since subtyping is a relation between a pair of types, it only comes into play in places in code where two types are involved. The main place is assignment. You have an expression of type A and you assign the result to a variable with type B. Is that assignment allowed?

Programming language folks usually generalize “assignment” to mean any place where a variable is given some value. That includes assignment expressions, but also covers initialized variable declarations and function calls where argument values passed to the function are bound to their corresponding parameters.

There are a couple of other places where subtyping comes into play, usually around type inference, but assignment is the main one: You have a context that requires some type B and a value of some type A. What are the types A and B where that code is valid?

That question is the heart of what a type checker does. The main user interface of a static system is compile errors, and the most common compile error is “I expected a value of this type but you gave me a value of this other type”.

Why have subtyping?#why-have-subtyping

You have a context that expects type B and you give it a value of type A. In languages without subtyping, that’s only OK if A and B are the exact same type. In Pascal, if you declare a variable with type integer, the only thing you can initialize it with is a value of type integer.

Subtyping exists largely to loosen that restriction—to allow multiple different types to flow into some context. Why might a language want to permit that?

The reason is polymorphism: Subtyping lets you write a piece of code and reuse that same code with a range of different (but related) types. In languages without subtyping, you can often find yourself copy/pasting the same function to work with multiple different input types. (Generics can help, but that’s another form of polymorphism that we’ll ignore for this post.)

In, say, Java, if you define a method that takes an Iterable, then you can pass a List to it, a Stack, etc. You get to amortize the usefulness of that method across all types that implement the Iterable interface. Subtyping is a force multiplier for your code.

(Of course, that benefit isn’t without significant costs in terms of language complexity, which is why I’m hoping to avoid it.)

Does Go have subtyping?#does-go-have-subtyping

If you search the (extremely well-written!) Go language spec for “subtype”, you get zero results. So the answer is a clear “no” at the textual level. However, we needer a deeper hermeneutics.

Java does have subtyping. Now, if you were to make a new language named “Blava” that was a literal copy/paste of the Java language specification with every use of “subtype” replaced with “blubtype”, would you say that Blava has subtyping? It behaves indistinguishably from a language with subtyping, so I’d be inclined to say yes.

The Go spec doesn’t mention “subtype”, but it does have a notion of “assignability”. When you have a context that expects some type and you give it a value of some other type, assignability determines which set of other types are allowed. Concretely, the rules are:

A non-interface type T is assignable to an interface type I if T implements I.

An interface type A is assignable to interface type B if A’s methods are a superset of B’s.

You know, that sounds an awful lot like subtyping. Is “assignable to” just Rob Pike’s idiosyncratic way of saying “subtype of”? Does Go have subtyping in everything except name? Are we just playing semantics? (I mean, we’re designing a programming language, so obviously everything we do is playing semantics. But I mean are we playing semantics with the language spec itself?) To fully answer that, we’ll need to look at all of the kinds of types in a program.

Composite types and variance#composite-types-and-variance

If the only types in Go’s type system were primitives like numbers, structs, and interfaces then I think you’d have a good argument that Go does have subtyping, just spelled differently. But once you start looking at slice types and function types, the story changes. (And array and channel types too, but slices and functions are enough to make the point.)

The thing that these latter kinds of types have in common is that they contain other types. A slice type has an inner type for the slice elements. A function type has a list of parameter types and a list of return types.

You ready for some more computer science jargon? We’ve been talking about relations on pairs of types like “is subtype” and “is assignable”. But now we have types that contain other types. That raises the question of whether a relation on the inner types of two composite types says anything about the relation between the two outer types.

For example, let’s say we have two slice types []E1 and []E2. They have element types E1 and E2, respectively. If E1 is assignable to E2 does that mean that []E1 is assignable to []E2? Does the assignability “propagate” from the inner types to the outer types?

Computer scientists call this property (meta-property?) variance. They phrase the question like “how does assignability of slice types vary with respect to their element types?”. There are a few possible answers to a question like this.

Variance of slice types#variance-of-slice-types

For slice types in Go specifically, there are a handful of assignability rules, but the only one that applies to slice types is:

V and T are identical.

In other words, for two slice types to be assignable, they have to be the exact same type. That in turn means they must have the exact same element types. Even if two element types are assignable, slices of those two types are not.

Judging by an endless series of confused people asking questions on StackOverflow, that behavior is unintuitive to programmers, both in Go and in other languages. Let’s say you have this Go program:

type Dog struct {
  name string
}

type Barker interface {
  Bark()
}

func (d Dog) Bark() {
  fmt.Println("Woof!")
}

Here we have a Dog concrete type, which is assignable to the interface Barker. So this is fine:

func speak(barker Barker) {
  barker.Bark()
}

func main() {
  speak(Dog{"Sparky"})
}

Given that, you might expect this to work too:

func speakAll(barkers []Barker) {
  for _, barker := range barkers {
    barker.Bark()
  }
}

func main() {
  dogs := []Dog{Dog{"Sparky"}, Dog{"Fido"}}
  speakAll(dogs)
}

But, no. The type system giveth and the type system taketh away:

example.go:29:11: cannot use dogs (variable of type []Dog) as
[]Barker value in argument to speakAll

If the type system didn’t yell at you, this program would be fine at runtime. All it does is call Bark() on every element in the array, and both Sparky and Fido do implement that method. So what’s the deal? In this case, the program is coincidentally fine because speakAll() is only reading from the slice. But what if we wrote:

type Tree struct {
  species string
}

func (t Tree) Bark() {
  fmt.Println("Rough (but not ruff)!")
}

func appendTree(barkers []Barker) []Barker {
  return append(barkers, Tree{"Elm"})
}

There’s nothing wrong with this appendTree() function. It adds a Tree to the given slice. Since Tree is assignable to Barker, that’s fine. But if you were to call this and pass in a []Dog, you’d end up with an array of dogs that had a tree stuck in it! That would violate the soundness of the language.

This is why Go only treats two slice types as assignable if they have the exact same element types. In PL parlance, slice types are invariant with respect to their element types. And, for a mutable data structure like slices, that rule makes sense.

(A reasonable person might wonder then why Java and C# don’t have this rule and instead say that array types are assignable if their element types are. And then, because as you can see, it isn’t safe to do so, they have to add runtime checks if you try to stuff an element of the wrong type into the array.)

So, OK, it makes sense for slice (and array) types to be invariant. What about function types?

Variance of function types#variance-of-function-types

To keep things simple, first we’ll consider just functions that don’t take any parameters and have a single return type. Given two function types like that, when are they assignable? Again, the only rule in the Go language spec that matches function types is V and T are identical. So two function types are only assignable if they have the exact same return types. Even if the return types are themselves assignable, if they are different types, the functions aren’t assignable.

Do we need to be that strict to preserve soundness? Actually, no! Here’s an example:

func returnDog() Dog {
  return Dog{"Rex"}
}

func useCallback(callback func() Barker) {
  barker := callback()
  barker.Bark()
}

func main() {
  useCallback(returnDog)
}

So we have a function, returnDog that returns a value of type Dog. We pass a reference to that function to useCallback() whic expects a function that returns a Barker. The Dog type does implement Barker. If this program were run, it would be perfectly safe. And, in fact, there’s nothing you could put inside useCallback() that would make passing returnDog to it violate the soundness of the type system.

It’s sound and semantically kosher in principle… but Go disallows it:

./prog.go:49:14: cannot use returnDog (value of type func() Dog) as
func() Barker value in argument to useCallback

Every other language I know that has subtyping and function types allows this. A function type A is a subtype of another function type B if the return type of A is a subtype of the return type of B. So the subtyping relation of the return types propagates out to determine the subtype relation of the function types. We call this covariance and say that function types are covariant in their return types. The “co-” prefix means that the subtyping relation between the inner types goes in the “same direction” as the subtyping relation it implies about the outer types.

That direction matters because relations like subtyping and assignability aren’t symmetric. The Dog type is assignable to Barker, but Barker is not assignable to Dog. The underlying value might be a Tree!

Are there cases where the variance of an inner type doesn’t go in the same direction as the outer types? Indeed there are, and they’re right there next to us. Instead of return types, let’s look at parameter types. Now let’s say we only care about functions that accept a single parameter and return nothing. Here’s an example:

func acceptBarker(barker Barker) {
  barker.Bark()
}

func useCallback(callback func(Dog)) {
  callback(Dog{"Laika"})
}

func main() {
  useCallback(acceptBarker)
}

Note that the parameter types are flipped compared to the return type example. Here, the callback type in useCallback() takes a more precise type of Dog. The function we pass to it, acceptBarker has a parameter whose type is Barker.

You may feel a slight disorientation here. The code feels weird and sort of backwards. Wait a minute and the dizziness will pass. Dramamine might help.

While this definitely isn’t as intuitive as return types being covariant, if you think about it carefully, you’ll see that the above program is completely sound. In other languages with subtyping, function type A is a subtype of function type B if the parameter types of B are subtypes of the parameter types of A. Note how A and B are reversed in the second half of that sentence. The variance of parameter types is reversed. In technical terms, we say that function types are contravariant in their parameter types. The prefix “contra-” means “against”.

(You might wonder what happens when you have a function type with a parameter whose type is itself a function type with some parameter type. How does that flow out? When there’s two levels of nesting it flips around to going in the same direction as the outermost type. The way I think about it is that contravariance is a 180° flip in the direction of the relation. If you nest contravariant types, you flip it twice and get back to the original direction.)

Contravariant parameter types are sound, but again Go doesn’t allow them. Two function types are only assignable if their parameter types are exactly the same.

Invariance in Go#invariance-in-go

In every language I know with subtyping, function types are covariant in their return types and contravariant in their parameter types. But in Go, function types are invariant.

Go is not a language known for getting in the programmer’s way when they want to do something, so why are function types more restrictive than would be necessary for soundness?

It’s not just function types either. All composite types are invariant in Go: arrays, slices, channels, maps, functions. So ground types—types that don’t contain any other type—have some subtype-like notion of assignability. But once you wrap a type in another, any notion of assignability goes away.

Why did the designers of Go do that? If you’re going to bother having interfaces and assignability, why not go all the way and have assignability for functions and other composite types where its sound?

If all the designers cared about was semantic correctness and having a beautiful elegant specification written in LaTeX, then they probably would have supported variance, at least for functions. (The other types all should be invariant since they are mutable. When types can flow both in and out, any other variance isn’t sound.)

But Go was designed from day one to be a high-performance systems language. It’s the exact opposite of an ivory tower language designed for proofs and publications. The goal of the language is to let real users ship real applications. And, importantly, be able to ship fast applications and reason about the performance of their code.

Representing values#representing-values

Up to this point, we’ve only been concerned with how types flow through the type checker at compile time. But—assuming there are no compile errors—the compiler eventually excretes some machine code which gets executed at runtime. When that happens, all of the types the type checker poked at have cracked out of their chrysalides and emerged as beautiful runtime value butterflies flitting around in memory.

The choice of how values of different types are represented in memory has a massive effect on performance. So how do the rules around assignability and subtyping interact with those representation choices?

In many object-oriented languages (Java, C#, Python, etc.) values of object types are represented by pointers to a heap-allocated structure. That structure has some header information for garbage collection and runtime type tracking, maybe some kind of pointer to a vtable for virtual method dispatch, then (finally!) the memory used to store the instance’s fields.

There are differences between language implementations, of course, but objects are generally both:

Slow to create since they are allocated on the heap.
Fairly large with some additional bookkeeping information stored for every single object.
Indirect, where a variable or field whose type is an object holds only a pointer to that object, which is always on the heap. Accessing state on an object always requires a pointer indirection which can be slow due to poor locality and cache misses.

Struct types#struct-types

Those are unnacceptable costs for a systems language like Go. When you want runtime polymorphism, of course, you have to pay for it somehow. But if you’re just storing data in memory, Go doesn’t want to make you pay for something you aren’t using. To that end values of struct types in Go store just the bytes needed for the struct’s own fields.

If a field of a struct is itself some struct type, the inner struct’s fields are splatted directly into the surrounding struct’s contiguous memory. If you have a local variable of a struct type, the fields are stored right on the stack (unless you take a pointer to the struct which escapes the function).

This reduces memory overhead for structs and (probably more importantly for performance) reduces pointer indirections. In a typical Java program, the heap ends up being a huge spiderweb of tiny objects all pointing to each other and the poor CPU an exhausted spider traipsing all over that web trying to find the actual bits of data it wants to eat.

In a typical Go program, more state is stored directly on the stack, and the heap is “chunkier” with fewer, larger blobs of memory. The CPU does fewer hops around the heap and chews on bigger data insects every time it does. That makes memory access more cache friendly and also lightens the load on the garbage collector since there are fewer individual allocations to traverse.

(Java does something similar for primitive types, as does C# for struct types.)

Interface types#interface-types

So structs are fast, great. But Go does feature runtime polymorphism in the form of interfaces. How does interface method dispatch work if a value is stored directly inline with no extra data to track its runtime type or method implementations?

The answer is that interfaces have a completely different runtime representation. A variable of interface type takes up two words:

A pointer to the type information used for runtime dispatch of the interface methods (in other words, basically a vtable).
A pointer to the actual data used by the concrete type implementing the interface. (In cases where the data is just a single word, I think it’s stored inline.)

The cute industry term for a representation like this is fat pointer: instead of a single word with a single pointer, it’s a pair of them, one for data and one for some kind of metadata or bookkeeping information.

One of the really cool things about Go is that you only use this representation—you only pay for the increased memory and indirection cost of this representation—when you ask for it and when you need it. In places where you need some virtual dispatch, you use an interface type and accept the overhead of a fat pointer and indirection. But in places where you just want to store a single concrete type, you use its underlying type and the memory is stored directly inline.

C# supports a similar distinction with classes and structs. But that’s mostly a “declaration time” choice. Once you’ve decided something is a class, every variable of that class’s type will store it as a reference to a heap-allocated object. Conversely, if you’ve declared something as a struct, it will always be stored inline on the stack or in the containing object (unless you go out of your way to box it).

In Go, the distinction between stored inline versus stored indirectly is made at each use site. That leads to some additional complexity for the user: they always have to think “should I use an interface, pointer, or struct type here?”, but it gives them more fine-grained control over how they spend memory and pointer indirection costs.

Implicit conversions#implicit-conversions

We’re close to understanding why Go lets you assign a struct to an interface but not a slice of those same structs to a slice of that same interface.

If structs and interfaces have entirely different memory representations, how does assignability work at all? When you do:

type Dog struct{
  name string
}

type Barker interface {
  Bark()
}

func main() {
  var barker Barker = Dog{"Rex"}
}

Shouldn’t that just mangle memory when it tries to treat the memory representation of a struct as if it were an interface? The answer is of course no. When compiling your code, Go knows the type of every variable and every expression. At every assignment, variable declaration, or parameter binding, it reports an error if the value isn’t assignable to the destination.

When the value is assignable, the compiler also knows whether or not those types are the same. If they’re exactly identical, then the assignment can be compiled to a single register move or memory copy. When they are different but still assignable types, the compiler silently inserts code to convert the value type’s memory representation to the destination type’s representation.

When you assign a value of a struct type to an interface type, the compiler inserts code to build a fat pointer, wire up its method table pointer to the right interface implementation, move the struct’s data onto the heap, etc.

Likewise, if you assign one interface type to another, the compiler inserts code to copy the data pointers over but look up the correct method table for the destination interface given the type information of the value’s interface type.

This right here is the reason that all composite types are invariant in Go. When assigning a single value to a related but different type, the compiler can easily insert fixed-cost code to convert the value’s runtime representation to the destination type’s. But to convert a slice of some struct type to a slice of an interface type would require an O(n) traversal of the entire slice to convert each element.

Function types are even harder. In order to support covariant return types and contravariant parameter types, the compiler would need to insert conversion code somewhere, but there’s no right place to put it. Putting it inside the function itself doesn’t work because it might be called with a variety of different parameter types and we don’t know what to convert it from. Putting it at the callsite before the parameters are passed likewise wouldn’t work because we don’t know what types every callback might require.

There is potentially something clever you could do by supporting multiple entrypoints to functions for each pair of source and destination types, but with multiple parameters you quickly run the risk of exponential code size explosions.

This is why languages that do support subtyping and variance almost always have a uniform memory representation for all objects that participate in the subtype hierarchy.

Does Go have subtyping?#does-go-have-subtyping

If you make it this far, congratulations. This ended up being a much deeper dive than I expected. I learned a lot exploring this corner of the language space and I hope you learned something too.

Getting back to the original question, I think we could accurately describe Go’s subtyping story in two equivalent ways:

Yes, Go has subtyping, but it has no support for variance and all composite types are invariant. This is, I think, how someone who is focused only on the abstract semantics of the language would describe it. If you were writing papers about type systems and needed to model Go’s you might adopt this perspective. If you didn’t care about how Go could be efficiently implemented because you were treating it purely as an abstraction, then this is a good way to look at it and compare it to other languages.

The main problem with the looking at the language this way is that it obscures why every composite type is invariant.
No, Go doesn’t have subtyping, but it does have implicit conversions between some pairs of types. This is how the designers of Go describe the language. It’s the way you’d want to look at the language if you were tasked with sitting down and writing a production quality implementation of it. It describes what the language actually does mechanically at compile time and runtime.

The challenge I found with this perspective is that it made it harder for me to relate Go’s design choices to other more explicitly object-oriented languages. You can look at this entire long article as my process of trying to figure out the first interpretation in terms of this one.

I started digging into this not because I’m an active Go user and want to know what’s going on under the hood. My job and hobby is designing programming languages, so I want to know how other languages work to see what good ideas are out there to be harvested.

So the question always on my mind at this point is, “Why did they design it this way and does that choice make sense in other languages?” And for this specific design choice, I think it’s pretty cool. You can imagine a language wanting three things:

Non-uniform representation: Values in memory take up only as much space as they need and avoid pointer indirection when possible to maximize runtime efficiency.
Polymorphism: The ability to reuse code to work with a range of values of different types.
Variance: Sort of the “lifted” form of polymorphism: The ability to reuse code to work with composite types that contain a range of inner types.

Those are all nice to have features, but it’s really hard to get all three at once. Most object-oriented languages sacrifice the first one to get the other two. That gives you flexibility and expressiveness but at a pervasive runtime cost spread throughout the entire program.

Some statically-typed languages with simpler type systems like C, Pascal, and SML (ignoring modules, which are a whole other thing) give up polymorphism and variance which can give you more efficient representations at the cost of less code reuse.

Languages like C++ and Rust more or less give you all three at the expense of the compiler monomorphizing and generating specialized versions of a function for the different types it gets passed, which makes compilation much slower and can have some runtime costs from all of the extra code sitting around in memory.

Go is aiming for a sweet spot where they give you fast compiles, efficient runtime execution, and as much flexibility as they can get away with. It sacrifices variance but keeps polymorphism at the individual value level. That married with implicit conversions enables non-uniform representation. Of the three, variance is probably the least valuable for users, so I think that’s a pretty smart trade-off.

Representing Heterogeneous Data

Fri, 04 Aug 2023 07:00:00 GMT

As I mentioned in the last post, I’m working on taking my little videogame scripting language and turning it into a statically typed one. As much as possible, I’m trying to make the language simple and familiar. But sometimes those goals are in opposition and the most familiar solution to a problem is kind of a mess.

So, I’m also exploring novel approaches and delving deeper into programming language history to scavenge forgotten ideas.

The heterogeneous data problem#the-heterogeneous-data-problem

One problem every language has to solve is giving users a way to represent heterogeneous data. By that, I mean:

Data that might or might not be present. Imagine you have a record for storing a street address:
```
rec Address
  var number Int
  var street String
  var apartmentNumber Int
  var city String
  var zipCode Int
  var state String
end
```
But some addresses don’t have apartment numbers. How do you store the apartment number when an address has one but also support its absence?
Data that might be in one of several different forms. You’re making a game where a hero can wield weapons. Melee weapons like swords have a single number for how much damage they do. Ranged weapons like crossbows have a pair of numbers for the minimum and maximum range they can reach. How do different kinds of weapons have different fields?

These are two sides of the exact same coin. You can treat optional data as data that can be in one of two forms: present with an associated value or absent with no value attached. Functional languages with an option or maybe type do exactly that: The language directly supports data that can have one of multiple forms, and they model absent data using that.

Conversely, you could model data being in one of several different forms by having separate fields for all possible forms it could be in. At any point in time, only one of the fields has a value and the others are all absent. If you’ve ever found yourself building a struct or class and writing a comment that says “If this field is blah then this other field will be null.” then you’ve taken this path (and probably felt a little gross doing it).

What other languages do#what-other-languages-do

I don’t know if broad language tours are your thing, but so much of my job working on Dart involves researching how other languages solve a problem that I can’t help myself anymore.

There are a handful of solutions to the problem. I’ll just throw out the ones I know:

Null. The most famously maligned approach is to allow any reference variable to potentially refer to “null”, “nil”, or nothing. This means every reference type can directly also represent an absent value. Of course, the problem is that many data fields aren’t heterogeneous and should always be present. If you make every single reference nullable, you’ve lost the ability to distinguish ones that can be absent from ones that really shouldn’t be.

This is why many newer statically typed languages either don’t support null at all (Rust and most other statically typed functional languages) or support non-nullable types (Dart, Kotlin, and TypeScript).
Variant types. A “variant” type is a type that can hold a value of one of multiple different types. You can think of nullable references as a limited variant type that can hold either a value of one specific type or the special null value but that’s it. Some languages have looser variants that let you store values of any type in the same variable.
Untagged unions. C lets you define a data structure whose fields all share overlapping memory. If you have a few different pieces of data that are disjoint—you should only ever have one of them and not the others—then this avoids the memory overhead of storing them all separately.

However, in C, the language itself doesn’t keep track of which piece of data you have in the union. It will freely let you write one field and then read out another and it will happily just reinterpret the bits in memory as that other type. Efficient, yes. Safe? No.
Sum types. Functional languages going all the way back to ML have a feature also sometimes confusingly called “unions” that is fairly different. Again, you have an object that can store one of a few different kinds of data. But the language also stores a tag in there so that it knows at runtime which piece of data you have. (This is why they’re also called “tagged unions” or “discriminated unions”.)

The language uses pattern matching to cleverly prevent you from accessing the data as the wrong type.
Subtyping. The object-oriented dual to sum types is subtyping: either inheritance or interface implementation. In an object-oriented language, we could model our weapon example like:
```
interface class Weapon {}

class MeleeWeapon implements Weapon {
  int damage;

  MeleeWeapon(this.damage);
}

class RangedWeapon implements Weapon {
  int minRange;
  int maxRange;

  RangedWeapon(this.minRange, this.maxRange);
}
```
Code that wants to work with weapons generally uses the Weapon supertype. The two subtypes for melee and ranged weapons each store the fields they need. If you want to go all the way to an object-oriented style, these fields would be private and then you’d have abstract methods in Weapon that are overridden in the subclasses to use them.

It’s a complex, heavyweight approach, but a powerful and flexible one.

There may be a couple of other weirder language features you can use to model varied data, but I think these are the big ones. Languages tend to pick and choose from this list:

Dynamically typed languages essentially treat all variables as variant types. And all the ones I know also go ahead and allow null too. If you’re not going to have any static checking anyway, you may as well be maximally permissive, I guess.
Statically typed functional languages lean really hard on sum types.
Object-oriented languages obviously primarily use subtyping, though most also have nullable reference types.
C makes pointer types nullable and supports untagged unions. It doesn’t have (checked) variants or subtyping, but it can approximate both by allowing pointers to be cast to different types. C++ takes everything C has and also explicitly supports subtyping.

Newer, bigger multi-paradigm languages like C# and Swift tend to take just about all of the approaches.

Whither for my little language?#whither-for-my-little-language

OK, so what’s the right approach for my aspirationally simple and elegant statically typed game scripting language?

I quite like object-oriented programming in general, but subtyping adds a lot of complexity to a static type system, so my current plan is to not have subtyping in the language at all. That rules out that approach.

My goal is for the language to be fairly high level and expressive. It’s supposed to be a language that makes making games fun, not necessarily a high-performance machine for engineering giant AAA games. I want you to have a good time tinkering on pixelly 2D games, not write the next Unreal Engine in it. To that end, the language is garbage-collected. That means memory safety, which rules out untagged unions.

Back when this language was dynamically typed, it had nil, so that’s an obvious approach. But I’ve spent, like, way too much of my time rooting out nullable references from Dart and the last thing I want to do with my hobby project is to go back to square one.

That basically just leaves sum types and variant types. Given that my language is statically typed and not object-oriented, sum types are the obvious approach. Everyone who uses sum types loves them, myself included. Algebraic datatypes are just cool.

And, in fact, I went ahead and implemented a protype of sum types and pattern matching and destructuring in my language. It worked. It was… just OK. To explain why requires a little context

An imperative, procedural language#an-imperative-procedural-language

My language is unabashedly imperative. I like imperative programming, especially for scripting little games. Games are giant balls of mutable state. I’ve watched my kids and many others learn to program, and imperatively modifying stuff seems to be a natural way to think about defining a process.

When you read a recipe for cake, you don’t see steps like: “Produce a new bowl of batter which is the previous bowl of batter and 2 cups of sugar.” It just says “Add 2 cups of sugar to the bowl.”

Now, I know all of the problems with mutation of state and imperative code when programming in the large. I get it. But this is supposed to be a fun little language for fun little games and, to me, imperative programming fits that to a tee.

The basic vibe I have for the language is similar to Pascal, C, or BASIC: In other words, a classic procedural language. Structures and functions. It looks like this:

rec MeleeWeapon
  var damage Int
end

def attack(weapon MeleeWeapon, monster Monster, distance Int)
  if distance > 1 then
    print("You are out of range.")
    return
  end

  var damage = rollDice(weapon.damage)
  if monster.health <= damage then
    print("You kill the monster!")
    monster.health = 0
  else
    print("You wound the monster.")
    monster.health = monster.health - damage
  end
end

What’s cool about simple procedural code is that even though I have no idea what language you know and you certainly have never programmed in this one, I’m still pretty confident that you understand this code.

With sum types#with-sum-types

Let’s see how it looks with something like sum types:

rec Weapon
case MeleeWeapon
  var damage Int
case RangedWeapon
  var minRange Int
  var maxRange Int
end

def attack(weapon Weapon, monster Monster, distance Int)
  var isInRange = match weapon
  case MeleeWeapon(damage) then distance == 1
  case RangedWeapon(min, max) then distance >= min and distance <= max
  end

  if !isInRange then
    print("You are out of range.")
    return
  end

  var damage = match weapon
  case MeleeWeapon(damage) then rollDice(damage)
  case RangedWeapon(min, max) then max - min
  end

  if monster.health <= damage then
    print("You kill the monster!")
    monster.health = 0
  else
    print("You wound the monster.")
    monster.health = monster.health - damage
  end
end

The sort of weird rec syntax is defining a sum type, Weapon, with type constructors MeleeWeapon and RangedWeapon. I’m still noodling on the syntax.

Now, the code here works. And it’s safe. The compiler and the structure of the pattern matching code itself prevent you from accessing the wrong fields from a weapon of a different kind. That’s cool.

But it’s so much weirder than the previous code. In a procedural language, the idiomatic way to access fields on records is simply record.field. That syntax is in almost every programming language all the way back to Algol. But once you hop over to sum types, you lose that syntax entirely and have to instead sort of “invert” the code and use pattern matching and destructuring.

I do love pattern matching and destructuring—I just spent the past year of my life adding it to Dart. But for this language, I’m pushing really hard on simplicity. If possible, I don’t want two different ways to access state on a value, depending on whether the field is case-specific or not.

More to the point, there’s no graceful way to handle mutable sum type fields using pattern matching. SML eschews mutability in general and then works around it by allowing you to define explicit mutable ref types. But that’s definitely not how my language rolls.

Variant records#variant-records

There is one other approach to heterogeneous data that I found that I didn’t put in the list up there because, as far as I can tell, it’s basically a dead end in the evolutionary history of programming languages.

Some versions of Pascal have a thing called “variant records”. A record in Pascal is your basic “collection of fields” struct type. A variant record says that some of those fields are only accessible when the record is one of a few different enumerated states.

In C, it’s common to wrap an untagged union in a struct along with a tag enum indicating which branch of the union is active:

typedef enum {
  WEAPON_MELEE,
  WEAPON_RANGED
} WeaponType;

typedef struct {
  WeaponType type;
  union {
    struct {
      int damage;
    } melee;
    
    struct {
      int minRange;
      int maxRange;
    } ranged;
  } as;
} Weapon;

Using it looks something like:

Weapon weapon;
weapon.type = WEAPON_MELEE;
weapon.as.melee.damage = 6;

A variant record in Pascal (as I understand it from the half dozen ancient slideshows I’ve been able to find about it) essentially models that pattern directly.

The cool thing about this feature is that the variant-specific fields are accessed using the same familiar field access syntax used everywhere else. That also means variant-specific fields can be mutable.

Of course, the not cool thing about using that same field syntax is that there’s nothing preventing you from accessing the wrong variant field:

Weapon weapon;
weapon.type = WEAPON_MELEE;
weapon.as.melee.damage = 6;

printf("Min range %d\n", weapon.as.ranged.minRange); // Oops.

There is a type tag, but the language doesn’t know and doesn’t check it. This is definitely true in C and I think true in Pascal. (It’s always hard to talk about Pascal definitively because there’s no “Pascal”, just a huge family of loosely-related Pascal-ish languages.)

In a memory safe language like mine, I definitely don’t want users to be able to reinterpret memory. But that’s a solvable problem.

Record cases#record-cases

Which, finally, brings us to the feature I designed for my language. It’s very close to variant records in Pascal. The type declaration looks just like the sum type example:

rec Weapon
case MeleeWeapon
  var damage Int
case RangedWeapon
  var minRange Int
  var maxRange Int
end

The difference is that you don’t need to rely on pattern matching to access the variant fields. They’re just fields:

def attack(weapon Weapon, monster Monster, distance Int)
  if weapon is MeleeWeapon and distance > 1 or
      distance < weapon.minRange or
      distance > weapon.maxRange then
    print("You are out of range.")
    return
  end

  var damage = if weapon is MeleeWeapon then
    rollDice(weapon.damage)
  else
    weapon.maxRange - weapon.minRange
  end

  if monster.health <= damage then
    print("You kill the monster!")
    monster.health = 0
  else
    print("You wound the monster.")
    monster.health = monster.health - damage
  end
end

Of course, you lose the compile-time safety that pattern matching gives you where you can’t access fields of the wrong type. But we don’t need to go all the way to C’s level of unsafety. Instead, when you access a case-specific field on a record, if the record’s type tag is set to a different case, the access throws a runtime error. This preserves memory safety.

This is a real trade-off. The feature I have here provides strictly less static safety than using sum types. There is a slight performance cost to checking the type tag when accessing case-specific fields. In return, you get simpler, more familiar syntax for working with case-specific fields, including mutable ones.

Also, it allows a single record to have a mixture of shared and case-specific fields:

rec Weapon
  var name String
  var bonus Int
case MeleeWeapon
  var damage Int
case RangedWeapon
  var minRange Int
  var maxRange Int
end

Here, name and bonus can be accessed on all Weapon instances, but the other fields are case specific. It sort of combines product and sum types into a single construct. I’ve found this to be really handy in practice.

I haven’t decided if I’m totally sold on this feature yet. But in the (admittedly small) amount of example code I’ve written using it so far, it seems to feel pretty nice. For a small game scripting language, I think it may strike a decent balance between static safety and simplicity.

Update: What about flow typing?#update-what-about-flow-typing

When I first posted this, the most common reply was why not do some sort of flow typing? In code like:

def attack(weapon Weapon, monster Monster, distance Int)
  if weapon is RangedWeapon and
        (distance < weapon.minRange or distance > weapon.maxRange) or
      distance > 1 then
    print("You are out of range.")
    return
  end

  # ...
end

The compiler could do control flow analysis to determine that the .minRange and .maxRange calls are guarded by an is RangedWeapon and thus allow them. But if you don’t guard the code with that kind of check, you’d get an error:

def attack(weapon Weapon, monster Monster, distance Int)
  if distance < weapon.minRange or # Error! Can't access .minRange here.
     distance > weapon.maxRange then
    print("You are out of range.")
    return
  end

  # ...
end

This is definitely a thing you can do! TypeScript, Kotlin, Flow, Dart, and others all support it. The general technique is called “control flow analysis” and the specific feature is called “flow typing”, “smart casts”, or “type promotion” depending on which language.

Is it a good fit for my language? I do like that it makes imperative code “just work” while being safe. But that “just” is doing a lot of heavy lifting. We do this analysis in Dart and it is fantastically complex. Proving that a certain piece of code can only be reached by going through some other piece of code first gets hard quickly in the presence of loops and closures. It seems like every release of Dart, we ship more extensions to flow analysis because users keep expecting it to be smarter and smarter.

Also, it isn’t sound in many cases that users expect to work. Once the variable that you’re type testing can escape the current function, the compiler generally can’t prove that it won’t be mutated between when you test its type and when you use it as the more precise type later.

Overall, my feeling is that it works out pretty well for Dart, but it’s a large sort of messy feature that feels a little too magical. A goal with my hobby language is that you should be able to have the whole language loaded into your head and rarely be surprised by what it does. Flow analysis in Dart still fairly often surprises me and I literally work on the language full-time.

There’s also the question of what you promote the tested variable to. In my language as it currently stands, there is no subtyping. MeleeWeapon isn’t a subtype of Weapon, it’s a case constructor. The weapon is MeleeWeapon syntax looks like a type test, but it’s really more like an enum case check.

So after that test, what type would weapon have? It would still have to be Weapon. I guess I could make this work by not promoting the type but by having the type checker track an extra “known case” property for each static type and then use that. That might work. But even with that, I worry that it would quickly become annoying. Let’s say you refactor the above code to:

def attack(weapon Weapon, monster Monster, distance Int)
  if weapon is RangedWeapon and checkRange(weapon, distance) or
      distance > 1 then
    print("You are out of range.")
    return
  end

  # ...
end

def checkRange(weapon Weapon, distance Int) Bool
  distance < weapon.minRange or distance > weapon.maxRange
end

That no longer works. Inside checkRange() the compiler has lost track that weapon is always a RangedWeapon. You could come up with a way to annotate that, but now we’re back to subtyping and all the complexity it involves.

So, overall, yes, subtyping and flow analysis is a thing that could work here, but I’m trying to avoid it because I feel like it’s a bigger lump of complexity than I want to take on.

I’d be more inclined to do sum types and destructuring, even though it feels a little weird in an imperative language, then do this kind of complex control flow analysis.

Type Checking If Expressions

Tue, 03 Jan 2023 08:00:00 GMT

I have this hobby project I’ve been hacking on for several years. It’s a fantasy console, very much inspired by the delightful PICO-8. Like PICO-8, my console has its own built-in scripting language. Because I’m me, I of course took the opportunity to design an entirely new language.

My goal for the project is a fun way to build small-ish 2D games. I want its scripting language to be expressive enough to be joyful, but small enough that you can learn the whole language and never need to consult a reference manual after that. My dream is a goofy little pixellated IDE where you can get lost in your own flow state and just make shit without having to periodically hit StackOverflow and then get distracted by the wonders/horrors of the Internet.

I don’t know if I’ll ever pull this off or the language will ever see the light of day, but it’s a fun therapeutic thing for me to noodle on.

A dynamically typed scripting language#a-dynamically-typed-scripting-language

To make a language that fits in your head (or at least my head, whose working space seems to get smaller every year), I needed to jettison as many feature as I could. My experience across a range of hobby and not-so-hobby languages is that static types add roughly an order of magnitude of complexity, so types were one of the first things to go. Like most scripting languages, I made mine dynamically typed.

Here’s an example:

def onTick()
  var d = 0
  if buttonHeld(2) then d = d - 1 end
  if buttonHeld(3) then d = d + 1 end

  if d != 0 then
    h = h + d
  else if h > 0 then
    h = h - 0.5
  else if h < 0 then
    h = h + 0.5
  end

  if h < -3.0 then h = -3.0 end
  if h > 3.0 then h = 3.0 end
  x = x + h

  if y < 200 then
    v = v + 0.8
  end
  y = y + v
  if y > 200 then
    y = 200
    v = 0
  end

  if buttonPressed(0) then
    if y == 200 then
      playSequence()
      v = -10.0
    end
  end
end

Another simplification I made is to eliminate the distinction between statements and expressions. As in Ruby, Kotlin, and most functional languages, everything is an expression. The previous chained if could be written in a more explicitly expression-y style like:

h = h + if d != 0 then
  d
else if h > 0 then
  -0.5
else if h < 0 then
  0.5
end

Unifying statements and expressions means the language doesn’t need a separate if statement and conditional expression. Also, I don’t know, I just like expression-oriented languages.

An imperative language#an-imperative-language

Even though the language is expression-oriented, it’s not explicitly functional. Functional languages are close to my heart, but this is a game scripting language. A game world is basically a big ball of incrementally updated mutable state. For the kind of programs and user experience I have in mind, I think an imperative, procedural style is easy to learn, and fun to program in. I want users thinking about their game, not, like, monads and persistent data structures.

So while everything is an expression in my language, it doesn’t at all shy away from side effects and imperative control flow. Variables are assignable. Fields are settable. There are loops and breaks and early returns. All of those are as natural and idiomatic as they are in C++, JavaScript, C#, or any of the other languages that the majority of the world’s code is written in.

Handmade Seattle#handmade-seattle

Last fall, I attended the wonderful Handmade Seattle conference. I had a particularly inspiring conversation with Devine Lu Linvega about their tiny uxn VM. They had this idea to build the smallest possible system and programming language for their own use. Then they rebuilt their own personal tools—text editor, music stuff, etc.—using that.

Now, UXN is really minimal. I get a certain satisfaction from programming in assembly, but it’s not the language I would want to use for my own joy. But it did make me rethink the scripting language for my fantasy console. I picked dynamic types because that made the language smaller and I figured it would be a good fit for my (entirely hypothetical at this point) users.

But is it what I’d want to use to make little 2D videogames? The game I’ve spent the most time hacking on is my also-perennially-incomplete roguelike Hauberk. I’ve rewritten it several times, but every incarnation has been in a statically typed language: C++, C#, Java, and now Dart.

My most pleasurable time spent working on Hauberk is when I’m refactoring and the type system guides me to what’s left to clean up. I just really like working with types. (It’s OK if you don’t. As our Burger Sovereign says, have it your way.)

After talking to Devine, I realized that if I was making this fantasy console for me personally, its language would be typed. So over the past few weeks, I’ve been sketching out a statically typed variant of my console’s scripting language. I don’t know if it will really come together, but I thought maybe it would be fun to write about the exploration.

Type checking `if` expressions#type-checking-if-expressions

I slapped together a new prototype interpreter for my language. (The main implementation is a bytecode VM in C++, which is pretty fast but not exactly easy to hack on.) Then I dutifully started adding a type checking pass to it. One of the first challenges I hit is how to type check if expressions.

As the title up there implies, that’s what this post is really about. Because it turns out that having if be an expression while also fully embracing an imperative style gets a little weird when it comes to type checking.

I’ll walk through a bunch of examples and build up to the type checking rules I have settled on (so far, at least). We’ll start simple:

var love = if isFriday then "in love" else "not in love" end

We need a type for the if expression so that we can infer a type for the variable love. In this case, the type is obviously String since both the then and else branches evaluate to strings.

So the basic rule we’ll start with is: An if expression’s type is the type of the branches.

Different branch types#different-branch-types

But what if they don’t have the same type? What about:

var love = if isFriday then "in love" else 0 end

Here, love could end up being initialized to either a String or an Int. Now what type do we choose? Crystal’s answer is String | Int. Union types are cool but definitely too complex for the language I’m trying to make.

In Kotlin, which is also typed and expression-oriented, the answer is, apparently, {Comparable<CapturedType(*)> & java.io.Serializable}. Which I have to say does not seem super helpful.

I assume that the compiler goes looking for a shared supertype of the two branch types, String and Int. Since String and Int both happen to implement Comparable (and I guess some serialization interface), you get that as the common supertype.

In object-oriented languages with subtyping and where the type hierarchy forms a lattice, this common supertype is the least upper bound, and it’s a natural answer to the problem. It shows up in other languages when type-checking conditional ?: expressions and a few other places.

It works, but, as we can see in the Kotlin example here, it doesn’t always produce intuitive or useful results. More to the point, one of the other features I jettisoned from my scripting language is subtyping, so LUB is off the table.

Without subtyping, every type is disjoint: a value of one type is never a value of any other type too. That means that if the two branches of an if have different types, then there is no possible type I can infer that contains all of their values. The only other response is to make it a type error.

That’s the next rule: If the branches have different types, it’s a compile error.

Imperative ifs and unused values#imperative-ifs-and-unused-values

That rule does work: It’s basically SML’s rule for if expressions. But I want my scripting language to feel familiar to users programming in an imperative style. Consider:

var daysNotInLove = 0
if isFriday then
  print("in love")
else
  daysNotInLove = 1
end

Here, the two branches have different types. The then branch has type String because in my language, print() returns its argument. (That makes it handy for stuffing some debug printing in the middle of an expression.) The else branch has type Int because an assignment expression yields the assigned value.

According to the previous rule, this is a type error because we don’t know what type of value the if expression evaluates to.

But it doesn’t matter since the if’s value isn’t being used anyway. There’s no need for the compiler to yell at you, and code like this turns out to be very common in practice.

To address this, the type checker takes some of the surrounding context into account. When an if expression appears in a location where its value won’t be used, then it’s no longer an error for the branches to have different types. How complex is tracking that context? Not too bad, actually. There are a handful of cases:

In a block or function body where you have a sequence of expressions, the result is the value of the last expression. The values of all of the preceding expressions are discarded. So in an expression sequence, all but the last expression are in a “value not used” context.
Like other expression-oriented languages, functions in my language implicitly return the value that the function body expression evaluates to:
```
def three() Int
  print("About to return three...")
  3
end

def onInit()
  print(three()) # Prints "About to return three..." then "3".
end
```
But if a function has no return type (the same as void or unit in other languages), it doesn’t return a value. In that case, even the last expression in the body is a “value not used” context.
Loop expressions don’t produce values, so their body is always a “value not used” context. (I’m toying with the idea of allowing break expressions to yield a value from the loop, but they don’t right now.)
Whenever an if or match expression is in a “value not used” context, then we push that context into the branches too. Likewise with the right-hand side of and and or logic operators since those are control flow expressions to.

That’s it. After I came up with this rule, I did some poking around and it seems like Kotlin does something similar. It frames it by saying that when you use an if “as an expression” then the two branches must have the same type. That’s roughly the distinction I’m making here too: when an if appears in a statement-like position where its value is discarded, then the branches can disagree.

Missing else#missing-else

This rule allows us to support an even more important flavor of if expressions that are common in imperative code: those without else clauses. In SML and some other functional languages, every if expression must have an else clause because the presumption is that you will be using the value produced by the expression and you need a value even when the condition is false.

But in imperative code, it’s obviously common to have ifs whose main purpose is a side effect and where an else clause isn’t needed. In fact, when I analyzed a huge corpus of real-world Dart, I found that only about 20% of if statements had else branches.

Now that we understand when an if expression is in a context where it’s value isn’t used, we can allow omitting else branches those. The next rule is: An if expression can omit the else branch when in a context where its value isn’t used.

Exiting branches#exiting-branches

We’re almost there. It’s starting to feel like we really are type-checking an imperative language, not ML in BASIC’s clothing. I coded this up and successfully wrote some little example programs. It was starting to feel like a real typed language!

I could stop here, but there’s one last bit of type checking logic for if expressions. I haven’t decided if it’s worth keeping. Consider:

def onInit()
  var love = if isFriday then "in love" else return end
end

When isFriday is true, then this initializes love with the string “in love”. When isFriday is false, then the return exits from the function entirely so love never gets initialized at all. So even though these branches don’t evaluate to the same type, love is always initialized with a String. This code should be fine.

Or, at least, it should be sound according to the type system. Whether this is good style is definitely open for debate. I could probably not allow code like this. But my default stance is to be as permissive as possible without breaking soundness, and this is a corner where I can be.

The trick is that expressions like break, return, and throw are special. While they are expressions grammatically, they don’t actually evaluate to values. If you do:

var x = return

That x never gets initialized. A return expression always jumps out of the surrounding code instead of producing a value. Languages that have expressions which can do control flow model this by giving these expressions a special type variously called “bottom”, ⊥ (“up tack”), Never, noreturn, etc. This type means “You’re never gonna get a value from me.”

When checking the two branches of an if expression, if one branch has that special type (the compiler calls it “unreachable” right now), then we just use the type of the other branch for the if expression’s type. That allows the above example to work. In the sample code I’ve written so far, it rarely comes into play. It’s usually more idiomatic to hoist that control flow out of the if entirely. But we can type check it easily, so the language lets you do it.

The rules altogether#the-rules-altogether

That’s where I’m at right now. It took me a few iterations to get to a point where all the if expressions I expected to be able to write in my example programs actually type checked correctly but it seems pretty stable now. The rules are:

When an if expression is in a context where its value is not used, then there is no restriction on what types the branches can have and we’re done.
Otherwise, there must be an else branch and:
If both branches have type “unreachable” then the if expression’s type is also “unreachable”.
If one branch has type “unreachable” then the if expression’s type is the type of the other branch.
Otherwise, the two branches must have the same type and the type of the if is that type.

Stupid Dog

Sun, 13 Feb 2022 08:00:00 GMT

It’s my 30th birthday, and Megan and I are walking down Duval Street in Key West. There is a pet store here, which must be the most unlikely place for a pet store in all the world. Everyone here is a tourist. What kind of idiot buys a dog while on vacation?

You are in the window surrounded by other puppies. A tottering orange ball of fuzz with little triangular ears, like a fox who lost its tail. Ink-black eyes and round button nose. My wife, pregnant and hormonal, is instantly in love with you. We go inside to play with the puppies, because why would you not want to play with cute puppies while on vacation? You curl up in my cupped hands, calm and sleepy.

Megan gives me a lingering series of Significant Looks, which I do my best to ignore. I generally try to avoid making dumb decisions, and impulsively buying a puppy while on vacation eight hours away from home is clearly in that category.

I make a silent vow to never let Megan know that I also fell in love with you the moment I saw you.

It’s the next day and Megan and I are sitting at a Burger King across the street from the pet store. She is presenting her case for why we should buy this very cute half-Pomeranian half-Yorkie puppy even though:

We are on vacation.
We are expecting our first child in a couple of months and already have more than enough to keep us busy.
We have absolutely no plans or intentions on getting a pet. Our house is tiny and barely fits the two of us and two cats, much less a baby and a dog.
We are an eight-hour drive from home.
You are, in fact, quite expensive.

Do I even need to write the numbered list here? It’s obviously a terrible idea. Family members that we’re vacationing with look at us like we are idiots when we say we’re considering it.

I think very very hard about my beloved first dog, Snickers, named because she was chocolate brown with swirls of white and caramel. She was a Sheltie, the smallest of her litter, a funny little runt of a puppy who entered my life when I was feeling small and cast out too.

When I was seven, I was playing in a neighbor’s front yard while she was out with me. I got distracted and heard a screech of tires behind me. When I turned around, Snickers was on her back motionless in the street. I carried her limp body all the way back to my house. The shocked look on my mother’s face when she saw me arrive, tears streaming down my face, broken dog in my arms, is my most vivid childhood memory.

I am the kind of person who analyzes 100 steps ahead. Getting a puppy means buying a ticket for the whole ride, including the end of it, in whatever form it takes. I am choosing to break my own heart. What if it’s not worth all that pain?

Megan is driving us north from Miami. You are sleeping in my lap, impossibly warm, cold wet nose pressed against my arm. We talk about names. You almost end up “Dorito” because of your triangular orange ears, but it doesn’t stick. Instead, you become “Ginny”. Not for Ginny Weasely like everyone assumes but short for the liquor genever.

When we buy you, I also buy some books on dog training because I am the kind of person who wants to be a very good dog owner that follows all the dog owning rules. Rule #1 is crate training starts immediately. A dog won’t respect you as the pack leader if they are free to sleep where you do. They’ll think they run the pack.

It’s late at night by the time we get home, and all the pet stores have closed. Next to the bed, I try to improvise some crate-like thing for your first night involving cardboard boxes, a cooler, and anything else I can find around the house. You yowl in terror, or perhaps just disappointment at the poor accommodations.

Eventually, I give up. I pick you up and lay you on my chest. You are asleep instantly, tiny heart fluttering against mine.

You are taking your sweet time exploring every single corner of the jungle that is our backyard, looking for lizards. It is three o’clock in the morning. The green and brown anoles are sleeping under leaves where you never find them. I am shambling after you in pajama pants and old boots telepathically trying to get you to pee with every ounce of willpower I have.

I am keenly aware of my dwindling opportunities to get a good night’s sleep before the baby arrives. Yet here we are. Over the past few weeks Megan and I have mostly learned to tell the difference between your yowl that means “I am a puppy and don’t like being alone in this crate” and the yowl that means “My bladder is too small to last the night and I need to pee right now.” Sorry about the mistakes we made while figuring that out.

I pee in the yard too, hoping you’ll get the hint. I deeply regret ever seeing you in that pet store window. Half-awake, I have vivid fantasies of building a time machine, traveling back in time, and slapping myself in the middle of that Key West Burger King.

After patrolling the perimeter of the yard in tiny puppy steps, twice, you finally pee. I tell you what a good girl you are and give you a treat.

Because you are an adorable tiny puppy and I am a nerd with a DSLR, I take copious photos of you and upload them to Flickr. One day, I get an email from cuteoverload.com asking if they can share some of them. The next day, you are Internet famous. Millions of people look at your butthole.

You are oblivious to your celebrity, as you are to most things in life.

As you grow, your puppy fuzz is replaced by silk, still orange on top and white on the bottom like a fox. The whorls of fur behind your ears are the softest thing I have ever felt in my life. I run my fingers through them constantly, to your delight.

I take you on walks every day, still trying to be some kind of adequate dog owner. The training manuals say that to teach a puppy walk properly on a leash you should stop walking whenever the dog pulls against it. Then wait patiently until the dog turns and makes eye contact with you. That way, they learn that you are in charge of walking and that pulling means the walk ends.

It takes us half an hour to walk a single block. Despite repeating this process hundreds of times, you fail to make the mental connection that if you stop yanking on the leash then we can actually walk where you want to go. You never, ever stop pulling on the damn leash.

I start a running joke of whispering insults in your ear that sound like terms of endearment. It’s not like you speak English, so when I say, “I hate your face,” all you hear is the caring tone I use to say it. “You’re so fucking stupid,” I coo into your ear. I rub your fuzzy belly to make sure you know how much I love your stupid face.

Five weeks earlier than expected, Megan and I become parents. We haven’t had time to set up a nursery yet so we hastily buy a bassinet. The first night the baby is home from the NICU, we realize there isn’t enough room at the foot of the bed for both the bassinet and the crate.

We let you sleep on the bed. You are immensely proud of earning a place on the bed while the new pink hairless dog is relegated to the funny crate.

You are listless and frothing at the mouth. Megan and I are terrified. You had been in the yard stalking all of the various animals that live in the barely-tamed-jungle that is Orlando. This time, you found a toad and you’d managed to get your mouth on it before I could get to you. Did it poison you? Are you going to die?

We rush you to the emergency vet. She takes your vitals while you sit there limp and apathetic. Your blood sugar is dangerously low. She gives you a treat. Then another. You immediately perk up. You’re fine, totally fine.

I remember that a couple of days prior I put my foot down and decided you were ready to graduate from soft puppy food to kibble which is better for your teeth. You were uninterested in the new food. I confidently told Megan, “If she gets hungry enough, she’ll eat. It’s not like she’s going to starve herself.”

Apparently, somehow during the thousands of years humans spent turning a wolf into the strange-colored, tiny, eternally cub-like thing that you are, we broke your survival instinct. Sorry about that. We learn to mix some wet food into your kibble.

It is something like four in the morning and Megan and I are pushing a baby stroller and a luggage cart across the SeaTac airport with you and the cats crammed in pet carriers perched on top of our suitcases. We’ve just flown 3,000 miles to our new home. Our brilliant idea to take a red-eye flight so the baby would sleep through it failed disastrously. Megan and I spent the entire flight taking turns trying to get the baby to sleep on our laps. You seem to have mostly survived the flight intact. The cats were less thrilled by the experience.

We get to the hotel, and I take you out just as the sun starts to come up. The parking lot of an Extended Stay in Redmond is my first view of our new home in the Pacific Northwest. It’s fifty degrees and rainy. I’m beyond tired. You take your sweet time before finding an acceptable spot to pee.

We are on PetFinder looking for a second dog. You are just so… weird. Not food motivated. Impossible to train. High strung. Frankly, not very smart. You bark all the time, at everything. We tell ourselves that a second dog will give you company when we’re out. They will teach you how to dog like a normal dog.

We find a part-Chihuahua part rat terrier that is, in defiance of everything I understand about genetics, practically your twin. We bring you to meet him at a dog park to see if you get along. You bark a lot. He tolerates you admirably well. For the rest of your lives, you will be his asshole boss. You’ll eat the best parts out of his food bowl and steal his treats. When he tries to sit next to us on the couch, you will wedge yourself in the middle and glare at us both.

He will live in fear of your ire which is capricious and often. We name him Benny for “beignet” because he’s golden with a sprinkling of white on top.

Maybe they were right about the crate training.

We rent a house out in the sticks. There is a green belt behind the yard and we get deer sometimes. Occasionally, we hear an owl hoot. Inexplicably, you are absolutely terrified of the sound. Megan and I hoot at you to tease you and you bark back at us, shivering in fear.

Later, we buy a house with a fenced in back yard. You stand proudly in the grass waiting for me to make eye contact with you. The moment I do, you throw your head back in delight and run full tilt to me. Over and over again from Megan to me and back. You wear yourself out and lay in the sun panting “keh keh keh”. When you snuggle with us on the couch later, you still smell like dirt and grass.

I am in the basement awkwardly taking photos of myself. I’m speaking at a conference and they asked for a headshot. You wander in and I pick you up and start taking pictures of us together. They are my favorite photos of myself.

When I finish my writing my book, I put one of them on the back cover. I get email from readers telling me how delighted they are with the cover.

I set that photo as my profile image on all of my various accounts. When ride share drivers pick me up, they joyfully tell me in not-great English that they love my dog. You are still a celebrity.

I am in bed staring at the dark ceiling, doing math in my head. Tonight, you are the same age in dog years that I am in real years. From this moment on, you are older than me. I picture you racing ahead of me into the future. I try not to think of where that path leads or when you’ll get there.

You are sleeping stretched out, pressed against my thigh, trying to get every square millimeter of contact that you can. I run my fingers through the silky fur on your back, feel your breath rise and fall.

We rent a cabin on the Pacific coast. You spend the first half of the drive shaking in excitement and anxiety, paws on the door looking out the window. The whole car smells like your horrid breath and Megan is coated in orange fur. Eventually you wear yourself out and pass out in her lap, head draped over her arm towards me. I rub the fur between your ears while I drive.

After settling in, we go explore tide pools. You sniff everything. Every rock, bit of kelp, rotting fish, even the wind. It’s cold and wet, but you don’t mind. When we get back to the cabin, I wipe sand and seaweed from your paws. You fall asleep in front of the fireplace.

The vet is explaining to us that you have hyperthyroidism and congestive heart disease. That’s why you pee all the time—often in the house—and why you seem so tense and wound up all the time. Well, that and because you are half Pomeranian.

You get put on an expensive pill regimen which you will take for the rest of your life. Since you aren’t interested in food even on a good day, getting you to take them will be a constant battle. At least the peeing inside gets marginally better. The barking does not.

I start writing a second book. I work on my laptop on the couch and you lay pressed against one leg the entire time. Once you’ve fallen asleep and let down your jealous guard over access to me, Benny claims the other side.

We take another trip out to the Olympic Peninsula with friends. You spend much of it sleeping on the couch in front of the fire, like an overweight self-warming throw pillow. You are given affection from everyone, which you regard as your proper due. In return, you grace us with an endless series of farts that leave the cabin smelling like some sort of reptilian tire fire.

It’s six o’clock in the morning. I am barely awake making coffee while you are outside for your morning consitutional. I hear a strange sort of yelp at the back door. I open it just in time to see you collapse. I scoop you up and run into the dining room, yelling out for Megan. I sit on the floor and cradle you, rocking back and forth. If these are your last moments, I want them to be peaceful.

I hold your limp body in my arms. Your tongue hangs out, lifeless and blue. I am seven years old again.

After an endless minute, you come back to us. Your back stiffens. Your legs poke out like a fainting goat and then relax. Your eyes start darting around and eventually you are able to pick your head up. Ten minutes later and you are back to normal, or at least as normal as you ever are. Honestly, a loss in cognitive function would be hard to detect.

The vet begins to use the phrase “congestive heart failure” now. The pill regimen gets more complex. We throw out yet another rug too pee-stained to repair and decide to not have a rug in the dining room “for now”.

You are mostly enjoying your retirement. You spend so much time laying on the back of the couch so that you can look out the back window and bark at anyone with the audacity to walk on the sidewalk that the cushion loses its shape entirely.

We alternate between trying to give you as much affection as we can in the time we have left and cursing your name as we discover another part of the house ruined by your sneaky peeing. I discover right in the middle of a push-up, my nose an inch from the floor, that you and Benny have surreptitiously saturated every inch of the carpet in the office. Megan and I learn how to put in new flooring.

We enter the phase where you take medications to cope with the side effects of other medications. You get over a dozen different pills a day, each on their own specific schedule. I print a chart and put it on the fridge to make sure I don’t miss any.

It is exhausting, but mostly for us. You are still relatively content, lazing around the house. Stealing Benny’s food. Barking at people outside the window.

Sometimes, at night, you crawl between my legs on top of the blankets, pinning me and forcing me to sleep on my back. I tolerate it as best I can. You’re warm and your ears are still velvet-soft.

One morning, I wake up to discover you have leaked little brown droplets all over the blanket. You are sitting up in bed, panting, clearly uncomfortable. (I’m not exactly comfortable with the situation myself.)

A couple of days later, you faint again. Perhaps a stroke. Dog.exe has stopped working. When you finish your reboot cycle, you aren’t the same. You don’t recognize Megan and snap at her viciously. You stand stock still in the middle of the dining room floor, eyes glazed.

It’s clearly time. I feel an unexpected relief. For years, I have held a deep fear that I wouldn’t know when it was over. That I wasn’t adult enough to handle the responsibility of making the call. I’d procrastinate too long and leave you in misery or choose prematurely and cut your life short. Or I would just be wracked in anguish the whole time. Your last gift to me is a swift decline so that there’s no doubt.

I learn there is an entire category of vet-adjacent businesses that specialize in this “transition”. Early on a Tuesday morning, I start calling them to see who can come out today, as soon as possible, right now. The women who answer the phone have well-rehearsed, soothing voices. I stumble over my words. I don’t know the right social protocol to ask, “Can you please come kill my dog?”

You are sitting next to me on the couch, pressed warmly against my side like always. You’ve mostly calmed down but are sitting up and tense. A kind woman radiating Pacific Northwest Earth Mother vibes is sitting on the other side of you with a small array of syringes in front of her on the coffee table. We’ve tucked a blanket under you.

The kids have already said goodbye to you, tears pouring down their faces. Surprisingly, that turns out to be the hardest part, realizing how much you are part of their stories too.

The first shot makes you fall asleep. Literally, not in the “put to sleep” figurative sense. As the woman pushes the plunger, you loll slowly over onto my leg. You’re napping, head on my thigh, more comfortable than I’ve seen you in weeks. Like old times.

I run my fingers through the fur on your back, up to where it gets shorter and velvety between your ears, around your side where it gets thinner and silkier, to your chest where it thickens into whorls. I leave my fingers there where I can feel your heart beating. I want to be here for it, to know that I was with you from beginning to end.

When the second shot goes in, your music stops.

With practiced timing, the woman has your body wrapped up in the blanket and out the door in minutes. The second the door closes, I feel the mask fall off my face and the tears start.

It’s later that day. I’ve thrown out the pills, pill poppers, prescriptions, aftercare instructions, wrecked towels and blankets, every reminder of your failing health.

I am out for a walk to get out of the house. It hits me a few blocks away. The answer to the question I carried since the day we got you. Will the life we have with you outweigh the pain at the end? Will it be worth it?

It is. Every minute.

It’s a couple of years later. Your ashes are in a little box in my nightstand. I remember the stress and exhaustion of the last few months of your life, but it feels vague, like I remember the stories I told people about it but not the actual experience.

I remember all of the good moments with you vividly. The feel of your fur between my fingers, your body pressed against my side, your horrific breath panting in my face. Megan and I talk about your antics.

I still miss you and your stupid face.

Ginny 2008 – 2019

640 Pages in 15 Months

Thu, 29 Jul 2021 07:00:00 GMT

My book Crafting Interpreters on programming languages is done. OK, OK. I know I said it was done like fifteen months ago. But now it’s really done. And by that I mean, the print, e-book and PDF versions are done. You can buy it. You can hold it in your hands. And I do mean “hands” plural. Because this little “handbook” turned out way larger than I anticipated:

This is a proof copy, so it looks a little different than the final design.

Look at that thing. 640 pages, eight inches wide, ten inches tall. If you get tired of reading it, it can serve as a doorstop or protect you from small-arms fire.

Remember back on Mr. Roger’s Neighborhood when he would take you to a factory and show you how pencils or umbrellas are made? I love that stuff, so I thought maybe you might like to see what I spent the past year on. You can read this as a peek behind the curtain, or maybe a long apology for why it took so long.

Where I left off#where-i-left-off

What I said in my last post was that Crafting Interpreters was finished, and by that I meant that I’d written all the code, dotted the last sentence, and scanned the last illustration. The content of the book was complete. But it wasn’t a book yet. Just a pile of Markdown and PNG files that my embarrassingly bad Python code would begrudgingly turn into a website.

My goal has always been to make a real book, with pages and everything. Completing the content was just one (granted, the most important) step in that journey. Once I finished the last chapter and put it online, I took about a month-long break. I had been writing every single day for almost four years and I was fried. Also, in case you don’t remember, early 2020 was not exactly a Totally Fine Time in World History.

A new build system#a-new-build-system

After a few weeks, my desire to have this project completely done returned and I eased my way back into working on it. I fixed a whole pile of typos and other mistakes that readers filed bugs for (thanks!).

Then, for no real good reason at all, I decided to rewrite the whole build system for the book in Dart. The build script I wrote for my first book was dead simple. Literally a single Python script that took a Markdown file for each book chapter and rendered it to HTML while weaving in the code snippets. The world’s dumbest static site generator.

I started with that for Crafting Interpreters but then it grew and grew. My second book includes every single line of code for two complete interpreters, which it builds up incrementally across thirty chapters. I needed to not just build the HTML for the book’s website, but also make sure the code really did work. I gave the build system the ability to not only generate the site for the book, but also to slice and dice the code. Given a chapter, or even a single point within a chapter, it can output a program containing all of the code for the interpreters up to that point. Then I can take that code, compile it, and run it through my automated test suite to make sure the code I’m showing you does what it’s supposed to.

Useful, but really straining the limits of how much code I want to maintain in a dynamically typed language like Python, at least, with my (low) level of Python expertise. Also, it was, frankly, really slow. So over a period of a couple of weeks, I rewrote the whole thing in Dart.

I work at Google on the Dart language team as my day job, so picking Dart was not an unbiased choice. But it’s my build system for my book, and I know Dart and many of its core libraries and packages like the back of my hand. Hell, I personally wrote the initial version of the canonical markdown package.

I found a pretty nice package for mustache templates, so I converted the book’s old Liquid templates to that. I didn’t find a good syntax highlighter. But it’s not like I looked very hard either. It seemed like just the kind of fun thing to implement from scratch, so I whipped one up loosely based on Pygments.

The end result is a new build system that generates exactly the HTML and syntax-highlighted code that I want. Also, it is literally ten times faster than the old Python one. As you’ll see, it turned out to be handy that I had better control over the Markdown processing, but at the time I was basically just doing this for fun and to procrastinate the real work.

I planned to implement some clever incremental rebuild logic in the dev server, but it builds so fast that I just rebuild everything on every refresh.

Once I had the new build system outputting nice clean HTML and I’d deleted the old Python stuff, it was time to get started.

Designing the book#designing-the-book

Doing a large graphic design project like a book works a lot like web dev or game programming where it sort of has two levels. First you set up your “framework” or “engine”. On the web, this is your web framework and all of your CSS and HTML templates. In games, it’s your game engine. Then you pour content into that structure. With the right framework, adding content is easy.

For graphic design using a program like Adobe InDesign, the way it works is you set up styles and masters. A master defines the margins and grid lines for a page. It’s the fences that corral all the text to keep those feral letters from running around eating all the whitespace. Styles are like CSS: they let you take a semantic kind of text or object and associate specific fonts, styles and colors for it.

In theory, you get the masters and styles right and then typesetting is pretty easy and mechanical. Now, I did not make my life easy when it comes to book design. Book design is literally a two-dimensional spatial exercise and I made my job harder both horizontally and vertically. If you’ve read any of it on the web, you know my book has:

Prose, of course. So many words.
A lot of asides that need to be right next to certain pieces of text, code or illustrations that they refer to. Some of these can get pretty long.
Plenty of code. Also, each code snippet has a little location blurb next to it telling you where the code goes in the resulting program.

How wide?#how-wide

Horizontally, the main text column needs to be wide enough to fit the longest line of code. I can use shorter lines and wrap the snippets more, but that makes them harder to read. It also makes them vertically taller, which causes its own problem.

Then I need room next to that for the asides, since they often remark directly on specific sentences. I can make those narrower too, of course. But then they wrap and get taller. Some asides are fairly long and if they get too tall, they start colliding with each other or overlapping location snippets.

Oh, and since I ended up writing a 200k+ word book, it’s going to have a high page count. That means a thick book. Thick books need wider inner margins so the text doesn’t disappear into the spine.

All of this points towards a pretty wide page. Most CS textbooks—at least the ones on my bookshelf—are 7.5 inches wide. I tried hard to come up with a design that fit the code, asides, and healthy margins in that width while still giving a text size that didn’t require a magnifying glass. Eventually, I conceded defeat.

Once I tried designing a set of metrics for an 8 inch wide page, everything fell into place. I could have enough breathing room around the text to make it enjoyable to read, a decent length for the code snippets, and plenty of room for the asides. (Using a narrower font for the asides helped too).

Here are the final horizontal metrics.

How tall?#how-tall

That left the other dimension. If I were going with a real publisher doing a full offset print run, I could pick whatever page size I wanted. But since I’m self publishing, that would mean paying up front for thousands of copies to be printed and, I don’t know, storing all the boxes in my garage or something.

Print on demand worked great for my last book, and I planned to do the same thing for this one. That meant sticking to the limited set of page sizes that KDP and IngramSpark support. The only reasonable one that is 8 inches wide is 8”×10”, so that’s what I picked. The end result is a book that feels big, but hopefully not awkwardly huge. I’m sorry. I promise to write a smaller book if I ever write another.

The metrics for a complete chapter spread. Vertically, text is aligned to a classic 12pt baseline grid.

The whole time I was picking these margins and metrics, I was also selecting fonts and building styles. You can’t do graphic design in a vacuum, so I picked a test chapter and just typeset it and tweaked the fonts over and over again. Eventually I got a set of fonts and styles I liked, some masters that seemed workable, and I was ready to go. The framework was done and now it was time to open the content hose.

XML, like its 1999#xml-like-its-1999

Of course, problem #1 is that there is no content hose. I have to build one. InDesign doesn’t know what the hell Markdown or my weird ass build system is. I sure as hell didn’t want to, like, copy every chapter from my browser, paste it into InDesign and then manually apply all the styles. I have a high pain tolerance, but I’m not a masochist.

What InDesign can do is import XML. Even better, you can set it up to automatically apply certain paragraph or character styles to certain tag names. InDesign’s XML support, alas, has not matured much since the time I described it as being implemented by a narcoleptic intern. For example, in HTML you can italicize a word in a header by taking an italics tag and nesting it inside a header tag. InDesign cannot comprehend such advanced data modeling. It needs a flat series of unnested tags and if you need italics in your header, you damn well better have a unique <italics-header> tag for it.

But now I had an ace up my sleeve. Since I had microscopic control over my build system and its Markdown processing, I could write my own custom XML exporter that generated exactly the tags that would make InDesign not cry and avoid as many InDesign XML import bugs as possible.

JavaScript, in my InDesign?#javascript-in-my-indesign

Even so, XML import only gets you so far. Specifically, it gets you a “story” in InDesign terms: a single continuous narrative of text that fills the main text box and spans multiple pages:

What the initial XML import looks like. The pink text is where an illustration goes and the "@" tells me where to anchor the aside.

The main story is where the prose and code snippets go, but the asides and the location markers need to be pulled out float off to the side. With my last book, I yanked those out manually. I literally cut each aside from the main text column and pasted it into a new text box. It took forever, and that book was less than half as long as this one with way fewer code snippets.

There are 1,133 code snippets in Crafting Interpreters. If I had to manually cut and paste the location markers for each of those, I would lose my mind. Worse, I would make a lot of mistakes, which would just create more work for myself.

I’m an engineer so I am morally opposed to error-prone manual effort that can be automated. Did you know that InDesign can be scripted using JavaScript? Well, you do now, which apparently places you in a tiny minority because there are, like, no docs for it out there. You can find a couple of auto-generated references, a few sad cries for help from graphic designers clearly out of their element with no responses, and that’s it.

JavaScripting InDesign is a special kind of pain. There is no debugger. There are no stack traces. There aren’t even debug prints. There is literally just alert(), and you can only call it once. And it halts your script. Fortunately, I actually learned JavaScript back when that’s all browsers gave you, so I can hack it.

Just like developing for IE6.

I managed to cobble together a horrific script that would find all of the asides and location markers, pull them out of the main text flow, and leave them off to the side:

Asides and location markers are pulled out of the main flow and into their own text boxes.

What I wasn’t able to get the script to do was position the boxes correctly. But InDesign has a thing called “anchors” where you can lock the position of an element relative to another. A couple of carefully crafted Object Styles would even set the horizontal metrics correctly and align the text to the right baseline. All I had to do was anchor each text box and it worked perfectly!

Wait, did I say “perfectly”? I meant it worked right about half the time and the other half the time InDesign would inexplicably turn off the fucking borders of nearby code snippets.

What it looks like when InDesign hates you.

I lost hours of my life to this stupid bug. Eventually, I realized that some completely random subset of location tags would have to be manually positioned because anchoring invariably broke some borders.

Editing, again#editing-again

All of the above took me a good month or so and then it was time to stop fiddling around and get to work on the actual content. The first thing I did was another editing pass of the entire book, front to back. I had already done three drafts of each chapter as I was writing them, but I wanted to do one more now that it was done so that I could get a better feel for continuity. It turns out I repeated the same dumb jokes a lot. I fixed (most of) those.

This took five months. There’s nothing fun to say about this, it was just a chore.

Copy editing#copy-editing

Next I hired an actual professional copy editor, Kari Somerton, to go through and do the same thing. She was great. Most of the editing world uses Microsoft Word and “Track Changes” to handle the editing process. Like most software engineers, I live and breathe plaintext and Git. That way I can see diffs of the changes, and go back through history.

I didn’t want to abandon my workflow so I asked Kari to ramp up on Git and my weird completely bespoke build system. She handled it with aplomb and churned through the book in no time. She found hundreds and hundreds of mistakes. This despite me doing four drafts and readers filing hundreds of issues already. Professional copy editors are worth every penny.

Typesetting the whole thing#typesetting-the-whole-thing

Once the words were as good as they were gonna get, it was time to get them onto pages. The process went like this:

Create a new InDesign file for the next chapter.
Export it to XML.
Import the XML into InDesign.
Run my little JavaScript script to pull out the asides and location markers.
Go through and anchor the side bar stuff.
Fix up whitespace at the end of pages.

The first five steps are a piece of cake. I’d wake up in the morning, brew a cup of coffee, shamble upstairs to the iMac, and get started. I could grind through those steps while half awake and get a chapter done in half an hour or so. It was peaceful. Almost meditative.

And then step six. You see, that right there is the hard thing about typesetting a book. And it’s the really hard thing about typesetting this book. Because it turns out there are a bunch of constraints on how content can be fit vertically in a page. Obviously, we can’t slice an illustration in half and put the top half on one page and the bottom half on the next. The asides also really need to fit on one page too, or it gets confusing to keep track of what they refer to.

Whenever possible, it’s nice for the code snippets to not get split across pages too. Some of those can be over a dozen lines long. (This is another reason why wider horizontal metrics helped. Because if I had made the code snippets narrower, they would end up taller, which would make them harder to fit on the page.) And you don’t want a header alone at the end of a page with no content after it. And it’s good to avoid widows and orphans…

Take all of those rules and restrictions, and mix in the completely fixed height of a page and you got yourself a real constraint solving problem. Or, in my case, 640 of them, all interwoven with each other. Because, you see, InDesign is happy to solve all of this for you by just pushing content to later pages. Code snippet too long? Move it to the next page. No room for prose under the header? Move it all to the next page. What that gives you is a ton of dead white space at the bottom of pages. It looks terrible and wastes space, like this:

I thoughtfully left room on the page for your own doodles.

Speaking of illustrations…#speaking-of-illustrations

At one level, the illustrations were easy. I specifically chose black and white pen and ink because it’s print friendly. When I first scanned the images as I wrote each chapter, I brought them in at glorious 1200 DPI. Here’s a crop:

A little level adjustment in Photoshop produces:

So detailed! Exporting these to high resolution bitmaps that print well was a snap, and they look great. (Well, as great as my handwriting looks, I guess.)

Incorporating the illustrations into the page layout was another story. Halfway through typesetting, a lightbulb went off and I finally realized why most books say, “Refer to Figure 123 to see blah blah blah…” That gives the typesetter freedom to put Figure 123 wherever the hell it fits on any nearby page.

In my dumb book, because I am an idiot, the prose just refers directly to the illustration. The illustration needs to be right there or the text doesn’t make sense. I didn’t think about that when each chapter was an infinitely scrolling web page, and by the time I realized, it was too late.

With the hundreds of illustrations and thousand-plus code snippets, I had given myself thirty giant interrelated bin-packing exercises. The hard part of typesetting was figuring out how to adjust things to minimize that dead space. Sometimes I’d split a code snippet in two. Maybe add a little extra padding around one image to spread stuff across the page a little. Or crowd another one so that it just fits on the page. Sometimes I’d tweak an illustration to make it shorter to fit on a page or taller to eat up some whitespace.

This was the real challenge of typesetting the book and why it took me two months to get through all the chapters.

Front matter and back matter#front-matter-and-back-matter

Did you know that there are professional indexers? People whose job it is to write indexes for books? They even write books about how to write indexes. (One would presume that these books have truly superb indexes.)

I did not hire one of those eminently skilled professionals. Instead, I spent two weeks going through every damn chapter again doing my best to pretend that I know what I’m doing. InDesign’s support for indexes is actually pretty nice. You can basically just select some text and say, “Make an index entry for this.” Then it collects all of those and generates an index for the whole book. But actually adding all of those entries is a mind-numbing chore.

The index is the main piece of back matter—the stuff at the end of a book after its main content. There is also front matter. You’ll never guess where that goes. I put together a title page, copyright page, dedication, and acknowledgements. Then I let InDesign generate a table of contents for me.

This was a magical moment. At this point, I had a complete book:

Here's what the inside looks like.

Or, at least, I had the inside of one.

Cover design#cover-design

A lot of authors obsess over their cover and envision it the whole time they are writing. Despite what the aphorism says, people do judge books by their cover and a good one makes a big difference. At least, that’s the case over in fiction land. In computer science, judging by the other books I have laying around, the artistic merit of the cover appears to be somewhat less critical. I guess when the prof says you have to buy the book to pass the class, a clip art cover is sufficiently compelling.

Since I am not a professor who can garner sales by fiat, I spent a lot of time on the cover design. I take photos, so I thought it could look nice to put something detailed on the cover to liven it up. I went through my thousands of photos trying to find something that fit. And, while I have some pretty pictures, none of them felt like they worked as covers. They felt arbitrary.

Eventually I realized that the visual language of the book is those pen and ink illustrations. So I drew a bigger more detailed version of the mountain illustration I use as a metaphor for the compilation process. I also hand-lettered a new title:

It’s a real typeface (Acumin Pro Extra Condensed) but I hand-traced a printout to give it some imperfect charm. I picked a color palette to try to give it a sort of a mimeographed 1950s scouting manual vibe.

Proofreading the proof#proofreading-the-proof

Now I really had a book. I uploaded the PDF exports to KDP and ordered a proof copy. A week later, a surprisingly heavy box arrived. This was the first moment I really understood just how big this book I wrote is. Up until this point, it was just data files. But seeing it fill up an Amazon box clarified the scale of the project in a way that the time I spent never quite did.

So I had a book, but it still wasn’t done. Because the typesetting process involved a lot of manual labor. To err is human, so now I had to proofread—to literally go through the proof and read it looking for mistakes. I marked them all with sticky notes:

I put an "x" on each sticky note to mark when I had applied the fix to the InDesign files.

Here’s where it got stressful. If you’re a programmer, then source control and diff is deeply ingrained in your workflow. Whenever I make a change, I take for granted that I can then see a diff in the commit to verify that I changed only what I intended and nothing else.

I did put the InDesign files in a Git repo, but they are giant opaque binary files. Also, InDesign has a habit of changing them even when it doesn’t seem like I’ve actually made any real changes. There’s nothing quite like syncing all the styles across the chapters, seeing every single file marked changed and wondering, “Did I just accidentally move every bullet list item 3 points to the left?” It felt like flying blind at exactly the stage in the process where I really wanted to see exactly what was happening.

Did I engineer my way out of this problem? You’re damn right I did. I wrote a Dart script that would take a PDF of the book, extract every page, and then generate a single huge PNG file with every page tiled across it. It looks like this:

If you zoom in, each page is about this big:

If you're on a retina display, this is a 100% zoom.

Every time I changed the InDesign files and committed them, I exported a PDF for that commit and generated a tile image. Then I wrote a little Photoshop action that would take two of those and draw a big red border around any pixels that differed. Here is what all of the proofreading changes look like:

And zoomed in:

There isn’t enough detail in the tile image to tell exactly what changed, but the red tells me which pages need a visual inspection. I know I should be proud of, like, writing an entire textbook on programming languages. But, honestly, I think I’m most proud of this dumb little script. It was such a relief to be able to programmatically verify that, yes, this PDF looks exactly like the last one except for the one expected change.

Ebooks are also books#ebooks-are-also-books

Once I incorporated all the proofreading fixes, the print edition was really for real totally done. But people also like reading ebooks and Kindle, so I needed to make those too.

Again, writing my own build system helped. I tweaked it to be able to export the antiquated XHTML that EPUB requires as well as all the weird metadata and manifest stuff that goes into one. A few command-line invocations later and I had Kindle and EPUB e-books. I tested in a bunch of readers and tweaked the CCS to try to find a compromise between all of their renderers.

Updating the site and launching#updating-the-site-and-launching

At this point—and now we’re talking this past weekend as I write this sentence—I had a folder with all of the final files for every edition of the book. All that remained was to update the front page of the book’s website to point to where people can get them. I took some photos and put on my web designer hat for a while. I tried to make it tolerably responsive.

Then I wrote this blog post. It’s weird to put that in the past tense since I just wrote it. But I guess it’s in the past now.

Tomorrow, I’m going to upload the files to the various sites and stores. I’ll update the site to have links to all of those as they go live. Once the stores have processed the uploads and everything is available, I’ll publish this post and update the site. I’ll write a note for the mailing list and feel nervous emailing that many people. If you’re reading this, I already did all that and the book is really, really done.

What next?#what-next

Ever since I finished the last chapter, people have been asking me what I’m going to do next. I have people I consider close friends now who have never known me when I wasn’t writing this book.

Many ask what I’m going to write next, or suggest a topic. I interpret this as a compliment—they like my writing and want more. But it also feels like asking a mom in labor whether she plans to have more kids. I’ve had my legs in the stirrups for six years, so I’m gonna just relax a bit after all the pushing is over before I even think about another book baby.

What I really plan to do is… not plan to do anything. This writing project has been an enormous exercise in self-applied delayed gratification. On top of that, the pandemic brought its own basket of deferred dreams. I’ve gotten so skilled at denying myself and postponing, so good at tuning out what I feel like doing, that it’s hard to even hear those feelings any more. I don’t remember what my own joy sounds like.

So I’m gonna check out for a while and go hunting for my pleasure centers. Maybe make some more music. Maybe go fishing. Probably spend more time with friends and family (safely). I might get back to working on my roguelike. Or maybe I won’t do any of those. Maybe I’ll just go out in the backyard and bask mindlessly in the sun like a lizard. The important part is I won’t decide until I feel like it.

I’m sure eventually I’ll get the itch to work on something more sizeable again. (But, God-willing, I will never spend six years on a single project again in my life.) Until then, I hope you enjoy the book. Despite all the work, I had a lot of fun making it, and it was always a joy to hear from readers who liked it. It’s the only thing that kept me going this long and enabled me to get it done.

Crafting “Crafting Interpreters”

Sun, 05 Apr 2020 07:00:00 GMT

It took three years and 200,000 words more than I expected, but my second book, Crafting Interpreters, is complete. I finished the third draft of the last chapter today, marking the last of around 1,400 days of continuous writing.

This book was much harder than my first book, along every axis. It’s larger, more technically complex, much more deeply intertwined, and it had the misfortune of aligning with a really difficult period in my life. Today feels less like coasting past the finish line at the Tour de France, arms raised in triumph, and more like dragging myself onto the beach, clutching sand in relief after a storm-thrashed ordeal at sea.

Before I get into all that, I have a minor confession to make. When I finished my first book, I wrote a long post about how I cobbled together enough willpower to reach the end of the last page. Everything in there is true, but there is one fact I superstitiously omitted.

Halfway through writing Game Programming Patterns, I discovered a new passion: programming languages. It had been a long time since a topic ignited my brain to the same degree, and I was on fire. I spent basically every free hour (and many not-so-free hours—sorry, family) designing and hacking on programming languages. I read every book I could get my hands on, went to conferences, blogged, I even dreamed about programming languages. This infatuation was the main reason I stopped working on my first book for two years.

I have a personality quirk where when I’m excited about something I just have to teach it to other people. Hermione Granger, arm waving feverishly to get the teacher’s attention, is my spirit animal. It was inevitable that I would write something about interpreters. But I couldn’t just drop one half-finished book to start another. I have gigs of unfinished projects laying around, but—maybe because the completed chapters were already online—I couldn’t bear to abandon Game Programming Patterns.

So I made a promise to myself. If I finished that book, then I would let myself write a second book on interpreters. In part because of that promise, I did manage to complete the chapters, and then the print and e-book editions. What I thought was merely a hobby and personal goal turned out to be a life-changing experience. My little self-published vanity project has hundreds of five-star reviews, and has been translated to Korean, Japanese, Chinese, German, and Polish. The book did so much better than I expected that I’m still not sure how to process it, beyond feeling immense gratitude to everyone who read it, bought a copy, or cheered me on.

The seed of a book#the-seed-of-a-book

Once I finished the print edition of Game Programming Patterns, I took some time off. But it didn’t take too long for that itch to write about interpreters to come back. I knew exactly what I was getting into with writing a book now, how hard the grind can be. At first, I just noodled around. I wasn’t committed to doing anything. It was more a sort of recreational intellectual exercise. If I were to do a book, what would it look like? You know, hypothetically speaking.

The very first note I wrote to myself said:

high-level goal: a *small* book that builds a complete, efficient
interpreter. instead of a wide text about programming language*s*,
it is a single path through the language space. aim for 60k words.

My first book was about 90,000 words, and I didn’t want to hike a trail that long again. I also had a meta-goal to make programming languages more approachable, and I figured a short text would help. I had this vision of something you could literally hold in your hand or have open next to your laptop while you followed along.

To make a small book, I needed a small language and a small implementation. One of my other side projects was a scripting language named Wren. Wren is written in C, with a simple single-pass bytecode compiler inspired by Lua. Building Wren taught me how much functionality you can pack into a few thousand lines of clean C code.

For this hypothetical book, I figured a bytecode VM in C like that would be a great fit. It would also give me the chance to cover a bunch of really fun topics like stack-based VMs, object representation, and garbage collection. But Wren wasn’t the right language. I like Wren (obviously), but it has some design quirks that I think make it a better language for users but maybe not for teaching. For the book, I wanted a dynamically-typed scripting language in the vein of languages like JavaScript, Python, and Lua.

I started tinkering on a new toy language, tentatively named “Vox”. The goal was to keep things as simple as possible without taking any shortcuts around the hard problems in implementing a language. I wanted a rich expression and statement syntax to cover parsing. First-class functions and closures because they are powerful and challenging to implement efficiently. Classes and methods because that paradigm is so prevalent but omitted by many compiler books.

At some point, I realized that dropping readers straight into C was too unfriendly of an introduction. It’s hard to teach high-level concepts like parsing and name resolution while also tracking pointers and managing memory. OK, so we’ll build two interpreters. First, a simple one in a high-level language to focus on concepts. Then a second bytecode VM in C to focus on performance and low-level implementation techniques.

Somehow, I didn’t notice that maybe this “handbook” wasn’t going to be as pocket-sized as I hoped.

My first choice for the high-level implementation language was JavaScript. I implemented most of a Vox interpreter in JS, but never really liked it. I wanted to write the interpreter in an object-oriented style because there are techniques like the Visitor pattern for doing language stuff in OOP that aren’t covered well elsewhere. Doing OOP in JS means deciding whether to use classes or a prototypal style. The former is cleaner but infuriates some segment of readers. The latter is verbose and confusing to those not already steeped in prototypes.

Also, I missed static types. People reading code in a book don’t get the luxury of seeing the code in a debugger where they can see what values are in various variables. Static type annotations in the code help.

So I switched to Java. I don’t love Java but it seemed like the least biased choice for a statically typed object-oriented language. I found you can tame a lot of its infamous verbosity by simply not programming in 1990s enterprise Java style. Maybe it’s not idiomatic to have public fields, but it’s a hell of a lot shorter.

In parallel, I started building the bytecode VM in C, porting over bits of Wren’s implementation and stripping out the Wren-specific stuff. I spent the spring and summer of 2016 circling between these three pieces—the design of Vox itself, the Java interpreter, and the C bytecode VM. This was a delightful, satisfying period of time. The three parts played off each other in challenging ways. Sometimes I would change the language to make one interpreter simpler, but find doing so made the other interpreter more complex. Other times I’d hit on some trick that made everything get smaller and cleaner.

Getting back on the horse#getting-back-on-the-horse

I remember the exact moment I committed to writing the book. I was stuck on a tricky language design problem: constructor syntax. I knew I wanted classes, which meant some way to construct instances. Adding a new keyword felt too special-purpose for my minimal language. I like Smalltalk and Ruby’s approach of making new be a method on the class object itself, but that requires metaclasses and a lot of other machinery.

I was struggling to find a way to add instantiation without making the language much bigger. Then I remembered JavaScript’s thing where you can simply invoke a “class” as if it were a function to create new instances. That has all sorts of weird baggage in JavaScript because everything does in JS, but the concept and syntax were perfect. I already had first-class classes. And I already had closures which meant a function call syntax that could be applied to arbitrary expressions. So “constructors” just became what you got when you invoked a class.

I felt like Vox had gelled, like it was a language now. And my two implementations were coming along well too. I was surprised by how few hacks or ugly corners I ran into. The codebases kind of fell together and the more I tweaked them, the nicer they got. It felt more like I had discovered them than that I had created them. It would be a shame to not write the book and put them out there into the world. They wanted me to.

I committed to writing the book, and I restarted my rule of writing every single day.

I had a few thousand lines of pretty Java and C code, but how do I turn that into a book that can be read in linear order? Compact codebases tend to be highly intertwined with many cyclic dependencies. I didn’t want readers to have to slog through ten chapters before they could even run main().

This was the real technical challenge of writing the book—how do I take two implementations of the same language, and break them into incremental pieces that I can build up a chapter at a time?

I made this problem harder for myself because of the meta-goal I had. One reason I didn’t get into languages until later in my career was because I was intimidated by the reputation compilers have as being only for hardcore computer science wizard types. I’m a college dropout, so I felt I wasn’t smart enough, or at least wasn’t educated enough to hack it. Eventually I discovered that those barriers existed only in my mind and that anyone can learn it.

My main overarching goal of the book is to pass on that feeling, to get readers to understand there’s no magic in there and nothing keeping them out. To nail that conceit, I wanted to include every single line of code used by the interpreters in the book. No parser generators, nothing left as an exercise for the reader. If you type in all of the code in the book, you get two complete, working interpreters. No tricks.

So not only did I need to break these two interpreters into chapters, I needed to do it without any cheating. I wanted a hard guarantee that at the end of each chapter, you had a program that you could type in, compile, run, and do something with. I knew I wouldn’t be able to verify this manually, so it was time to create some tools.

A bespoke build system#a-bespoke-build-system

I wrote my first book in Markdown. I slapped together a tiny Python script that converts the Markdown to HTML and transcludes the code snippets which are stored in separate C++ files. When I started my second book, I took that script and started growing it. It evolved throughout writing the book, but in the end, here is how it works.

All of the code for the interpreters are stored in separate source files. I have a Java project that contains the complete Java interpreter that you get by the end of that part of the book. Likewise, there’s a C project for the bytecode VM. I can edit and build those in an IDE, run tests, debug them, etc. They’re real programs.

Meanwhile, the text of the book is authored in Markdown, one file per chapter, just like my first book. To include a snippet of code in the book, I put a tag in the Markdown like this:

Which can be any of:

^code is-alpha

Once we've found an identifier, we scan the rest of it using:

Here, the ^code line says “look up the snippet named ‘is-alpha’ and insert it here.” When the build script generates the HTML for this chapter, it goes off and hunts through the code for that snippet. Over in the code, special comments delimit snippets. The one included here looks like this:

//> Scanning on Demand is-alpha
static bool isAlpha(char c) {
  return (c >= 'a' && c <= 'z') ||
         (c >= 'A' && c <= 'Z') ||
          c == '_';
}
//< Scanning on Demand is-alpha

The //> line begins the snippet and says what chapter the snippet appears in and the name of the snippet. The //< line ends the snippet. Pretty straightforward.

This let me build the book, but didn’t ensure that the thing I built actually worked. So I wrote a separate script that instead of building the book, builds programs. For each chapter, it collects all of the snippets that appear in that chapter and the previous ones and writes them out to separate source files. In other words, it produces a separate interpreter, one for each chapter, containing only the code that readers have seen so far.

I put together a Makefile to build those per-chapter versions of each interpreter to make sure they compiled. Of course, compiling successfully doesn’t mean they do anything useful. Writing a single correct interpreter is hard. Writing thirty of them—there are thirty chapters in the book—is much harder.

I had already harvested a little test runner from Wren and ported most of Wren’s tests over to be Lox tests. (I changed the name of the language in the book since there was already a language out there named “Vox.”). I took that test runner and extended it to be able to run the tests on each chapter’s version of the interpreters. Of course, the tests don’t all pass—the interpreters aren’t complete! So I added metadata to track which tests I expected to pass by which point in the book. With this in place, I could automatically verify that the code that I was showing readers did exactly what I expected.

More complex snippets#more-complex-snippets

The snippet markers look pretty straightforward, and in many cases they are. But reality tends to get messier and I didn’t allow myself to sweep any of that mess under nearby rugs. Some changes don’t just add code to the interpreter. I try to minimize it, but often you need to replace some existing code. A few lines of code may appear in chapter 5 and then later get superseded in chapter 9 by something more powerful.

Obviously, I can’t jam both of those snippets into the same source file and expect it to compile. Remember, the source files that I hand author are themselves valid Java and C programs that I can build and run. If a function contained several versions of its body mixed together, odds are slim that the compiler will like what it sees.

So, for any piece of code that later gets replaced—in other words code that is not part of the very final version of each interpreter—there is a different snippet syntax:

static void concatenate() {
/* Strings concatenate < Garbage Collection concatenate-peek
  ObjString* b = AS_STRING(pop());
  ObjString* a = AS_STRING(pop());
*/
//> Garbage Collection concatenate-peek
  ObjString* b = AS_STRING(peek(0));
  ObjString* a = AS_STRING(peek(1));
//< Garbage Collection concatenate-peek

  int length = a->length + b->length;
  char* chars = ALLOCATE(char, length + 1);
  memcpy(chars, a->chars, a->length);
  memcpy(chars + a->length, b->chars, b->length);
  chars[length] = '\0';
}

This block comment contains a snippet of code. The header indicates that this snippet is named “concatenate” and first appears in the “Strings” chapter. Then, later, it gets removed when the “concatenate-peek” snippet in the “Garbage Collection” chapter appears. In other words, that latter snippet replaces the previous two lines.

By storing the code for this snippet inside a block comment, I ensure that the code as it is in the raw source file is still valid. In some places where the interpreter gets revised multiple times, the code can get pretty complex. Here is the main() function of the bytecode VM:

int main(int argc, const char* argv[]) {
//> A Virtual Machine main-init-vm
  initVM();

//< A Virtual Machine main-init-vm
/* Chunks of Bytecode main-chunk < Scanning on Demand args
  Chunk chunk;
  initChunk(&chunk);
*/
/* Chunks of Bytecode main-constant < Scanning on Demand args

  int constant = addConstant(&chunk, 1.2);
*/
/* Chunks of Bytecode main-constant < Chunks of Bytecode main-chunk-line
  writeChunk(&chunk, OP_CONSTANT);
  writeChunk(&chunk, constant);

*/
/* Chunks of Bytecode main-chunk-line < Scanning on Demand args
  writeChunk(&chunk, OP_CONSTANT, 123);
  writeChunk(&chunk, constant, 123);
*/
/* A Virtual Machine main-chunk < Scanning on Demand args

  constant = addConstant(&chunk, 3.4);
  writeChunk(&chunk, OP_CONSTANT, 123);
  writeChunk(&chunk, constant, 123);

  writeChunk(&chunk, OP_ADD, 123);

  constant = addConstant(&chunk, 5.6);
  writeChunk(&chunk, OP_CONSTANT, 123);
  writeChunk(&chunk, constant, 123);

  writeChunk(&chunk, OP_DIVIDE, 123);
*/
/* A Virtual Machine main-negate < Scanning on Demand args
  writeChunk(&chunk, OP_NEGATE, 123);
*/
/* Chunks of Bytecode main-chunk < Chunks of Bytecode main-chunk-line
  writeChunk(&chunk, OP_RETURN);
*/
/* Chunks of Bytecode main-chunk-line < Scanning on Demand args

  writeChunk(&chunk, OP_RETURN, 123);
*/
/* Chunks of Bytecode main-disassemble-chunk < Scanning on Demand args

  disassembleChunk(&chunk, "test chunk");
*/
/* A Virtual Machine main-interpret < Scanning on Demand args
  interpret(&chunk);
*/
//> Scanning on Demand args
  if (argc == 1) {
    repl();
  } else if (argc == 2) {
    runFile(argv[1]);
  } else {
    fprintf(stderr, "Usage: clox [path]\n");
    exit(64);
  }
  
  freeVM();
//< Scanning on Demand args
/* A Virtual Machine main-free-vm < Scanning on Demand args
  freeVM();
*/
/* Chunks of Bytecode main-chunk < Scanning on Demand args
  freeChunk(&chunk);
*/
  return 0;
}

Maintaining this is not super fun. But, thankfully, I have a build and test system to tell me when I break something.

Slicing up the interpreters#slicing-up-the-interpreters

So I had a tool that could let me split the interpreters across the chapters. If it was possible to break these interpreters into chapters at all, it would let me do so. Now I just had to figure out where to carve the seams. This was the most technically challenging part of the book writing process. I wasn’t sure if it was going to work at all.

I spent several weeks sketching out potential lists of chapters, sprinkling snippet markers throughout the code, and seeing if the result built. I’d get a compile error because a snippet in an early chapter tried to call a function in some later chapter and I would have to go back and reorganize things. I hand-drew dependency graphs between language features and tried to untangle them.

Here’s an example of how this process unfolded:

To teach functions I want show that recursion works.
But to have recursive functions I need control flow. Otherwise, every recursive function recurses infinitely without a base case. So control flow has to come before functions.
For control flow, I need side effects so that I can show that a certain code path is not taken. The obvious way to do side effects is to have a print() function that displays output.
But I don’t have functions yet. That’s a cycle. Crap.

Sometimes I had to change the language itself to break cycles. The above example is why Lox has a built in print statement instead of a print function. Because that way we can introduce the print statement before control flow, which is in turn before functions.

I had to break a couple of cycles like that but, eventually, to my surprise, I got it all sorted out. I had a complete list of chapters for both interpreters. Every line of code was sorted into a snippet that belonged to one of those chapters. I could build and run each chapter’s code. Best of all, each chapter had a reasonably coherent concept and a roughly similar amount of code.

Before, I felt like I had a language and code that wanted to get out there into the world. Now I felt like I had a book. Or, at least, I had all of the code for a book.

A chapter at a time#a-chapter-at-a-time

I wrote my first book one chapter at a time. I drafted, edited, illustrated each chapter and put it online before moving to the next one. Serial publishing for the digital age. I really loved that process. It helped build an audience for the book and gave me incremental feedback which made the book better and kept me going. I don’t think I could write a whole book in the dark.

I intended to publish this book the same way, but the deeply interconnected nature of the chapters made that much harder. I didn’t want to discover a problem with the code in chapter 28 that forced me to tweak things in an earlier chapter that readers had already read. I didn’t want to paint myself into a corner or invalidate any previously-published material.

So the entire time I was designing the language, coding the interpreters, and splitting the codebases into chapters, I had not done any actual writing. I didn’t want to put down any prose until I knew the code was solid. So I spent the summer of 2016 just hacking on code. It was, honestly, a blast. The programming part is definitely the fun part, and it was a joy to tinker on the code and figure out how to break it into chapters. Sort of like making a jigsaw puzzle and solving it at the same time.

After a few months, it was all there. Every single line of code for the entire book. A complete list of chapters. And I hadn’t written a single word of prose. In theory, “all” that remained was writing some text to explain the code I had already written along with some pictures. But, for me at least, English is a much more taxing language to write than C or Java. I had all of the difficult work ahead of me, and all of the fun was done.

Illustrating by hand#illustrating-by-hand

Well, not all of the fun. I did still have the illustrations to do. With my last book, I hand-drew little sketchy diagrams to show various bits of architecture. I wanted even more illustrations for this book to make the concepts less abstract, less opaque. Unlike a videogame, you can’t see a garbage collector doing its thing. Visual metaphors really help.

I liked the hand-drawn look. It furthered my meta-goal of making the material more approachable, more human. But I wanted to up the quality. I wanted them to be more intricate and contain more information. I wanted the drawings to be more detailed. Less like margin doodles and more like, well, illustrations. Maybe even some lowercase letters.

The ultimate goal for me is a print book, so I stuck with black and white ink. I wanted a tighter, more “spidery” style, so I got some technical pens. People often ask me what programs I used for the illustrations, assuming I did them all digitally. Here are the main tools I used:

I went with Pigma Microns in 01 and 005. If I were doing it again, I think I'd do Faber-Castell Pitt pens.

There are two kinds of illustrations in the books: diagram-like ones that show meaningful information, and drawings that are for metaphors or just to be silly jokes. The process is different for each.

I draw each diagram in pencil on graph paper. That lets me erase and move things around until I get it where I like:

All of the vertical and horizontal lines in the illustrations generally fall on the graph paper rules or halfway between them.

Then I tape a piece of tracing paper on top and draw over it in ink:

I make mistakes sometimes, usually when lettering like "upvaluels" here. I fix that in Photoshop after scanning.

I hand letter everything. It takes a long time. I used to do graphic design, and I have this weird tic where any time I see something that looks handwritten, I look for multiple instances of the same letter to see if they are different or if the design just used a handwriting font. It’s almost always a handwriting font and I die a little inside to see the illusion evaporate.

Well, this is my damned book and no reader will ever feel that disappointment. Every single fucking letter in every one of the illustrations was hand lettered and is unique.

Here is the hand-lettered logotype for the book. Each "R" is different!

Also, if that’s not obsessive enough, I spent time changing my own handwriting to better match the text font of the book. I taught myself to write double-story “a” and “g” letters and practiced by filling pages of paper with the same letter over and over.

Look at the loop under the "g" in "filling" and the finial on the "a" in "apples".

I also wanted to make sure that the illustrations and text matched each other across the book. To give the text a consistent size, I printed a little height guide:

The dotted line indicates the x-height. I picked a ratio for that to match the fonts I use for text and code.

I slid this paper under the tracing paper and lettered on top of those lines to keep the metrics the same across the book.

To keep the diagram size and line thickness consistent, each illustration has a pair of registration marks a fixed distance apart:

The little marks that the pencils are pointing at.

I scan each illustration into Photoshop for clean up and processing. I use those marks when cropping to ensure that the image maintains the right size relative to other images.

I recorded a video of the whole process if you want to see it in action:

Even in timelapse, it takes a long time.

Writing this all out makes me sound like a crazy person. What the hell am I doing with my life? Or, more importantly, what could I have been doing instead of doing all that?

Too late now, I guess. The picture-like drawings have a different workflow since they don’t have a lot of straight lines or align to a grid.

How will readers understand what a stack is without this helpful illustration?

I draw those on regular sketch paper using a non-photo blue pencil. Then I ink on top of that. I scan the paper in RGB and use the blue channel, which mostly makes the blue pencil marks disappear.

The sketch paper bleeds the ink more than I like but I didn't want to change paper partway through the book, so I stuck with it.

It’s a lot of work for each image, and this doesn’t include all of the work after scanning it. And I wanted a lot of them. By the end, I had this stack of paper:

Such a small image for so much work.

I went through two full pads of tracing paper, two pads of graph paper, a sketch pad, and several pens. I drew 181 illustrations.

Writing is suffering#writing-is-suffering

I had the code, and I had a process for illustrations. The remaining work was just writing all the words and drawing all the pictures. So that’s what I did. I started at chapter one and started writing. For each chapter, I wrote an outline and then a first draft. I did an editing pass over that to fix all the major problems. Then a second pass where I read the whole chapter out loud to fix cadence and other stuff.

This is the same process I used for the first book. I stumbled onto something that worked, so I wasn’t about to mess it up. I posted each chapter online, and then spent a day fixing bugs that readers noticed. Then I moved on to the next chapter.

I wrote. And wrote. And wrote. Every single day. Every now and then I would have a trip or something where I couldn’t write. As with my first book, I would bank days by writing multiple sessions per day beforehand and then spend those banked days on days that I didn’t write. But for the most part, I wrote every day.

In the blog post I wrote after my first book, I whined about how I had to write on days when I traveled for work, on holidays, when the kids had sniffles. At the time, it truly was one of the hardest things I’ve ever done.

This time was something else entirely. I wrote the day my grandfather died (peacefully, unsurprisingly) and the day my aunt died (tragically, days after retiring). I wrote the day I found out my Mom had cancer and my children saw me cry for the first time. I was flying to Louisiana to keep my Mom company when I turned on my phone during the layover and discovered a dear friend had had a stroke. I wrote that evening. I woke up the next day and found out she had died. I wrote that morning sitting next to my brother in the waiting room of the hospital while my Mom got her PET scan.

The morning of my friend’s memorial service, I wrote in the hotel. Later that day, I openly sobbed in front of a room full of people. The next day, my wife found out her aunt had terminal cancer. I wrote on the flight home.

See that dog up there in my profile photo? That’s Ginny. She’s on the back cover of my first book. Her myriad health problems finally caught up with her last spring. People sometimes ask, “When did you know you were an adult?” For me it was the day I made the call to put my dog down. The hardest part was watching my kids say goodbye to her. I’m tearing up now writing about it. I ran my fingers through Ginny’s silky fur as the sedatives took her away. I only got through 59 words that afternoon.

I wrote the day the US somehow elected a racist, abusive, corrupt demagogue, and every day afterwards as I saw my country and others turn towards hate and authoritarianism. I wrote while climate change and income inequality worsened. And now here I am writing at home on the same desk where I work now, quarantined like most of you all, hoping to survive the worst pandemic the world has seen in a century.

This is not about how disciplined I was. Because during what have been some of the worst years of my life, a weird inversion happened. It’s not that I was going through that shit and still writing in addition to it. I had to keep writing. Writing was one thing I could still control in the face of many things I could not. If I could make it through the book, maybe I could make it through the other things too. If I had skipped a day it would have meant that the cancer or the deaths beat me that day, that they were stronger than me. I feared what it would mean to me to let go.

I got through these four years and kept writing, but I paid a price. When I read the earlier chapters, they have a whimsy and light-heartedness that later chapters lack. We’re all going through dark times, and I don’t feel light. The past few years left a mark on me, and that mark shows up in the book. I miss the goofier person I used to be, sometimes. But I’d like to believe that maybe the person I am now is a little more honest. Maybe some of those jokes were a mask.

And, thankfully, Mom is in remission.

Psychological self examination aside, I did keep up the writing. Which is good because, man did I underestimate this book. I was aiming for 60,000 words and hoped to get it done in about a year. Here I am four years later sitting on a quarter of a million words.

People sometimes ask what it’s like writing something that big. I’ve been asking myself that for the past couple of weeks. And the weird thing is, I don’t know. I’ve had my head down for the past four years and haven’t looked past the next paragraph or two the entire time. What does it feel like to write an email or draw a picture? Writing the book felt like that. I just happened to do it over and over again. I feel like a marathon runner who’s been watching his feet the whole time and didn’t even notice when he stumbled over the finish line.

Now what?#now-what

Crafting Interpreters is complete now. I had to stop here for a minute and look at that sentence. I’ve been working on this book every day for around 1,400 days. I can’t wait to take a break. So that’s the next step. My plan was to finish the book right before spring break and enjoy a week on the beach with family.

That beach trip went the way of so many other plans in early 2020, but I still intend to take a long break. I don’t know if you noticed, but we all have a lot of other shit to deal with right now. I’m going to relax.

Every morning since 2016, I’ve woken up with a task I had to do. Until I got my writing done for the day, it was on my mind, weighing me down. Writing left me drained. If you’ve ever had a newborn, you know the feeling of always having to carry the baby around. After a while, it’s like you forget what it’s like to have two free arms. I’ve been carrying this baby for four years, so I’m looking forward to having both arms for a while.

Once I’m recharged, the real fun starts. Having the book online is important, but for me, Crafting Interpreters was always meant to be a book with pages and a cover. So after a long bout of editing and bug fixing, I’m going to get started doing the page layout for the print edition. I love graphic design, and I can’t wait to hold it in my hands.

If you’d like to hold it in your hands when it comes out, I have a mailing list where I’ll let you know when the book is done. In the meantime, I think I’ve earned some rest.