Our team has a bit of a disagreement on using containers for agentic coding.

We all prefer coding with our settings "just so." We've got our vimrc's, our bashrc's, our bookmarks, our "set -o vi"'s, our gitconfig's, our color schemes, our prompts… coding on our own machines is better.

And yet we've all experienced the opposite: a CI job, failing mysteriously every tenth build, on a machine where you can't get at the logs or the noisy neighbours. (Perhaps we've even used tmate (see https://blog.philz.dev/blog/ci-debugging/) to break in, or perhaps we're inspired and have set up a Go SSH server that embeds a tailscale funnel (a la https://tailscale.dev/blog/embedded-funnel...).

And there's the in between, too: we develop on a Mac, but need strace (and don't want to disable System Integrity Protection to use dtruss) or want to use subtrace, which requires eBPF, or just want the same version of Python that's in our deployment environment… and we end up working semi-remotely in a Linux VM or container one way or another…

And, so, when we're working with agentic coding assistants, do we want the agents to work in a container or on our local machine? Our coding agent has been defaulting to container, and I think that's the right call, though contentious! I have found a few key features make working with an agent at a distance not just palatable, but better than working locally.

Accessible Containers

Both local and hosted Sketch containers are accessible via SSH as well as a terminal in the web UI. If you need to login to fly.io, or manually do some git operations, those access points are sufficient. (My esoteric favorite is "git fetch ssh://root@sketch-ebmc-zv4m-y7w9-ysdt/app" followed by operations with FETCH_HEAD.)

Providing SSH access also gives you VSCode integration, with VSCode remoting. Sketch sessions offer a link in the UI to open up the project with VSCode. We'll be honest: it's not as good as VSCode locally, but it works for code navigation.

A corollary is that the containers shouldn't disappear while you're still working on the problem. If the containers are too ephemeral, that is, if they're only around while the agent is doing its job, you don't get a chance to inspect what's going on if you need to.

Pretty Good Diffs

If you're reviewing the code the agent is writing, you need to look at diffs. Sketch's right-hand-editable diffs use Monaco to give you a pretty good editor experience at looking at the diffs, and, importantly, to leave comments for the agent.

HTTP Proxies

When the agent starts an HTTP server, Sketch detects an open port, and proxies it to a new, unique, TLS-protected endpoint that only you (and, eventually, anyone you share it with) has access to.

This feels like magic. No port conflicts; works from your phone if you want to test mobile; it's seamless.

Custom Base Images

Sketch (currently only in local mode) lets you specify a base image to use. For the most part, agents are great at installing what you need, but, to save time and money, you can bake that into an image. You need something very similar to this image for CI anyway, and you can usually ask the agent to generate it based on what it finds in github workflows and such.

A Browser Tool

Some of the container-less agents I've used have started a web server, and asked me ever so politely to open up my browser and go look at it. That's not how this works… the agent should use its own browser to go look at it. I've not found Anthropic's models to have a keen eye for identifying failing CSS styles, but loading the page and checking the console logs, and verifying the text gets you a long, long way. (You can, of course, do this locally by providing the Playwright MCP.)

A Tabular Comparison

Put in tabular form, here are some advantages for each way of running agents. There are, of course, nuances to all of these.

So, which is better?

I use both. Containers let me work without evaluating "do I want to let my agent run command x" all the time, because I know the container has limited permissions to begin with. Containers let me experiment with runtimes and frameworks that I have no immediate intention of learning how to install on my laptop. Containers let me work in parallel without worrying about the state of my git repo. On the one hand, with containers, I'll have to use git to get the changes back. On the other hand, without containers, I'll have to use git to figure out what the agent did and what I was doing, and why did I start the agent in a dirty workspace, oh my goodness. If I'm in the midst of a thing in the editor, however, and I need the agent to do a small task, the container feels cumbersome, and I'll reach for a local agent…

What do you think?

We'd love to hear your thoughts on containers vs. local development for agentic coding! Fill out our quick form to share your experience and preferences.

The image at the top of this blog post is from The Internet Guide for New Users by Daniel P. Dern.

We had a series of mini-outages at sketch.dev on July 15th, caused by LLM-written code.

Timeline

A deployment appeared initially stable. Some time afterwards, CPU spiked, and service incrementally slowed to a crawl.

Profiles pointed to some absurdly complicated SQL queries that were doing multiple full table scans. The hypothesis was that they had reached a tipping point at which the database could no longer keep up. Either way, those queries needed reform.

We fixed the queries and deployed. The same thing happened: initial stability followed by a downward spiral.

A bit of discussion indicated that the trigger for the CPU spikes both times was our CEO logging in. We re-deployed to get a clean start, permanently banned him from the service, and moved on.

Profiles continued to indicate database contention, but that no longer seemed plausible. Walking up the stack, there was one infrequent code path responsible for triggering all of those queries. This code path had recently been refactored. We reverted the refactoring, deployed, un-banned the CEO, and set about analysis.

Proximate cause

We (obviously) use Sketch to build Sketch. This refactoring was done by an LLM, and then reviewed by a human. The bug was in a large chunk of code that moved from one file to another.

Before the move, it looked like this:

for {
	repos, repoResp, err := ghClient.Apps.ListUserRepos(ctx, *installation.ID, repoOpt)
	if err != nil {
		// Log error but continue with other installations
		log.Printf("Error fetching repositories for installation %d: %v", *installation.ID, err)
		break
	}
	// ...
}

After the move, it looked like this:

for {
	repos, repoResp, err := ghClient.ListUserRepos(ctx, *installation.ID, repoOpt)
	if err != nil {
		// Log error but continue with other installations
		log.Printf("Error fetching repositories for installation %d: %v", *installation.ID, err)
		continue
	}
	// ...
}

The break became a continue, turning errors into infinite loops.

Root causes

This is a small enough change in a larger code movement that we didn’t notice it during code review.

We as an industry could use better tooling on this front. Git will detect move-and-change at the file level, but not at the patch hunk level, even for pretty large hunks. (To be fair, there are API challenges.)

It’s very easy to miss important changes in a sea of green and red that’s otherwise mostly identical. That's why we have diffs in the first place.

This kind of error has bitten me before, far before LLMs were around. But this problem is exacerbated by LLM coding agents. A human doing this refactor would select the original text, cut it, move to the new file, and paste it. Any changes after that would be intentional.

LLM coding agents work by writing patches. That means that to move code, they write two patches, a deletion and an insertion. This leaves room for transcription errors.

Looking at the code, it’s easy to see why a transcription error occurred:

if err != nil {
	// Log error but continue with other installations
	log.Printf("Error fetching repositories for installation %d: %v", *installation.ID, err)
	break
}

The comment said but continue. The code said break.

There were two competing sources of signal here for what token to predict at the critical moment: transcription and local prediction. Transcription said break. Local prediction said continue. Unfortunately for us, local prediction won.

Prevention

We just added clipboard support to Sketch's agent environment. While patching files, the agent can now copy to and read from a clipboard. Because clipboards reproduce code byte-for-byte, but refactors sometimes require indentation level changes, particularly in Python, the tool also supports adjusting indentation while pasting. (Integrating an LSP for automatic re-indent-on-paste is future work.) It doesn’t have many miles on it yet, but initial results look promising, at least with the more powerful models.

And I’d love to see git add cross-hunk change detection. It would always have been helpful, but in a world in which more code is written by imperfect transcription, it will be even more important.

After a few months with Sketch, these are things I rarely do by hand any more. Sketch's diff view makes it easy to review what Sketch has suggested, so I start with prompting Sketch, and then I review what it's done.

1. Rebases/Merges

Co-worker snuck that big refactor in and you're staring at a wall of merge markers. Ask Sketch to do it.

2. Grafana Dashboards and Prometheus Queries

Added some metrics in the last deploy and need to build out the dashboard? Start using Infrastructure as Code for Dashboards, and ask Sketch to do it. Review the results and iterate. Give Sketch API keys to talk to Grafana.

3. One-off SQL Queries

4. Build Failures

Use your discretion on whether it's the kind of build failure that is amenable to this, though.

5. Add ESLint to a Project

Trust me, adding the eslint rules, and then adding all the ignores for the existing code, taking care to find out that JSX comment syntax makes the ignore comments tricky to automate, is a pain. Sketch does it fine.

6. Writing GitHub Actions Workflows

Look, I like writing shell in YAML as much as anyone else…

7. Git Bisect

It is hard from a general description of what a coding agent is to understand how they can help you be more productive programmer. This is an example of how I got value out of an agent today.

Users reported that sketch containers would build up zombie processes. One way to get that from a Go process is to call exec.Start() and not call Wait(). So I grepped through the code for Start() and read the call sites. They were all fine.

I regularly dive into a moderate issue, spend a few minutes on it, and if I do not make any progress, move on. Given my grep had failed, I was going to call it there. Sketch still works with the zombies, it just makes some operational issues for our hosted deployment a bit more annoying. But on a lark I decided to ask sketch to do some more legwork for me.

After a while it came back with an idea. It was wrong.

Its reasoning was what you might expect from a peer when you're pair programming and skimming code: cmd.Wait()'s error is not handled, there's some odd timeout handling in another goroutine. But it doesn't matter, I believe this code will work.

Again I thought I was done, but out of habit I responded to Sketch just the way I would to a human pair programmer.

After this, I forgot about it and was working on another task.

When I found the tab again, Sketch had reached a new conclusion.

That is exactly the problem. Not only that, on reading its summary, it was clear to me that was the problem. I missed it because I grepped for Start() in our codebase, not our dependencies. So I missed the one in the third party pty package. And it looked to me like its solution would work.

At this point I could have said, "great, do it" and would have had a commit. But I wanted to get in there, replicate it for myself, and remove that TERM/sleep/KILL idiom that's unnecessary and so did it by hand.

So that is an example of how an agent made me more productive today. In days past, I would have stopped after my initial grep/read-through failed and gone to another task. Instead sketch gave me some momentum and helped me fix issue #181.

When I program with Sketch, I find myself part product manager, part software architect, part code reviewer, and part tester. There's a blog post to write about each, but let's talk about reviewing code.

In our diff view, the right-hand side is editable. If you don't like something the agent did, you can just edit it directly, right there when and where you're reviewing the diff.

I learned this trick many moons ago from John Reese at Google, who introduced me to the idea and kindly shared his perforce-flavored script for the same goal. I long ago adapted it for git in. It's a shell script and a vimrc that lets you navigate (Ctrl-N for next) across your diff and edit the right hand side. I've been doing it to add comments and double-check my patches for decades! It makes just as much sense when the code you're reviewing is coming from an agent.

Of course, our diff view also lets you leave comments for the agent. We thought about introducing a complicated data model for it, but it turns out that just appending your comments to the chat box works quite well.

On the evening of May 2nd, the Rockets beat the Warriors 115-107. While I was watching that, I was also scratching an itch: Sketch should have a browser at its disposal! After all, if an agent can do so much with just bash, it could do even more with Chromium.

Here's the feature now:

To my surprise, Sketch was able to get the bones of this feature built with minimal prompting. I used the "Farm Out Step 0" strategy to get a research report on what Go library to use to control Chrome and what the API should look like. I pasted that into Sketch, and watched the Warriors get crushed. By the time I went to bed, I was giddily looking at a PNG of xkcd.com inside the Sketch chat window!

After the initial excitement, the most painful bit was installing Chromium in Sketch's containers. (This would break again when we switched to an Ubuntu 24.04 base image, but I snappily digress…) Sending images to LLMs is a bit subtle, too, and I accidentally sent Claude many base64 encoded images as chat messages, speeding right through my token budget.

After the browser tools, we also added the "background process" tool. Instead of running a command once with a timeout, and returning the results, the LLM calls the bash tool with a "background=true" flag, which returns the pid and the name of the temp file where the output has been redirected.

These tools have become a crowd favorite and surprisingly effective. With a dev server that can spin up mock data for your web components, you can give Sketch goals, not instructions and it'll iterate for longer. It does sometimes declare success too early, but since you've got the screenshots right there, it's easy to spur it to keep going with some specific feedback.

Sketch itself is open source, and we've got a hosted version up on sketch.dev. Give it a spin and take some screenshots.

This is the second part of my ongoing self-education in how to adapt my programming experience to a world with computers that talk. The first part, How I program with LLMs, covered ways LLMs can be adapted into our existing tools (basically, autocomplete) and how careful prompting can replace traditional web search. Now I want to talk about the harder, and more rewarding act of using agents to program.

Define Agent

It is worthwhile starting with a definition of the word “agent” in the context of LLMs. The “AI” hype cycle has been throwing this very generic word around longer than agents have actually been useful constructs. As a result there is a bit of smoke and mirrors marketing and general mysticism to dig through to find any value in the word. For someone with an engineering background there is now a straightforward definition: an agent is 9 lines of code. That is, an agent is a for loop which contains an LLM call. The LLM can execute commands and see their output without a human in the loop.

That’s it. Like all simple things your instinct, quite reasonably, could be to say “so what?” It is a for loop. But the results are an astonishing improvement on the power of a raw large language model.

Whiteboard programming

Consider yourself in front of a whiteboard, writing a function in C with a marker designed to test if a UTF-8 string is valid. (This has actually happened to me, it is a standard interview technique. The stakes were high, this interview changed the course of my career!) How well you do at this task depends on your experience as a programmer, and your ability to fudge your inability to use external resources. You need to remember the encoding of UTF-8. You need to not confuse your C grammar with all the other C-like programming languages you have used over your career (is it name-then-type or type-then-name?). In daily life you get feedback from a compiler if you make a mistake, you can look up a specification of UTF-8, and best of all you can write your program and sprinkle some printfs in it to see what you got wrong.

Asking an agentless LLM to write code is equivalent to asking you to write code on a whiteboard. It is an exercise in dredging up half-forgotten memories, executing a parser on an extraordinarily inefficient substrate, attempting to avoid hallucinating an actually-useful programming interface. The fact LLMs can invent programs out of whole cloth is an astonishing technical achievement that I consider myself blessed to have lived to see, but it is also not a huge shock that attaching a GPU to a virtual whiteboard is not going to do a significant amount of useful programming.

But what happens if we give an LLM more than a virtual whiteboard? What if it could call the compiler, see the compiler errors, and get a chance to fix them before we see the result? What if it could use `grep` and `cat` to read existing files in a project? What if it could patch multiple existing files, including the unit tests and repeatedly run the tests? Agents are feedback-driven LLMs.

Agents are LLMs with environmental feedback

Just as humans thrive in environments with feedback, LLMs go from nice demos to useful programmers when given a surprisingly small core set of tools that are very familiar to programmers:

• bash(cmd)
• patch(hunks)
• todo(tasks)
• web_nav(url), web_eval(script), web_logs(), web_screenshot()
• keyword_search(keywords)
• codereview()

Agents are very good at navigating code bases using the bash tool: find, cat, grep -R, just the way those of us who predate IDEs tend to navigate code bases. Upfront it is instructed to commit changes to git, which it does by using the bash tool to run git add, git commit, etc.

The result, compared to an LLM generating code without these tools available is significantly different. Of note:

• API use is greatly improved because the agent can web search for documentation and `curl` the docs into its context window.
  
• Compiler feedback reduces syntax errors and hallucinated interfaces.
  
• The compiler in a full development environment also improves dependency management, by helping the LLM understand features of the particular version of a dependency that a project uses. (This is an ongoing weakness of LLMs though, they will use documentation from a newer API version or make assumptions that apply only to older versions of a dependency. We have plans to address with this sketch.dev)
  
• Test failures help find errors in generated code, and have a positive reinforcing effect of getting the LLM to write tests for new code.
  
• LLMs can handle larger code bases than fit in the context window because they selectively choose which pieces of the codebase to read.
  
• Agents can try the end product themselves: run the code, take a screenshot of a page from the browser, feed it back into the model and keep tweaking the CSS based on the end-to-end rendering. When things go really wrong, read the server logs, find the panic, fix it and add a test.

The downside of agents is time. A single sentence request that would have required a 200 token response can now generate tens of thousands of intermediate tokens driving tools, some web searching, and many runs of a project test suite. This takes several minutes and there are only so many cups of coffee we can healthily brew in one day.

Today it may also appear costly (my last significant agent-driven commit cost me $1.15 in API credits!) but the cost will rapidly disappear as the chips driving the models continue to improve. GPUs are not nearly as restricted in their future progress as CPUs which are far more constrained in how they can physically improve (compiler-driven ILP only goes so far). The fact that we still call LLM chips “graphics” chips shows the economic machine underneath software has a lot of restructuring to do before it is fully focused on LLM performance-per-watt.

Ultimately, agents spend CPU and GPU cycles doing intermediary work so that humans do not need to. Any time I can mechanise my labor I get more done, so agents are a huge step forward for me. I only end up writing a tiny fraction of the programs I want to due to a shortage of hours in the day, and with agent support I get a little further into the list of programs I wish I could write. May we all be so lucky as I have been this last year. As such, I am thoroughly convinced that agents are worth significant engineering investment addressing their limitations.

It is relatively easy to see an example of an agent producing useful work. Drop one into your project, break off a small task and type it in, see what it does. Let me give you two examples.

Example: Github App auth

Let me talk you through an example of where I used an agent to do a significant amount of work on a project. I implemented the first pass of Github App auth for the hosted sketch.dev using sketch. It did the whole thing with 3-4 pieces of feedback as I clicked around the interface finding errors. That is an astonishing achievement. It is easy for the haughty to disregard the work of gluing well-known APIs together as “not real” programming, but it has been the practical experience of my career that for every hour of truly interesting programming I can do, I have had to do 10 or more hours of dreary work with APIs, libraries, dysfunctional compilers, build systems, or obtuse package managers to actually make that hour useful. Having a tool that lets me do 30 minutes of “not real” programming by writing a few carefully constructed sentences, and lets me vacuum the kids' room while it works, preserves momentum.

But now comes the work. It implemented the GitHub app auth flow I wanted. It even met the stringent requirement I put on it: I asked it if it could avoid saving a token per user, and instead use the global private credentials of the app to drive everything to keep the database simple. It did it! In the process of doing it, it wrote some very bad code.

First, is the huge security vulnerability it created. Because it let anyone who had authorized the app work with any repository authorized with the app, even if they did not have access to it. Disaster. Fortunately it is so bad that if we had got to testing with others on the team we would have spotted it quickly when our own private repos started appearing on each other’s repository lists. (And was such an obvious problem I caught it even before that.)
A quick explanation of the problem to sketch.dev got it to fix it, it implemented authorization checks for users and got it right. Another amazing achievement, I wrote one sentence and got a reworked functioning commit on a branch.

The next problem was performance. While the new code worked, it would have been unworkably slow as soon as it had several users. It generated a list of repositories the user had access to by:

for install := range allAppInstallations {
	for r := range install.Repositories() {
		if r.IsCollaborator(user) {
			// add to available repositories
		}
	}
}

This meant that every time I wanted to display to the user a list of repositories they had authorized, I had to, for each GitHub org that had allowed the use of Sketch, do an API call to fetch their repository list, and then for every single repository do another API call to check if this person is a user. This means the number of API calls grows with the total number of product users, which is not going to work.

It turns out, the problem was my original naive requirement to avoid storing per-user tokens from GitHub. It turns out they do not have any efficient app-auth-level API calls to determine what a user can access, the only way to do it efficiently is to ask what access the auth token has, and use an auth token for the user.

Once I realized this I told sketch to go back and remove my original requirement: save per-user auth tokens and use them for everything. It quickly came up with efficient API calls.

Telling this story took more words than I typed in total into Sketch to generate the GitHub auth code I wanted, and writing it took more effort than the code reviews that caught all the issues. I need to emphasize that because it is so easy to take a part of an anecdote like this and use the limits and mistakes of these tools to declare them “useless” or “dangerous”, while my experience is not that at all. What we have today can clearly not replace me as a programmer, but it did let me get done in a day a dreary task that I would have struggled to traditionally complete in a week. And I vacuumed the kid’s room while I was at it.

Example: SQL conventions around JSON

Here is an example of something my agent needs to do regularly that it struggled with until I found a way to help it (and I believe captures a typical limitation people run into when first trying to work with LLMs).

I learned an odd way of using SQL at Tailscale (from Brad and Maisem): make every table a JSON object. In particular, have only one “real” column and the rest generated from the JSON. So the typical table looks like:

CREATE TABLE IF NOT EXISTS Cookie (
  Cookie   TEXT    NOT NULL AS (Data->>'cookie')  STORED UNIQUE, -- PK
  UserID   INTEGER NOT NULL AS (Data->>'user_id') STORED REFERENCES User (UserID),
  Created  INTEGER NOT NULL AS (unixepoch(Data->>'created')) STORED,
  LastUsed INTEGER AS (unixepoch(Data->>'last_used')) CHECK (LastUsed>0),
  Data     JSONB   NOT NULL
);

This has many pros and many cons. It acts as a poor man’s ORM as each table has an “obvious” data type matching each record. It makes adding to the schema trivial. You can choose to ADD COLUMN if you like but you do not have to. The SQL column constraints act as good dynamic checks on the quality of your JSON. It greatly increases the amount of stored data per row. You have to structure all your INSERT and UPDATEs in terms of JSON. Half a foot in the document database world, but I can still write an old fashioned JOIN. Etc.

Pros and cons aside (that can be a fun blog post for the future), our agent often tripped over this style. When creating new tables and columns, it would sometimes, but not always, follow the generated column pattern. The very first time we added a table that did not use this all-generated-column style it got further confused, and would choose almost at random it seemed between styles.

It turned out to be really easy to fix the agent’s behavior. At the top of the SQL schema file I tried adding a three sentence description of this. The key line seems to be “each table has a single concrete Data JSON column, all other columns are generated from it” and then a comment on the tables that did not follow this pattern explaining they were an exception to the norm, and the behavior improved dramatically.

This is slightly counter-intuitive. My lived experience of instructions like this is that engineers heavily discount them. It may be ad(-like) blindness, it may be the challenge of keeping comments up to date, or it may be that most comments are not worth much attention.. Instead the industry norm for communicating this kind of knowledge is an unpleasant routine of engineers who do not know writing a PR that does it the wrong way, then receiving comments on it and telling them to do more work. LLMs seem to give the comments more work, hopefully for the better.

“Asset” and “debt” models of code

One of the arguments against LLM as a code generation tool is that generating code is only a small fraction of the overall cost of code. The ongoing work of dealing with existing code is the vast majority of the cost, goes the argument. There are code bases where this statement is clearly true. In heavily used products with a growing user base adding new ways of using programs, engineers do spend most of their time navigating the unwritten misunderstood interdependencies of existing code. To someone whose job is that, a computer you can talk to that produces a usable result for “implement bubble sort in fortran” sits somewhere between a toy and nuisance. Sometimes attempts to compare this to financial concepts such as “assets” and “debts” are used, but I will skip them as they never seem to quite fit.

Whether this understanding of engineering, which is correct for some projects, is correct for engineering as a whole is questionable. Very few programs ever reach the point that they are heavily used and long-lived. Almost everything has few users, or is short-lived, or both. Let’s not extrapolate from the experiences of engineers who only take jobs maintaining large existing products to the entire industry.

Fortunately we do not have to answer the question of what the entirety of programming looks like to say if agents are valuable, because agents are potentially useful even in the maintenance of existing products. An agent is not just code generation. It is an LLM with a series of tools that reads code, and changes code by editing files.

Agents are as happy removing code as adding it.

The result is change. Yes, the change is “more work” because the person driving an agent has to understand the change being made. Yes, the agent may not have sufficient understanding to change large products yet. But change is the ultimate goal of the engineer driving the tool, and agents are demonstrating the ability to edit moderately large projects with some care. That makes them potentially useful tools everywhere in the programming industry. If agents are not good enough yet (and that’s a big if, it is well worth testing), they are on the right track and now have all the fundamentals to get there.

A related, but tricker topic is one of the quieter arguments passed around for harder-to-use programming tools (for example, programming languages like C with few amenities and convoluted build systems) is that these tools act as gatekeepers on a project, stopping low-quality mediocre development. You cannot have sprawling dependencies on a project if no-one can figure out how to add a dependency. If you believe in an argument like this, then anything that makes it easier to write code: type safety, garbage collection, package management, and LLM-driven agents make things worse. If your goal is to decelerate and avoid change then an agent is not useful.

Why are we seeing agents now?

Unlike somewhat mysterious ideas like the transformer architecture that underpins LLMs, mechanical feedback into an LLM seems “obvious.” It is clear to those of us who think about developer tools: I have been working on it for over a year now, but our first revision of sketch.dev in January, despite having go tooling wired into an LLM, hardly counts as an agent by the standards of what we can use today. (The difference in utility between the first revision of sketch and the current sketch open source project is astonishing.) The utility of feedback is also clear to everyone who works in the ML field, as reinforcement learning has been one of the core tenants of the field for 50 years.

The answer is a critical chunk of the work for making agents useful is in the training process of the underlying models. The LLMs of 2023 could not drive agents, the LLMs of 2025 are optimized for it. Models have to robustly call the tools they are given and make good use of them. We are only now starting to see frontier models that are good at this. And while our goal is to eventually work entirely with open models, the open models are trailing the frontier models in our tool calling evals. We are confident the story will change in six months, but for now, useful repeated tool calling is a new feature for the underlying models.

What is next

It is challenging to think about what is next in a fast-moving field where most engineers today are not even using these tools yet. But those of us building these tools need to think about it.

Most use of agents today is either in the IDE or in a checked out repository on a development machine. This is an easy way to get started, it’s easy to install a vscode fork or a command line tool and run it. But it has two significant limits.

The first major limit is that agents need a lot of safeguards built into them to avoid them running amok. One of my machines has production credentials squirreled away on it I can use to do deployments. As part of running commands is the agent going to grab those credentials and run my deploy script for its uncommitted changes? Avoiding this, if your agent runs directly on your real computer, involves a lot of asking the programmer to babysit the tool calls and give the agent permission to run commands. And even then, there are dangers. I can say “yes to all” to running curl, only to forget a web server I’m developing that’s running on localhost doesn’t have any auth yet and can read arbitrary files off disk. Oops, my production credentials are exposed again.

The second major limit is requiring developers to use their bespoke manually-configured development environments to run an agent means we are effectively serializing execution of an agent. As mentioned earlier, one of the major weaknesses of agents is they take several minutes per turn to produce good results (and probably will for the foreseeable future). One way to make better use of our time is for an engineer to be driving several agents simultaneously, but this sort of agent deployment makes that impractical.

We are exploring solving both of these issues in sketch.dev using containers. By default sketch creates a little development environment in a container with a copy of the source code and the runner has the ability to extract git commits from the container. This lets you run many simultaneously. (Other agents are also exploring this space, agents in general are a very busy space!)

To give you an example of how this parallelism can work in practice, while I was working on the GitHub auth mentioned in the above example, I was about to complain in a group chat about how ugly a form I made is. Instead, I opened a second copy of Sketch and pasted in a screenshot of the form and wrote “this is ugly, please make it less ugly.” I went back to thinking about auth, and when I remembered I had typed that a half hour later I looked at the results and decided yes, it is an improvement. So I asked it to rebase, it resolved the merge conflict (one of my least favorite programming tasks!), and I pushed the update. It was not nearly the quality a real designer can produce, but it was certainly better than the awful unstyled form I had created during testing. In a past life I would have created an issue in the issue tracker, by which I mean I would have made a note in my personal to-do.txt file to create an issue, because issues are visible to other programmers and require typing something more coherent and constructive than pasting a screenshot and saying “this is ugly, please fix.” One of the great things about talking to an agent is if you only have a tiny bit of mental energy left you have a good chance of getting something of value out of 30 seconds work. The likelihood I would have ever turned the to-do.txt entry into a real issue is, honestly, pretty low.

So our takeaway from exploring the UX of agents for the past six months is that we may finally have a good use for “development” containers.

What does the IDE become?

An open question that we are spending a lot of time exploring is: what does the IDE become in this environment? Let us say we start work by talking to an agent. The execution container can be completely derived from GitHub, the changes shown as a diff and pushed as a branch (or PR). Is that the actual workflow?

In practice, many of the commits generated by sketch, or any other agent we have tried, require some human cleanup afterwards. (Our experience is, when programmers first use an agent most commits need manual intervention, but practice writing prompts reduces the number of necessary intervention.) It could be as simple as editing a comment or changing a variable name. It can be more significant. How do we make that work with a containerized world?

So far we have several workflows that we quite like that other agents have not yet explored. One is making the diff view editable. You can type in the right-hand-side of Sketch’s diff view and it ends up in the commit and pushed for you. It is great for one-line edits.

For the sorts of fixups where you want to run sed, or grep through the changes, or run the tests in an interesting way, we have had great success with giving the user ssh access to the container. Not only can you shell in (and we have a little web terminal in the UI too), it is easy to turn this into a vscode:// URL that can be opened directly in a traditional IDE, which is sometimes exactly what we want.

Finally, we let you write “code review” style comments on the diff view in sketch.dev and send them back to the agent as feedback. Commenting directly on a line of a diff can greatly reduce the amount we have to type (and is very familiar from our long practice in the code review process).

Overall, we are convinced that containers can be useful and warranted for programming. The idea has been around for a long time but I have never personally wanted to start programming in a container. But cleaning up a diff that an agent wrote for me in a container is far more interesting.

A final note

The process of learning and experimenting with LLM-derived technology has been an exercise in humility. In general I love learning new things when the art of programming changes: dealing with the switch to multi-core programming, rethinking software design when SSDs replaced HDDs and dropped seek latencies, when everything was suddenly reachable on the same inter-network, these sorts of shifts in the industry are a joy to deal with. (Not to be confused with useless makework like the latest JavaScript framework, the latest cloud provider service, or the latest cluster orchestration software.) These sorts of challenges affected how my programs work: the choice of algorithms, languages, libraries, etc. But LLMs, and more specifically Agents, affect the process of writing programs in a new and confusing way. Absolutely every fundamental assumption about how I work has to be questioned, and it ripples through all the experience I have accumulated. There are days when it feels like I would be better off if I did not know anything about programming and started from scratch. And it is still changing.

The way this all works today is very different than six months ago, and I do not believe we are at a stable point yet. I believe a lot of norms around team interactions will also be changing. For example, the mostly-broken process of half-hearted code review that has been adopted across the industry no longer solves the problems it barely solved before. It needs to be reinvented. The “IDE”, which has never been nearly as integrated as it has claimed to be, needs to be torn up and repurposed. The industry now seems aware of this, but has not taken an agent-first approach yet. There is a lot to do, and I suspect in six months things will be very different again. Curiosity and humility will get us through it, but even more than usual I would suggest turning away from internet forums where people talk in circles about this technology. That is a job for an agent.

This post also appears on my personal blog.

Thanks to Sean McCullough, Philip Zeyliger, and Thomas Ptacek for reviewing this post.

Wikipedia:

User experience (UX) is how a user interacts with and experiences a product, system, or service.

Developer experience (DX) is a user experience from a developer's point of view.

It is time to add:

Agent experience (AX) is a user experience from an AI agent's point of view.

Agent loops are simple. But that simplicity hides some depth. An agent's behavior is deeply influenced by its environment. (The same is true of humans.) To build a setting in which an agent is effective, it helps to view the world from its point of view.

The primary components of AX are obvious and familiar: the system prompt, and the tool selection, naming, design, and documentation. All of these must be designed for how LLMs process information, with an strong emphasis on clarity and precision.

There are less obvious components, though, such as communication design: what information you give an agent and when.

And some of it, as with any form of UX, comes down to polish. We just added auto-installation of common tools to sketch. If an agent executes yamllint, and yamllint isn't present, it is silently, magically installed in the container, and the bash command succeeds the first time. Instead of having to pause its work to futz with apt/snap/yum/brew, which is a distraction, things Just Work. AI agents suffer from yak shaving too.

AX is in some sense easier than UX/DX, because you have captive users that you can watch any time you want! It is so much easier that we sometimes call it "eval" instead.

But it's not just eval. To improve the AX, you have to actually watch and empathize, just like you would with human users…while bearing in mind the ever-changing ways in which they can be decidedly non-human.

Also published at commaok.xyz/ai/ax/

As I was learning prompt engineering, I encountered the same (good!) core advice repeatedly: invest in evals, make incremental improvements.

That’s great. But…where do the words come from? The ones that we evaluate, and iterate on, and augment with examples? When writing code, if you do a good job up front, you’ll spend less time fixing bugs and chasing performance issues. How do you do a better job up front of writing and editing a prompt?

The answer has slowly become clear to me: Have an LLM write and edit your prompts. You are still doing the heavy lifting. The understanding and context you provide is the crux of the work. But an LLM is an unparalleled linguistic optimizer and amanuensis. They are remarkably good at taking muddled, sprawling intentions and crystallizing them into clear, effective prompts.

I’m a native English speaker with extensive writing experience, most of it technical. I am a reasonably clear thinker and communicator. I’ve been writing prompts for a year. And yet frontier models systematically outperform my attempts to do it all myself. Maybe if I learned by osmosis at Anthropic or had another few years of prompt engineering experience this wouldn’t be true. But, to a first approximation, that describes nobody, so I’m pretty comfortable giving out this advice.

Frontier LLMs seem to be particularly good at providing strong guardrails to prevent unwanted but stubbornly persistent behaviors. For example, Claude generated this line, which proved to be strikingly effective in its context, and which I would never have come up with: “If you are about to do X - stop - and reconsider.”

How to prompt for prompts

When asking an LLM to write a prompt, I bring three things to the table.

• Clear instructions. I tell it I want it to help me write a prompt. Obvious, but I still sometimes manage to forget this. It is often also fruitful to request that it ask you probing, critical questions. Make it crystal clear you want engagement, not affirmation. (My Claude personal preference begins: "I value intellectual, rigorous, critical discussion. I want a tennis partner who returns serves with spin - challenge my assumptions, ask probing questions, suggest better approaches, and engage rather than affirm." This makes Claude charmingly fiesty.)
• Context. Lots and lots of context. What I am trying to accomplish, for whom, and why? I pour in my ideas, dreams, hopes, confusions, plans, fears, draft sentences that seem promising, whatever. I ramble and circumlocute. Get it all down. All of it. Order doesn’t matter. It is easy to cut later.
• Judgment. Ira Glass popularized the Taste Gap, the observation that judgment precedes ability: “All of us who do creative work, we get into it because we have good taste. But there is this gap. For the first couple years you make stuff, it’s just not that good.” The rough draft generated by the LLM usually has obvious flaws: possible misinterpretations, misplaced emphasis, omitted nuance, wrong feel. This is where you can leverage the Taste Gap: It is much easier to identify what’s wrong than it is to make it right. Identify everything that doesn’t seem right–even if it is hard to nail down precisely–and tell the LLM, and let it iterate. Tweak, quibble, argue, point out shortcomings, cut. Over and over. If the conversation goes off the rails, learn some lessons from it, copy/paste a bunch of text, and start over.

The same approach works for refining a prompt. Explain what you’re doing, bring context (the original prompt and what’s going wrong), encourage the model not to make major changes (unless you think it’s necessary!), put on your judge’s wig, and iterate together.

Don’t just ask an LLM to improve a prompt. If you do this, sure, it’ll make a bunch of plausible-sounding changes. But anything it can infer from the original prompt is already implicit in the original prompt. You need to bring new information: new thoughts, observed problems, additional judgments. You could ask it to speculatively expand the prompt and then work to cut it back–the judgment involved in that paring down is new information. But the more context and judgment you bring, the more improvements are available.

Lastly, don’t be afraid to make changes yourself, no matter what the LLM thinks. Trust your judgment. And then, of course, eval!

To each their own

Oh, one more thing. I have a pet theory: Models have a house palette. Claude responds better to prompts that Claude writes. ChatGPT does a better job following instructions that it wrote.

I have not put this to an empirical test. (If anyone out there wants to do an experiment...)

This applies to humans, too. Words meant for humans should be written by humans.

Also published at commaok.xyz/ai/prompt-engineering-and-the-taste-gap/

There are obvious similarities between task-oriented communication with a human vs with an AI agent...and a few places where they diverge. This blog post lays out a simple conceptual framework for thinking about these interactions.

Suppose I ask a colleague to tackle a task for me. How do I communicate context about that task to them? There are three options:

• Push: Give them information up front
• Pull: Wait for them to ask me questions
• Respond: Provide feedback on work they’ve completed

Each of these approaches has trade-offs.

It’s hard to know exactly what information would be useful to Push to them. Obviously they need to know what I have in mind, but telling them extraneous information is a waste of everyone’s time, and in extreme cases can be downright obnoxious.

Pulling is less likely to generate unnecessary communication, but it also generates interruptions for me and adds delays for them. It can take a lot of work to even discover what questions to ask. And there’s a difficult balance to strike between over- and under-asking.

Responding can be very efficient: It is naturally batched, and all the resulting communication is necessary. But it also generates wasted work along the way, and can lead to significant frustration if there were large communication failures earlier.

In practice, we all use a blend of these approaches. The exact combination depends on the people, the task, the existing shared context, the level of trust, and the relative cost of communication, interruptions, and work.

Using agents

This same set of options appears when asking an AI agent to do work for you.

The more information you Push to them up front, the more likely you’ll get a better outcome, but there are rapidly diminishing returns. Cutting-edge AI agent systems are capable of figuring out quite a lot on their own from surprisingly little context.

As the AI agents get more autonomous, though, they are used for bigger chunks of work, so they get slower, which means that you don’t want to wait around to be available for Pulling, and interruptions are disruptive. And as agents tackle more on their own, it makes sense to launch more of them concurrently, which means that interruptions can easily saturate your focus bandwidth and sap your productivity.

I’ve started doing more at Response time, and worrying less about wasted work. An agent’s work is cheap (compared to my time) and getting cheaper, and I can always just spin up more agents.

With agents, there’s also a fourth option:

• Restart: Learn from the attempt, throw away everything, and start from scratch, Pushing the extra information that was missing the first time

(You can also do this with humans, but it’s...not very nice.)

Adding Restart into the mix turns out to be surprisingly effective. Push a tiny bit of information, Respond only with significant feedback, learn from the result, Restart with better initial guidance, repeat.

Marc-Antoine Ruel explained this approach well:

Often I ask it to create a change, look at the diff, realize it's wrong but it gave me an idea. I delete the branch and write the correct code manually. Then once it’s scaffolded correctly, sometimes I let it finish the job. I find this faster (less draining emotionally?) than trying to argue.

(If you’re skeptical about having AI-written code in your repo, this may be a thing to try. Let the AI run ahead, make mistakes, pave a path, and convince you that the game is worth the candle. Then write the code yourself, fore-warned and fore-armed.)

Designing Agents

When building an AI agent, you encounter the same set of considerations, but from the design perspective. The product UX choices strongly influence when and how communication occurs.

For example, OpenAI’s DeepResearch settled on the user Pushing their question, then one immediate round of Pull, and then presumably Respond. (I personally have never done any Respond iterations with DeepResearch.) They judged that the quick Pull improved results meaningfully without delaying or interrupting too much.

Here at Sketch (an AI coding agent), for human interactions, we’ve focused mainly on Respond and Restart. Yes, Pushed information to specify the task is necessary, but we want to be able to work well even with very little provided up front.

AI agents interact with their environments, and the same communication design questions arise for agent-environment communication as for agent-human communication. Take commit message styles; that information is available from git without bothering a human. We could analyze those and Push it to the agent at the very beginning. That’s predictable and reliable, but is irrelevant overhead and distraction if no commits get created. Or we could Respond by asking it to rewrite a commit message if/when we automatically detect that it is not in the desired style. (Restarting is always human-initiated, so that’s out.) In practice, we let the agent Pull that information when it is ready to write commits. And we have guard rails: If the agent hasn’t Pulled style information, we refuse to let it commit.

The communication design is a prime question for us when adding any new feature. We employ a combination of all approaches, on a case by case basis, based on what happens to work well.

The field of Agent Experience (AX) is mostly ad hoc, currently mostly confined to prompt engineering and MCP server design. It will likely grow.

AI agents have hit the mainstream, but as an industry we have yet to develop a clear theory (that I have seen) around communication design, both UX and AX. Hopefully framing this question will help open up the discussion.

Also published at commaok.xyz/ai/push-pull-respond-restart/

In the 2000s, we learned to Google.

In the 2010s, we learned to ask Stack Overflow.

In the early 2020s, we learned to prompt LLMs.

That was so last year. Now that it’s 2025, we’re learning to prompt AI Agents, such as sketch.dev (us), Claude Code, OpenAI Codex, and oh so many more. (Hi, friends!)

This is post is a list of techniques that we’ve found to be effective for working with coding agents.

Draw the Owl

Make a code change illustrating what you’re up to. Ask the agent to finish the job. Or do a thing all the way once and tell the agent to read that commit and repeat it.

Use a checklist

For bigger projects:

• Ask the agent to make a plan and break it down into a detailed checklist.
• Write the checklist (temporarily) to your repository.
• Have it work through the checklist, checking items off as it goes.
• Delete the checklist.

Send out a scout

Hand the AI agent a task just to find out where the sticky bits are, so you don’t have to make those mistakes.

Fail early, fail often

Agents are pretty good at getting oriented, and failure is cheap. Throw out your problem, and let it work at it. If it fails, start over, adding the tidbits of information it couldn’t figure out on its own. If the agent goes off on a tangent, don’t try to redirect it or fix it. Cut bait and start over with a similar prompt.

Go Concurrent

Agents tend to be slower than normal human attention span. You can hide that latency by firing up a bunch of them and then checking in with them when you’re ready. You can also have multiple agents tackle the same task and compare the results.

Provide goals, not instructions

Whether it’s speeding up your tests or fixing a class of eslint warnings or iterating on browser screenshots, if the agent can evaluate success, let it do its thing.

Farm out step 0

There are a variety of “deep research” products out there. Have them prepare detailed docs on an API or technology, and pass that along. There are a bunch of UI prototyping tools out there. Explore there and then let the agent integrate.

My co-workers and I have been working on an AI Programming Assistant called Sketch for the last few months. The thing I've been most surprised by is how shockingly simple the main loop of using an LLM with tool use is:

def loop(llm):
    msg = user_input()
    while True:
        output, tool_calls = llm(msg)
        print("Agent: ", output)
        if tool_calls:
            msg = [ handle_tool_call(tc) for tc in tool_calls ]
        else:
            msg = user_input()

There's some pomp and circumstance to make the above work (here's the full script) , but the core idea is the above 9 lines. Here, llm() is a function that sends the system prompt, the conversation so far, and the next message to the LLM API.

Tool use is the fancy term for "the LLM returns some output that corresponds to a schema," and, in the full script, we tell the LLM (in its system prompt and tool description prompts) that it has access to bash.

With just that one very general purpose tool, the current models (we use Claude 3.7 Sonnet extensively) can nail many problems, some of them in "one shot." Whereas I used to look up an esoteric git operation and then cut and paste, now I just ask Sketch to do it. Whereas I used to handle git merges manually, now I let Sketch take a first pass. Whereas I used to change a type and go through the resulting type checker errors one by one (or, let's be real, with perl -pie ridiculousness), I give it a shot with Sketch. If appropriately prompted, the agentic loop can be persistent. If you don't have some tool installed, it'll install it. If your `grep` has different command line options, it adapts. (It can also be infuriating! "Oh, this test doesn't pass... let's just skip it," it sometimes says, maddeningly.)

For many workflows, agentic tools specialize. Sketch's quiver of tools is not just bash, as we've found that a handful of extra tools improve the quality, speed up iterations, and facilitate better developer workflows. Tools that let the LLM edit text correctly are surprisingly tricky. Seeing the LLM struggle with sed one-liners re-affirms that visual (as opposed to line) editors are a marvel.

I have no doubt that agent loops will get incorporated into more day to day automation tedium that's historically been too specific for general purpose tools and too esoteric and unstable to automate traditionally. I keep thinking of how much time I've spent correlating stack traces with git commits, and how good LLMs are at doing a first pass on it. We'll be seeing more custom, ad hoc, throw-away LLM agent loops in our bin/ directories. Grab your favorite bearer token and give it a shot.

Also published at philz.dev/blog/agent-loop/.

For a repo with sufficiently fast tests and sufficiently infrequent commits, GitHub Actions is more than enough to implement a simple merge queue, which can also resolve formatting issues.

We want origin/main to always have passing tests and well-formatted code. When a repo has enough contributors, it's desirable to automate running the tests before pushing to main. GitHub itself offers a merge queue for pull requests, but if you're not using PRs, it is easy to put together a lightweight alternative, as we recently did.

To use it, you run:

git push -f origin HEAD:refs/heads/queue-main-$USER

This pushes the current commit to a GitHub branch with a magic name. A GitHub Action notices the change, runs formatting tools, pushing any necessary formatting changes back to the queue, runs the test workflows in parallel, and, finally, if/when all the tests pass, does a non-force push to main.

If you have few enough commits per day, and your tests run fast enough, this works acceptably well, but conflicts can still occur. (It's a bit counterintuitive, like the birthday problem: the probability of having 2 commits in the same minute across 480 minutes is about 9%.) So far, we've judged the occasional conflict-and-rebase as preferable to breaking main!

We also decided to have the queue fix any formatting issues instead of rejecting the push. Ultimately, there's always some editor or tool or script that doesn't take into account prettier or gofumpt, and time and attention are precious. 99% of the time all we'd do with the formatting change is apply a deterministic tool and retry. Just say no to failing pushes because of whitespace!

If you want to replicate our fun, here's a quick walkthrough of the interesting bits of our hook.

Main Queue

The main queue is .github/workflows/queue-main.yml.

We want this Action to trigger on a change to any branch named queue-main-$USER:

name: Main Branch Commit Queue
on:
  push:
    branches:
      - "queue-main-*"

The queue action itself doesn't need to do anything but read the repo, so we restrict permissions. We'll end up granting more permissions to sub-actions soon, though.

permissions: read-all

The first job is running the formatter. We block on that up front so that the code that gets tested is the same code that gets pushed. We need to give the formatting workflow write permissions, and tell it that it is in "fix" rather than "check" mode. We’ll step through the formatting workflow in the next section.

jobs:
  formatting:
    uses: ./.github/workflows/formatting.yml
    permissions:
      contents: write
    with:
      auto_fix: true

Now we run all of our tests, specifying the commit SHA of our (possibly newly) formatted code. Still nested under the jobs key, we fan out our testing actions in parallel, going back to minimal permissions:

  go-test:
    needs: [formatting]
    uses: ./.github/workflows/go_test.yml
    permissions: read-all
    with:
      ref: ${{ needs.formatting.outputs.commit_sha }}

  ui-test:
    needs: [formatting]
    uses: ./.github/workflows/webui_test.yml
    permissions: read-all
    with:
      ref: ${{ needs.formatting.outputs.commit_sha }}

Finally, we push the results to main. We make sure to push the correct SHA.

  push-to-main:
    runs-on: ubuntu-latest
    needs: [go-test, ui-test, formatting]
    permissions:
      contents: write
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Push to main
        run: |
          COMMIT_TO_PUSH="HEAD"
          if [[ "${{ needs.formatting.outputs.commit_sha }}" != "" ]]; then
            echo "Using formatted commit: ${{ needs.formatting.outputs.commit_sha }}"
            COMMIT_TO_PUSH="${{ needs.formatting.outputs.commit_sha }}"
          fi
          git push https://${GITHUB_ACTOR}:${GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git "${COMMIT_TO_PUSH}":main
        env:
          GITHUB_TOKEN: ${{ github.token }}

Dev Queue

"Testing GitHub Actions" is usually an oxymoron.

But we have a dev hook, .github/workflows/queue-dev.yml, that runs on queue-dev-* instead of queue-main-*. It does all the same steps, except it doesn't push to main. This makes it possible to iterate on the queue action without impacting anything else.

Formatting

Here's a walkthrough of .github/workflows/formatting.yml.

The formatting workflow needs to support both "check" and "fix" modes and needs to return the new commit to be pushed. It must not trigger directly when using the main/dev merge queues. And it needs to be able to write back to the main repo.

name: Code Formatting
on:
  workflow_call:
    inputs:
      auto_fix:
        description: "Automatically fix formatting issues instead of just checking"
        required: false
        type: boolean
        default: false
    outputs:
      commit_sha:
        description: "The SHA of the commit with formatting fixes"
        value: ${{ jobs.formatting.outputs.commit_sha }}
  push:
    branches-ignore:
      - "queue-main-*"
      - "queue-dev-*"
  pull_request:

permissions:
  contents: write

We need to grab the relevant commit and run the formatters in the correct mode:

jobs:
  formatting:
    outputs:
      commit_sha: ${{ steps.get_commit.outputs.commit_sha }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.head_ref || github.ref }}
          # Need full history for queue-main pushes
          fetch-depth: 0

      # omitted here: installing prettier, gofumpt, see original file if interested

      - name: Check formatting
        if: inputs.auto_fix != true
        run: bin/run-formatters.sh check

      - name: Fix formatting
        if: inputs.auto_fix == true
        run: bin/run-formatters.sh fix

And finally, if formatting made any changes, we need to push those back so that we can test and keep them. If there’s only a single commit, we amend it, to keep a clean git history. If there are multiple changes outstanding, we add a new cleanup commit at the end. We could amend each commit in turn, to make formatting fully atomic, but that can cause rebase pain; we’ve opted for a slightly ugly history over rebase annoyance. (Other teams might reasonably make different choices here.)

      # Commit formatting fixes if auto_fix is true
      - name: Commit and push formatting fixes if needed
        if: inputs.auto_fix == true
        run: |
          # Only proceed if there are changes to commit
          if [[ -z $(git status --porcelain) ]]; then
            echo "No formatting changes detected, skipping commit"
            exit 0
          fi

          git config --global user.name "Autoformatter"
          git config --global user.email "bot@sketch.dev"
          git add .

          git fetch origin main
          MERGE_BASE=$(git merge-base HEAD origin/main)
          COMMIT_COUNT=$(git rev-list --count $MERGE_BASE..HEAD)

          if [ "$COMMIT_COUNT" -eq 1 ]; then
            echo "Found exactly one commit since merge-base with origin/main. Amending the commit."
            git commit --amend --no-edit
          else
            echo "Found multiple commits ($COMMIT_COUNT) since merge-base with origin/main. Creating a new commit."
            git commit -m "all: fix formatting"
          fi

          git push -f https://${GITHUB_ACTOR}:${GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git HEAD

Lastly, we expose the new branch head for use by the other steps.

      - name: Get commit SHA
        id: get_commit
        run: echo "commit_sha=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT

Tests

The test actions are pretty straightforward. The one interesting piece is making sure that we are testing the correct commit. This uses a bit of trickery. When inputs.ref is set by the queue, it gets checked out. Otherwise it is empty, and action/checkout defaults to the standard checkout.

      - uses: actions/checkout@v4
        with:
          ref: ${{ inputs.ref }}

Improvements

There are definitely improvements we could make, and probably will eventually. For example, it'd be nice for the merge queue to attempt a rebase when you lose a commit race.

But even the fast and loose version we have is pretty darn nice, and like many small-company tools, is simple enough that you rapidly can grok it and tweak it to your heart's content.

Should I copy this?

Fill in the blank:

> I just came across some code and thought, "wait, how did this ever work?" The answer, of course, is _______.

If you have spent your career at larger companies, the answer is probably something like: Test coverage!

If you have spent your career at startups, the answer is more likely: It didn't!

There's a radical amount of divergence in the experiences, needs, and tooling of "software engineers". Sadly, it's easy to forget this, and end up in silly arguments.

Merge queues are often found at bigger companies, which must, eventually, serialize the work being done by many programmers, and do so without introducing test regressions and chaos.

We are a small, young company, staffed by experienced folk. We operate very fast and light: No PRs, no pre-commit codereview. (Post-commit codereview happens exactly as much as people feel like it, without ego at stake.) If you're doing the work, you make the decisions.

This merge queue works well for us. It's not mandated, but we all choose to use it.

If it sounds like a good fit for you, steal at will.

Alan Kay: "Simple things should be simple, complex things should be possible." Git absolutely nails part two. (And in git's defense, it is possible that there are no simple things in version control.)

This blog post is about manipulating git (both "intentionally deceiving" and "running git commands") to send a very precise fragment of a git repository to a server, do some work with it, and transfer that work back to the original repository, smoothly and transparently. Prepare to roll up your sleeves.

Background

merde.ai uses LLMs to resolve git merge/rebase conflicts for you (website, blog post). In order to do that well, it needs part of a git repository: relevant history with diffs and commit messages. And yet it doesn't need all of your git history. That would be bad in many ways.

What we'd really like is to grab a chunk of a repository that is just big enough for the problem at hand, but no bigger.

False starts

Git being git, it has lots of officially supported built-in ways to do a partial clone. Here are a few of them:

• Shallow clone by depth: git clone --depth=1 <url>. This limits the history transferred to the most topologically recent commits. This is useful for CI systems that only need the most recent commit to test.
• Shallow clone by date: git clone --shallow-since="3 days ago" <url>. This limits the history transferred by the commit time. This is useful for humans who don't care about the full history of a project.
• Sparse checkout: git clone --no-checkout <url>; cd <repo>; git sparse-checkout set <path1> <path2>; git checkout. This lets you check out only specific paths. This is useful for large monorepos in which you only care about a particular set of directories. As of git 2.42.0 it is still experimental.
• File size filters: git clone --filter=blob:limit=1m <url>. This lets you skip large files, which might be assets that you don't need for your task.

And there are more (--single-branch, --filter=blob:none, and lots more filtering options).

None of these fit the bill for us. The shape of what we need from the repository doesn't fall along any tidy lines like time or directory.

And even if it did, there's another problem: We want to push information rather than fetch it. In theory, the two directions are symmetrical, but in practice, it gets ugly fast. HTTP wasn't really built for servers to send requests to clients, and websockets add a bunch of goo and layers to manage. Plus the implementation of fetching involves running coordinated processes on both sides, and it is scary to spin up a fairly unrestricted git serving process from client code.

Git being git, there's an answer for that, git bundle: git bundle create repo.bundle main~5..main. This will create a bundle that contains only new information from the last five commits on main. You can take this bundle, which is a single file, encrypt with age, burn a CD, attach it to your carrier pigeon, and then trap and read and decrypt and apply it on the other end, assuming that your airgapped machine has the rest of the history already. Git bundle is also good for making a simple, clean, complete, single-file backup of a git repository.

We could use this to grab only the commits in the part of the git graph involved in the merge/rebase. (From here out, I'll just say merge. The same ideas apply to rebase.)

Unfortunately, this is too little information. Suppose that you edit foo.txt, and it is the source of a merge conflict. It is very useful to know the original contents of foo.txt. But git bundle assumes that you're doing an incremental backup, and you already have the complete state of the world prior to the conflict. We don't.

Also, git bundle reasonably assumes that you're using it as intended, and it refuses to apply bundles to repositories unless you can supply the history that the bundle lacks. This is good! But not for us.

Doing it the hard way

Git being git, there is always a way. We just need to step outside the well-trodden, well-supported paths and get our hands dirty.

Let's go back to first principles. What do we really want?

• We want the commit metadata (messages, graph structure) of the commits in the merge.
• We want all the files that changed in those commits, including before and after versions. (Not just the files with conflicts! The other changes are important for establishing context and intent.)
• We want all that in a form we can actually use on the server.

Let's do that now.

Everything we want here is a git object: commits, trees, and blobs. (merde doesn't yet support merging submodules.) So let's start by building a complete list of the git objects we want.

First, we need to identify the commits we are interested in. There are three inputs here: the main branch, the topic branch, and their merge base, which is the shared ancestor from which they diverged. (merde doesn't support octopus merges.)

We can get this with:

git rev-list main topic --not base

(This excludes base, but we want base. Embarrassingly, I fussed around a bunch with adding base via the git command, and adjusting my parsing, until I remembered I could just...add base directly in my code. White line fever strikes again!)

Now we have a list of commits. In order to select the relevant repository contents, we need to find the trees for those commits. These trees contain references to other trees (subdirectories) and to the blobs for the files.

There's a git for that. For every commit SHA in our list, append ^{tree} (so it looks like 1a34fe51^{tree}), and then pipe that through:

git cat-file --buffer --batch-check=%(objectname)

This bulk-extracts the trees for our commits. Note that if one of the commits was a reversion of a previous commit, such that the entire repo contents were in the same state as before, we might end up with some duplicate trees. (Content addressability ftw!) That's OK.

Now we're in a position to figure out which directories and files we should collect. We iterate over all the trees and run:

git ls-tree -r -t

This is like a filesystem walk for a git tree. It prints all objects (subtrees/blobs) and their associated path, as reachable from the root tree. What we're looking for is paths that have different objects when viewed across all trees. If any given path has the same object across all trees, then it hasn't changed, so it won't be part of any diff or any conflict, so we can ignore it.

We have now identified all the objects we care about! It's the list of commits we generated earlier, each of their trees, and all the trees/objects corresponding to paths whose contents vary.

Lastly, we need to collect all of this into a tidy parcel. More git! We pipe the list of the SHAs of all of these objects to:

git pack-objects --stdout

This will make a packfile (of course). Packfiles are a single, compressed file containing a set of git objects. A perfect packet of bytes to send to the server.

The code for everything we described above is available on GitHub.

Potemkin repositories

Of course, now the server has to actually do something with this packfile. git unpack-objects will unpack all the objects into a repo, but the instant you try to do anything with them, git will become very unhappy very quickly. There are lots of missing objects! This is the situation that git bundle was protecting us from.

The next step is to turn this unwieldy, ragged, ripped corner of a git repository into a fully-formed repository that we can use. We do that by erecting facades everywhere.

git fsck helps you diagnose problems in your repository. We have plenty of those! And we are going to lie in order to fix them.

There's a lot of bookkeeping in this next part, so we'll gloss over some of the gory details and stay focused on the high level approach.

git fsck gives us a list of all the blobs, trees, and commits that are broken or missing. (Somewhat reluctantly. But we ask oh so nicely.) We now walk through and repair them, from the leaves inward.

Start with blobs. A missing blob corresponds to a file that the client has, but we don't. That's fine. We will invent a file to put there, and hash it. Everywhere we see this client blob hash, we will swap in the new server blob hash.

What about trees? We need to rewrite all the trees, recursively. The trees we received refer to client blob hashes, but that's no good. For every tree, do the blob hash swap as needed and create a new tree. And now this generates a new hash swap we need to do: everywhere we see this client tree hash, we will swap in the new server tree hash.

Having set up fake trees, we need to fix the commits. We need to do the tree hash swap, which generates new commit hashes. That's no good, because commits refer to their parent hashes, but those are client commit hashes. We also have to track client commit hashes and their corresponding server commit hashes.

At the end of all of this, we have a fully functioning, legit potemkin git repo. You can check out commits, diff, merge, revert, branch, cavort, and whirl. But of course, lots of the files are mysteriously…devoid of useful content. That smells like success.

But we're not done yet.

Once we've completed the hard work of LLM-assisted merging (that was the whole point!), we need to send the results back to the client. git pack-objects and git unpack-objects looks like the right tool for the job again. We can calculate what objects the server has that the client needs, much like we did on the client initially.

There's only one hitch: all of the hashes will be wrong! Our new merge commit hash is wrong, because its parent is a server commit hash, not a client commit hash. Our new merge commit's tree hash is wrong, because its subtrees are server commit tree hashes, not client tree hashes.

We need to undo all of the client/server hash translation we did on the way in. This creates a bunch of objects that the client will understand (but the server doesn't!). We then make a packfile from those and return it to the client, which adds them to the repository, none the wiser of the shenanigans involved in creating those objects. And it makes a new branch pointing to the merge commit, so that there's an easy way to refer to it as a human.

Home safe

Adding objects to a repository doesn't alter the working directory or the index. Aside from the extra disk space used (which would eventually get garbage collected), there's no downside, no risk that we accidentally trample your data.

This makes using merde very low risk. The worst case scenario is you end up with some extra stuff in the repository's object store which disappears in a month or so. Or merde fails to complete the merge. That happens. Success is better than failure, but failure is better than generating garbage merges. Of course, garbage merges still get through sometimes. That's LLMs; that's software; that's life.

The best case scenario, of course, is that you are happily oblivious to the fact that a server somewhere just lied through its teeth to git and that an LLM did a bunch of token burbling, you forget you had a conflict, and you just…get back to whatever you really wanted to be doing in the first place.

Automation is at its best when doing jobs that are tedious, dangerous, or awful.

Handling git conflicts is awful. And tedious. So we set out to automate it.

Harder than it looks

This sounds like a perfect problem for an LLM. Text in, text out, plenty of context and structure.

We started with the obvious approach: Attempt the merge. (Or the rebase. I’ll just say merge from here on.) Then feed the conflicts to an LLM and ask it to resolve them.

This…doesn’t work very well. On a benchmark we built ourselves from real world conflicts, it was scoring single digits.

This is unsurprising. When I’m manually resolving a conflict, I don’t just look at the conflicted area. I go digging. So we taught the LLM to dig, through prompt engineering and plain ol’ engineering.

It took a while, but we’re pushing fifty-fifty on the benchmark now, with more tricks waiting in the wings. (If solving merge conflicts so others don’t have to sounds like heaps of fun, and you’re in the Bay Area, let’s talk.) And more importantly, when it doesn’t work, the conflict resolution mostly fails rather than producing merges that are worse than a human would.

Hiding in here is a lesson. LLMs are remarkably powerful, but they take real work to integrate. Getting good results from them requires engineering effort and investment.

An aside for folks who like geeking out about specific LLMs. At least for the way we’re approaching the problem, merge quality turns out to be heavily dependent on model quality. Claude and DeepSeek V3 are roughly tied (we use Claude), followed by 4o. The “slow thinker” models like o1 and R1 don’t provide any improvement. After 4o, there’s a big gap, and then the rest of the field.

When trying smaller models we ran into a familiar phenomenon: smaller models have quirks that greatly increase the effort it takes to use them. For example, Llama 3.3 70b was worse than Llama 3.1 70b because it has a tendency to add whitespace at the end of lines of code. This might be fine when generating code for an editor that might then strip that whitespace off; it’s less fine when you are attempting to diligently and faithfully merge code in which trailing whitespace might be meaningful. (Programming languages with multi-line string literals can have significant whitespace at the end of a line.) The effort involved in working around this model weakness is significant. A quick and dirty regex cleanup would superficially improve benchmark performance. But when that same regex is applied to a better model that doesn’t play fast and loose with whitespace, it will do nothing but hurt. To borrow a line from Bertrand Russell: benchmark hacking has all the benefits of theft over honest toil. Better not to play that game.

Considered dangerous?

Git’s standard line is that humans should do conflict resolution whenever there is any doubt. Tools should not be clever, as that might hide potential issues. Put succinctly by the Zen of Python:

> In the face of ambiguity, refuse the temptation to guess.

This is a harm reduction strategy. (Also a CYA strategy.) AI-resolved conflicts will contain mistakes. Some of those mistakes will be bad. Have we just traded some tedious and awful for an extra helping of dangerous?

Any time you’re considering a policy, it’s worth having a clear-eyed view of the alternatives. The alternatives aren’t “use AI” vs “don’t use AI”. The alternatives are “use AI, which a human can review” vs “leave it all to the humans”.

In practice, many humans do a mediocre job of conflict resolution. It is tedious and awful! Furthermore, many humans faced with a merge conflict choose to bail instead. People manually re-create branches or abandon work. It would be less effort to review and test the autogenerated resolution.

And of course, using the tool is optional. All engineering is about trade-offs. Now that we have high quality LLMs, and a way to apply them to the task, that will sometimes be the best choice. (And of course, there are the standard arguments about specialization of labor: We can fine tune the git-LLM bus to a level that copy/paste into a browser window will never reach.)

But also, yes. Be responsible. Don’t just use what the tool spits out, the same as when using AI codegen and when copy/pasting from Stack Overflow.

Design considerations

Working in git often feels like catching a boomerang with your teeth. (And to be clear, I like git.)

We wanted trying merde.ai to be low stakes, so you could just play with it, without fear.

Unlike git merge and git rebase, which do things to your working directory, merde merge and merde rebase create new branches for you, without touching your working directory or current branch at all. If you like the result, you can switch to that branch, rename it something useful, and start using it. Your old branch will still be there. If you don’t like the result, delete the new branch, and it is as if nothing ever happened.

So if you are attempting a merge, and you hit a conflict, you don’t ask merde to take over. Instead, you bail on the merge, or finish it poorly, or do whatever you like. And then you ask merde to take a crack at it from scratch. And then you assess the results, modify them, use them, or throw them away. Let the boomerang fall where it may.

Try it out

Head over to merde.ai to create an account, grab a token, and download the client. When you invoke it, it’ll do some local analysis, collate just enough of your repository for the server to do its work (roughly correlated to the size of the merge, but that’s a topic for another blog post), ship it off to the server, wait, and create a local branch with the results.

There’s also a far less powerful GitHub bot called @merde-bot. If you have an open public PR that has conflicts, write @merde-bot in a comment in the PR, and it’ll (probably) open a PR on your PR that resolves those conflicts. You can then review that PR and merge it as desired. This does make for a somewhat ungainly git history…but if you don’t care, and you just want the darn thing to merge, ask the bot. (And then be patient. It can take tens of minutes, depending on system load.)

Feedback

This is all pretty new. We’d love feedback, via a GitHub issue or over email. Tell us what works, tell us what doesn’t, tell us your hopes and dreams, tell us you want us to take your money.

This document is a summary of my personal experiences using generative models while programming over the past year. It has not been a passive process. I have intentionally sought ways to use LLMs while programming to learn about them. The result has been that I now regularly use LLMs while working and I consider their benefits net-positive on my productivity. (My attempts to go back to programming without them are unpleasant.)

Along the way I have found oft-repeated steps that can be automated, and a few of us are working on building those into a tool specifically for Go programming: sketch.dev. It’s very early but so far the experience has been positive.

Background

I am typically curious about new technology. It took very little experimentation with LLMs for me to want to see if I could extract practical value. There is an allure to a technology that can (at least some of the time) craft sophisticated responses to challenging questions. It is even more exciting to watch a computer attempt to write a piece of a program as requested, and make solid progress.

The only technological shift I have experienced that feels similar to me happened in 1995, when we first configured our LAN with a usable default route. We replaced the shared computer in the other room running Trumpet Winsock with a machine that could route a dialup connection, and all at once I had The Internet on tap. Having the internet all the time was astonishing, and felt like the future. Probably far more to me in that moment than to many who had been on the internet longer at universities, because I was immediately dropped into high internet technology: web browsers, JPEGs, and millions of people. Access to a powerful LLM feels like that.

So I followed this curiosity, to see if a tool that can generate something mostly not wrong most of the time could be a net benefit in my daily work. The answer appears to be yes, generative models are useful for me when I program. It has not been easy to get to this point. My underlying fascination with the new technology is the only way I have managed to figure it out, so I am sympathetic when other engineers claim LLMs are “useless.” But as I have been asked more than once how I can possibly use them effectively, this post is my attempt to describe what I have found so far.

Overview

There are three ways I use LLMs in my day-to-day programming:

1. Autocomplete. This makes me more productive by doing a lot of the more-obvious typing for me. It turns out the current state of the art can be improved on here, but that’s a conversation for another day. Even the standard products you can get off the shelf are better for me than nothing. I convinced myself of that by trying to give them up. I could not go a week without getting frustrated by how much mundane typing I had to do before having a FIM model. This is the place to experiment first.
   
2. Search. If I have a question about a complex environment, say “how do I make a button transparent in CSS” I will get a far better answer asking any consumer-based LLM, o1, sonnet 3.5, etc, than I do using an old fashioned web search engine and trying to parse the details out of whatever page I land on. (Sometimes the LLM is wrong. So are people. The other day I put my shoe on my head and asked my two year old what she thought of my hat. She dealt with it and gave me a proper scolding. I can deal with LLMs being wrong sometimes too.)
   
3. Chat-driven programming. This is the hardest of the three. This is where I get the most value of LLMs, but also the one that bothers me the most. It involves learning a lot and adjusting how you program, and on principle I don’t like that. It requires at least as much messing about to get value out of LLM chat as it does to learn to use a slide rule, with the added annoyance that it is a non-deterministic service that is regularly changing its behavior and user interface. Indeed, the long-term goal in my work is to replace the need for chat-driven programming, to bring the power of these models to a developer in a way that is not so off-putting. But as of now I am dedicated to approaching the problem incrementally, which means figuring out how to do best with what we have and improve it.

As this is about the practice of programming, this has been a fundamentally qualitative process that is hard to write about with quantitative rigor. The closest I will get to data is to say: it appears from my records that for every two hours of programming I do now, I accept more than 10 autocomplete suggestions, use LLM for a search-like task once, and program in a chat session once.

The rest of this is about extracting value from chat-driven programming.

Why use chat at all?

Let me try to motivate this for the skeptical. A lot of the value I personally get out of chat-driven programming is I reach a point in the day when I know what needs to be written, I can describe it, but I don’t have the energy to create a new file, start typing, then start looking up the libraries I need. (I’m an early-morning person, so this is usually any time after 11am for me, though it can also be any time I context-switch into a different language/framework/etc.) LLMs perform that service for me in programming. They give me a first draft, with some good ideas, with several of the dependencies I need, and often some mistakes. Often, I find fixing those mistakes is a lot easier than starting from scratch.

This means chat-based programming may not be for you. I am doing a particular kind of programming, product development, which could be roughly described as trying to bring programs to a user through a robust interface. That means I am building a lot, throwing away a lot, and bouncing around between environments. Some days I mostly write typescript, some days mostly Go. I spent a week in a C++ codebase last month exploring an idea, and just had an opportunity to learn the HTTP server-side events format. I am all over the place, constantly forgetting and relearning. If you spend more time proving your optimization of a cryptographic algorithm is not vulnerable to timing attacks than you do writing the code, I don’t think any of my observations here are going to be useful to you.

Chat-based LLMs do best with exam-style questions

Give an LLM a specific objective and all the background material it needs so it can craft a well-contained code review packet and expect it to adjust as you question it. There are two major elements to this:

1. Avoid creating a situation with so much complexity and ambiguity that the LLM gets confused and produces bad results. This is why I have had little success with chat inside my IDE. My workspace is often messy, the repository I am working on is by default too large, it is filled with distractions. One thing humans appear to be much better than LLMs at (as of January 2025) is not getting distracted. That is why I still use an LLM via a web browser, because I want a blank slate on which to craft a well-contained request.
   
2. Ask for work that is easy to verify. Your job as a programmer using an LLM is to read the code it produces, think about it, and decide if the work is good. You can ask an LLM to do things you would never ask a human to do. “Rewrite all of your new tests introducing an <intermediate concept designed to make the tests easier to read>” is an appalling thing to ask a human, you’re going to have days of tense back-and-forth about whether the cost of the work is worth the benefit. An LLM will do it in 60 seconds and not make you fight to get it done. Take advantage of the fact that redoing work is extremely cheap.

The ideal task for an LLM is one where it needs to use a lot of common libraries (more than a human can remember, so it is doing a lot of small-scale research for you), working to an interface you designed or produces a small interface you can verify as sensible quickly, and it can write readable tests. Sometimes this means choosing the library for it, if you want something obscure (though with open source code LLMs are quite good at this).

You always need to pass an LLM’s code through a compiler and run the tests before spending time reading it. They all produce code that doesn’t compile sometimes. (Always making errors I find surprisingly human, every time I see one I think, there but for the grace of God go I.) The better LLMs are very good at recovering from their mistakes, often all they need is for you to paste the compiler error or test failure into the chat and they fix the code.

Extra code structure is much cheaper

There are vague tradeoffs we make every day around the cost of writing, the cost of reading, and the cost of refactoring code. Let’s take Go package boundaries as an example. The standard library has a package “net/http” that contains some fundamental types for dealing with wire format encoding, MIME types, etc. It contains an HTTP client, and an HTTP server. Should it be one package, or several? Reasonable people can disagree! So much so, I do not know if there is a correct answer today. What we have works, after 15 years of use it is still not clear to me that some other package arrangement would work better.

Advantages of a larger package include: centralized documentation for callers, easier initial writing, easier refactoring, easier sharing of helper code without devising robust interfaces for them (which often involves pulling the fundamental types of a package out into yet another leaf package filled with types). The disadvantages include the package being harder to read because many different things are going on (try reading the net/http client implementation without tripping up and finding yourself in the server code for a few minutes), or it being harder to use because there is too much going on in it. For example I have a codebase that uses a C library in some fundamental types, but parts of the codebase need to be in a binary widely distributed to many platforms that does not technically need the C library, so have more packages than you might expect in the codebase isolating the use of the C library to avoid cgo in the multi-platform binary.

There are no right answers here, instead we are trading off different types of work that an engineer will have to do (upfront and ongoing). LLMs influence those tradeoffs:

• As LLMs do better with exam-style questions, more and smaller packages make it easier to give a complete and yet isolated context for a piece of work. This is true for humans too, which is why we use packages at all, but we trade off package size against the extra typing/plumbing/filing to make more readable code. With an LLM both doing and benefiting from a big chunk of that extra work, the tradeoff shifts. (As a bonus, we humans get more readable code!)
  
• Smaller and more numerous packages can be compiled and tested independently of unrelated code. This is helpful for the LLM dev cycle as it means changes can be compiled and tested before other dependent packages are refactored, and the package has a simpler test environment.

An example

Let me work an example to combine a few of the discussed ideas:

Write a reservoir sampler for the quartiles of floats.

First off, package structure. Were I doing this before LLMs, I would have chosen to have some sort of streamstat package that contained several algorithms, maybe one per file. This does not seem to be a unique opinion, here is an open source quantile package following that model. Now, I want just this one algorithm in its own package. Other variants or related algorithms can have their own package.

Next up, what do we get from an LLM. The first pass is not bad. That prompt, with some details about wanting it in Go got me quartile_sampler.go:

// QuartileSampler maintains a reservoir sample to estimate quartiles of a stream  
// of float64 values. It uses reservoir sampling to maintain a representative  
// sample when processing large streams of data.  
type QuartileSampler struct {  
   reservoir []float64  
   seen      int  
   maxSize   int  
   rng       *rand.Rand  
}

// NewQuartileSampler creates a new sampler that maintains up to maxSize samples.  
// A larger maxSize provides more accurate estimates but uses more memory.  
func NewQuartileSampler(maxSize int) *QuartileSampler { … }

The core interface is good too:

// Add adds a new value to the sampler.  
func (qs *QuartileSampler) Add(value float64) { … }

// Quartiles returns the estimated 25th, 50th (median), and 75th percentiles  
// of the data seen so far. Returns (0,0,0) if no data has been added.  
func (qs *QuartileSampler) Quartiles() (q1, median, q3 float64) {  
   if len(qs.reservoir) == 0 {  
       return 0, 0, 0  
   }  
    
   // Make a copy to avoid modifying the reservoir  
   sorted := make([]float64, len(qs.reservoir))  
   copy(sorted, qs.reservoir)  
   slices.Sort(sorted)  
    
   n := len(sorted)  
   q1 = percentile(sorted, 0.25)  
   median = percentile(sorted, 0.50)  
   q3 = percentile(sorted, 0.75)  
    
   return q1, median, q3  
}

Great! There are also tests.

An aside: this may be the place to stop. Sometimes I use LLM codegen as a form of specialized search. E.g. I’m curious about reservoir sampling, but want to see how the algorithm would be applied under some surprising constraint, for example time-windowed sampling. Instead of doing a literature search I might amend my prompt for an implementation that tracks freshness. (I could also ask it to include references to the literature in the comments, which I could manually check to see if it’s making things up or if there’s some solid research to work from.) Often I spend 60 seconds reading some generated code, see an obvious trick I hadn’t thought of, then throw it away and start over. Now I know the trick is possible. This is why it is so hard to attribute value generated by LLMs. Yes sometimes it makes bad code, gets stuck in a rut, makes up something impossible (it hallucinated a part of the monaco API I wish existed the other day) and wastes my time. It can also save me hours by pointing out something relevant I don’t know.

Back to the code. Fascinatingly, the initial code produced didn’t compile. In the middle of the Quartiles implementation there was the line:

n := len(sorted)

Which is a fine line, sorted is a slice defined a few lines earlier. But the value is never used so gopls (and the Go compiler if you run go build) immediately says:

declared and not used: n

This is a very easy fix. If I paste the error back into the LLM it will correct it. Though in this case, as I’m reading the code, it’s quite clear to me that I can just delete the line myself, so I do.

Now the tests. I got what I expected. In quartile_sampler_test.go:

func TestQuartileSampler(t *testing.T) {  
   tests := []struct {  
       name     string  
       data     []float64  
       wantQ1   float64  
       wantMed  float64  
       wantQ3   float64  
       epsilon  float64 // allowed error for floating point comparison  
   }{  
       {  
           name:    "basic sequence",  
           data:    []float64{1, 2, 3, 4, 5, 6, 7},  
           wantQ1:  2,  
           wantMed: 4,  
           wantQ3:  6,  
           epsilon: 0.1,  
       },  
    …  
   }

   for _, tt := range tests {  
       t.Run(tt.name, func(t *testing.T) {  
           qs := NewQuartileSampler(100) // Use large reservoir for accurate testing  
           for _, v := range tt.data {  
               qs.Add(v)  
           }

           q1, med, q3 := qs.Quartiles()  
            
           if math.Abs(q1-tt.wantQ1) > tt.epsilon {  
               t.Errorf("Q1 = %v, want %v (±%v)", q1, tt.wantQ1, tt.epsilon)  
           }  
           if math.Abs(med-tt.wantMed) > tt.epsilon {  
               t.Errorf("Median = %v, want %v (±%v)", med, tt.wantMed, tt.epsilon)  
           }  
           if math.Abs(q3-tt.wantQ3) > tt.epsilon {  
               t.Errorf("Q3 = %v, want %v (±%v)", q3, tt.wantQ3, tt.epsilon)  
           }  
       })  
   }
}

Exactly the sort of thing I would write! I would run some cases through another implementation to generate expected outputs and copy them into a test like this. But there are two issues with this.

The first is the LLM did not run these numbers through another implementation. (To the best of my knowledge. When using a sophisticated LLM service, it is hard to say for sure what is happening behind the scenes.) It made them up, and LLMs have a reputation for being weak at arithmetic. So this sort of test, while reasonable for a human to write because we base it on the output of another tool, or if we are particularly old-school do some arithmetic ourselves, is not great from an LLM.

The second issue with this is we can do better. I am happy we now live in a time when programmers write their own tests, but we do not hold ourselves to the same standards with tests as we do with production code. That is a reasonable tradeoff, there are only so many hours in the day. But what LLMs lack in arithmetical prowess, they make up for in enthusiasm.

Let’s ask for an even better test.

In the tests, implement the simplest, most readable version of the standard code for quartiles over a fixed set of known values in a slice. Then pass the test cases through the standard code and the reservoir sampler and confirm they are within an epsilon of each other. Structure the comparison code such that it can be used in a fuzz test too.

This got us some new test code:

// referenceQuartiles calculates the exact quartiles for a slice of float64 values  
// using linear interpolation, matching the behavior expected from the sampler.  
func referenceQuartiles(data []float64) (q1, median, q3 float64) { … }

// compareQuartiles checks if two sets of quartiles are within epsilon of each other.  
// Returns true if they match within the tolerance, false otherwise.  
func compareQuartiles(q1a, meda, q3a, q1b, medb, q3b, epsilon float64) bool { … }

// checkQuartiles is a test helper that compares sampler output against the reference  
// implementation and reports any differences.  
func checkQuartiles(t *testing.T, data []float64, epsilon float64) {  
   t.Helper()  
    
   // Get reference values  
   wantQ1, wantMed, wantQ3 := referenceQuartiles(data)  
    
   // Get sampler values using a large reservoir for accuracy  
   qs := NewQuartileSampler(1000)  
   for _, v := range data {  
       qs.Add(v)  
   }  
   gotQ1, gotMed, gotQ3 := qs.Quartiles()  
    
   if !compareQuartiles(gotQ1, gotMed, gotQ3, wantQ1, wantMed, wantQ3, epsilon) {  
       t.Errorf("Quartiles mismatch:\ngot  (q1=%v, med=%v, q3=%v)\nwant (q1=%v, med=%v, q3=%v)\nepsilon=%v",  
           gotQ1, gotMed, gotQ3, wantQ1, wantMed, wantQ3, epsilon)  
   }  
}

The original test from above has been reworked to to use checkQuartiles and we have something new:

func FuzzQuartileSampler(f *testing.F) {  
   // Add some seed corpus  
   f.Add([]float64{1, 2, 3, 4, 5})  
    
   f.Fuzz(func(t *testing.T, data []float64) {  
       // Use a larger epsilon for fuzzing since we might get more extreme values  
       checkQuartiles(t, data, 0.2)  
   })  
}

This is fun, because it’s wrong. My running gopls tool immediately says:

fuzzing arguments can only have the following types:
    string, bool, float32, float64,
    int, int8, int16, int32, int64,
    uint, uint8, uint16, uint32, uint64,
    []byte

Pasting that error back into the LLM gets it to regenerate the fuzz test such that it is built around a func(t *testing.T, data []byte) function that uses math.Float64frombits to extract floats from the data slice. Interactions like this point us towards automating the feedback from tools: all it needed was the obvious error message to make solid progress towards something useful. I was not needed.

Doing a quick survey of the last few weeks of my LLM chat history shows (which as I mentioned earlier, is not a proper quantitative analysis by any measure) that more than 80% of the time there is a tooling error, the LLM can make useful progress without me adding any insight. About half the time it can completely resolve the issue without me saying anything of note, I am just acting as the messenger.

Where are we going? Better tests, maybe even less DRY

There was a programming movement some 25 years ago focused around the principle “don’t repeat yourself.” As is so often the case with short snappy principles taught to undergrads, it got taken too far. There is a lot of cost associated with abstracting out a piece of code so it can be reused, it requires creating intermediate abstractions that must be learned, and it requires adding features to the factored out code to make it maximally useful to the maximum number of people, which means we depend on libraries filled with useless distracting features.

The past 10-15 years has seen a far more tempered approach to writing code, with many programmers understanding it is better to reimplement a concept if the cost of sharing the implementation is higher than the cost of implementing and maintaining separate code. It is far less common for me to write on a code review “this isn’t worth it, separate the implementations.” (Which is fortunate, because people really don’t want to hear things like that after they have done all the work.) Programmers are getting better at tradeoffs.

What we have now is a world where the tradeoffs have shifted. It is now easier to write more comprehensive tests. You can have the LLM write the fuzz test implementation you want but didn’t have the hours to build properly. You can spend a lot more time writing tests to be readable, because the LLM is not sitting there constantly thinking “it would be better for the company if I went and picked another bug off the issue tracker than doing this.” So the tradeoff shifts in favor of having more specialized implementations.

The place where I expect this to be most visible is language-specific REST API wrappers. Every major company API comes with dozens of these, usually low quality, wrappers written by people who aren’t actually using their implementations for a specific goal, instead are trying to capture every nook and cranny of an API in a large and complex interface. Even when it is done well, I have found it easier to go to the REST documentation (usually a set of curl commands), and implement a language wrapper for the 1% of the API I actually care about. It cuts down the amount of the API I need to learn upfront, and it cuts down how much future programmers (myself) reading the code need to understand.

For example, as part of my recent work on sketch.dev I implemented a Gemini API wrapper in Go. Even though the official wrapper in Go has been carefully handcrafted by people who know the language well and clearly care, there is a lot to read to understand it:

$ go doc -all genai | wc -l  
    1155

My simplistic initial wrapper was 200 lines of code total, one method, three types. Reading the entire implementation is 20% of the work of reading the documentation of the official package, and if you decide to try digging into its implementation you will discover that it is a wrapper around another largely code-generated implementation with protos and grpc and the works. All I want is to cURL and parse a JSON object.

There obviously comes a point in a project, where Gemini is the foundation of the entire app, where nearly every feature is used, where building on gRPC aligns well with the telemetry system elsewhere in your organization, where you should use the large official wrapper. But most of the time it is so much more time consuming, both upfront and ongoing, to do so given we almost always want only some wafer-thin sliver of whatever API we need to use today, that custom clients, largely written by a GPU, are far more effective for getting work done.

So I foresee a world with far more specialized code, with fewer generalized packages, and more readable tests. Reusable code will continue to thrive around small robust interfaces and otherwise will be pulled apart into specialized code. Depending how well this is done, it will lead to either better software or worse software. I would expect both, with a long-term trend towards better software by the metrics that matter.

Automating these observations: sketch.dev

As a programmer my instinct is to make computers do work for me. It is a lot of work getting value out of LLMs, how can a computer do it?

I believe the key to solving a problem is not to overgeneralize. Solve a particular problem and then expand slowly. So instead of building a general-purpose UI for chat programming that is just as good at COBOL as it is for Haskell, we want to focus on one particular environment. The bulk of my programming is in Go, and so what I want is easy to imagine for a Go programmer:

• something like the Go playground, built around editing a package and tests
  
• with a chat interface onto editable code
  
• a little UNIX env where we can run go get and go test
  
• goimports integration
  
• gopls integration
  
• automatic model feedback: on model edit run go get, go build, go test, feedback missing packages, compiler errors, test failures to the model to try and get them fixed automatically

A few of us have built an early prototype of this: sketch.dev.

The goal is not a “Web IDE” but rather to challenge the notion that chat-based programming even belongs in what is traditionally called an IDE. IDEs are collections of tools arranged for people. It is a delicate environment where I know what is going on. I do not want an LLM spewing its first draft all over my current branch. While an LLM is ultimately a developer tool, it is one that needs its own IDE to get the feedback it needs to operate effectively.

Put another way: we didn’t embed goimports into sketch for it to be used by humans, but to get Go code closer to compiling using automatic signals, so that the compiler can provide better error feedback to the LLM driving it. It might be better to think of sketch.dev as a “Go IDE for LLMs”.

This is all very recent work with a lot left to do, e.g. git integration so we can load existing packages for editing and drop the results on a branch. Better test feedback. More console control. (If the answer is to run sed, run sed. Be you the human or the LLM.) We are still exploring, but are convinced that focusing an environment for a particular kind of programming will give us better results than the generalized tool.