Note1

Here is a thing nobody else in the east quarter seems to have noticed, and Tham has tried to work out why, and his best theory is that it isn’t the adding. Everyone can add. It’s that nobody around here sets one true thing next to another on purpose. Facts are for having, not for knocking together, and there isn’t a word on the board for doing it anyway.

Corn is three silvers a bushel. Meal is five a sack, one sack to the bushel. The miller’s labor is Announced at one. Three plus one is four. Meal is five. There is a silver sitting between those numbers, per bushel, per day, and it has Tham’s name on it because he is the one who bothered to add.

“You’re doing the thing with your face again,” Lahra said. She ran the stall next to where he did his counting — tawny and unbothered, and she’d been making correct change without looking since before he could walk.

“I’m doing arithmetic.”

“You’re doing arithmetic at people. It’s different.” She handed a customer two blue apples and three coppers back from a silver, exact. “Go on then. How much today.”

“Eight bushels through the mill, eight silvers clear. Same as yesterday. The board doesn’t change, Lahra. That’s the whole point of the board.”

“Corn was four, once.”

“No it wasn’t.”

She shrugged, the way she did at customers who argued about weight. “Eight a day. I bank twelve and I don’t have to walk to the mill for it. What lands, let go, Tham.” She said it the way you said any recitation, and went back to her apples.

He went back to knocking the numbers together. He’d been doing it since he was six. He’d never once knocked one loose. That wasn’t what knocking did. It told you where the next one had to be.

He took his eight silvers down past the south field, where the straw man worked. He’d been told, by a Mouth two Mouths back, that he hadn’t any thoughts in him, and ever since he’d been the most relaxing person in the quarter to talk to.

“Morning,” Tham said.

“Morning!” The straw man leaned on his hoe. His rows were perfect. “Lovely one. Though I wouldn’t know.”

“You wouldn’t know it’s lovely?”

“Haven’t got the equipment to judge.” He tapped the side of his head, and the sound was dry. “I just say what the day looks like it wants said. Works out fine.”

It did work out fine. He was the best farmer for six miles, which was a lot of judgment for no thoughts.

Past his field the Road started — yellow brick, the one Announcement they’d laid in stone instead of chalk, older than the screening. Tham had never walked it more than a quarter mile. There were poppies further on, red fields where the Road went soft and people who walked too far sat down and didn’t get up again. Not dead. Paused. The Reassurance came and collected them eventually. He had no business in the city the post couldn’t carry, and the post came daily — three winged carriers, a slip for the Mouth’s hand by noon.

He stopped at the confirmation hall on the way back and put two silvers on corn holds at three. Nine-to-eight, like everything. The clerk wrote his name next to the line.

He was crossing the square when the sky made a sound it had not been Announced as able to make.

A long falling whistle, and then a house in the square. Timber frame, shake roof, glass in the windows — flat glass, untinted, the kind nobody wore — sitting at a slight angle across what had been the Mouth’s front porch, where She took the morning air, and where She had been taking it.

The silver shoes stuck out from under the boards. Toes up. They were the only thing about Her that never took the tint of the lenses — actual silver, mirror-bright, the same color glassed or bare-eyed. The color of nothing at all.

Nobody screamed. Old Pell from the granary looked at it for ten seconds, then walked to the blue post on the corner and pulled the cord.

“Keepers’ll sort it,” Lahra said, which wasn’t what anyone called them anymore. She came out of her stall and stood next to Tham and after that didn’t say anything, which for Lahra was a lot.

The Reassurance arrived inside the quarter hour. Two of them, grey robes, bare eyes — the only unglassed faces in the quarter, and you didn’t look straight at those. The taller one knelt by the silver shoes, didn’t touch them, then rose and went around the far side of the house and out of sight. The shorter one opened a ledger and started talking in the voice they used.

“A house,” he said. “Dwelling-structure. Every dwelling-structure rests upon the earth. This one rests upon the earth.” He waited. “It rests more recently than most. That’s a permitted value of resting.” He waited again, the same length, his eyes on the flat glass in the windows a beat longer than the rhythm wanted. “The Mouth speaks the word of the city. The word is on the board. The board may be read. A new Mouth will be Announced.” He looked up. “The board may be read. If anyone has a thought that hasn’t found where it goes yet, come and say it to me and we’ll find where it goes.” He glanced at the ledger. “Pell.”

“Fine.”

“Lahra.”

She nodded once and went back inside her stall.

“Thamkel.”

“Tham’s fine,” Tham said. “I’m good.”

The grey man made a mark. People drifted toward the board and read it and their shoulders came down. Corn was three silvers. The day was the day it had been.

The shoes stayed where they were, toes up, and nobody went near them.

Except Tham was standing there setting true things next to each other.

The Mouth speaks for the city. The Mouth is under a house.

He looked across the square. The board was readable from here — corn three, meal five, the day’s word in chalk in Her hand from this morning. Still true.

The house itself he could nearly slot. Things flew — the carriers flew, three a day, something with a slip in its grip. Something could have a house in its grip. That was just size.

The whistle, though. The whistle was a fall. The grey man had said rests and rests and never once fell, and Tham had watched a whole square’s shoulders come down on the noun and walk right past the verb.

The carriers bring tomorrow’s word. He looked up — sky empty, as it would be till noon. A slip, and the Mouth’s hand to take it.

The hand was under a house.

So tomorrow: a slip, and no hand. The board holds at today. Corn three, meal five, gap of one — forever, now, or until—

A new Mouth will be Announced. The grey man had said it not ten minutes ago, in the voice, and Tham’s shoulders had come down with everyone else’s.

Announced. By a Mouth. Of which there were, as of this morning, none.

He held that. Two true things and no number between them — the same shape as the mill, except the mill gap had a silver sitting in it and this one had nothing in it at all.

Two truths that don’t meet are two truths. He’d known that one since he could talk. He said it to himself now, in the cadence, and waited for his shoulders. They didn’t come down. That had never once happened before.

So nothing was wrong.

That was where it wanted to stop. He could feel the wanting — the warm closing-over, the same shape as the grey man’s voice. He had landed here a thousand times and it had always been the floor.

He turned, so he was facing south. Past Lahra’s stall, past the straw man’s perfect rows, the Road started. Yellow brick — yellow the way the shoes were silver, the same color through the glass or not. Older than the screening. The one thing the city had ever set down to outlast every Mouth there’d be.

Nothing he’d set next to anything added up to walk. He knew that. The gap was real and the Road was real and the thing that put them together and pointed south wasn’t on the board and wasn’t arithmetic, and he didn’t have a name for it.

Lahra was right beside him, looking at the board, shoulders down. He was fairly sure she could add at least as fast as he could.

He went home the long way, past the straw man, who waved. The green smudge of the city sat on the horizon like a thumbprint on the inside of the lens.

There was a grey robe on his front step. The tall one, who’d gone around the far side of the house and not come back where Tham could see. He was sitting, not standing, bare-eyed, and tired in a way Tham hadn’t known a face could be without the glass on it.

“You didn’t read the board,” the grey man said.

“I know what’s on it.”

“Yes.” A pause, the length the short one had used in the square. “There is a place under the city. The board there has more on it. Everything on it is true.” Another measured pause. “Everything on the board here is true. A larger board is a permitted value of a board.” He almost smiled. “The thing you did in the square. The part that wasn’t adding. There’s a word for it, on the board down there.” He let that sit. “dath oz. You’ll want a coat. It’s four days on the Road and the second two are cold.”

Tham felt his shoulders start to come down. A bigger board. More true things. A name for the thing he didn’t have a name for, already chalked in by someone who’d found it first — that was what was being offered, and the offer was warm, and the warmth had a shape he recognized

He looked past the grey man at the green on the horizon, and then down at his own two feet, which were not silver, and which were his.

“There’s a recitation,” he said. “The past is shut.”

“It is.” The grey man got up off the step, and it sounded less like power than like someone admitting to a thing he’d done. “We shut it.” And then, quieter, the cadence of a recitation Tham had never been given: “No Keeper hath the Keeper. Get the coat.”

Tham went in for the coat. The warmth hadn’t gone anywhere. He was walking anyway.

1. There’s this paper

Earlier last week I got nerd-sniped by a paper called Condensation: a theory of concepts (Eisenstat 2025). It’s the kind of paper where the abstract makes a claim so clean you assume you must be misreading it: roughly, there is a right answer to “what are the concepts in this data,” and any two agents who carve it up well enough will agree on what those concepts are.1

If that sounds like John Wentworth’s natural abstractions hypothesis, yes, the family resemblance is strong. I wrote about something adjacent a while back. Condensation is a different formalization, but the punchline rhymes: structure in the data constrains what any good representation can look like. People on LessWrong seem to dig it, and I wanted to see what the fuss was about.

The paper is forty pages of math and gives you no algorithm; it tells you what a good carving looks like, not how to find one. I spent a slightly embarrassing amount of time trying to get the basic objects to do something on a computer. This post is how far I got.2

2. Concepts, scopes, and a score

Say you observe three tokens3:

cat dog cat

I’ll define a concept as a piece of information about that data.4 “The topic is animals” is a concept. “Token 2 is dog, not cat” is also a concept. Concepts can come with a scope: the set of tokens a concept is about. The topic concept has scope {1,2,3}, because knowing the topic tells you something about all three tokens. The “token 2 is dog” concept has scope {2}, because it’s only about token 2.

A representation is a list of (concept, scope) pairs. With three tokens there are seven possible scopes ({1}, {2}, {3}, {1,2}, {1,3}, {2,3}, {1,2,3}); a typical representation puts something at a few of them and leaves the rest empty.5 Which scopes get concepts is the structure the theory cares about.

One rule that representations need to satisfy for us to care about them: from the concepts whose scope includes token i, you must be able to rebuild token i. No information goes missing. (If you only filed “topic = animals” and nothing else, you couldn’t rebuild cat vs dog; that representation is invalid.)

The star of this show, condensation, then gives you a score function χ that measures how efficient a representation is. It works like this: someone asks “tell me everything relevant to tokens {1,2}.” You answer by reading every concept whose scope overlaps {1,2}. In our example that’s the topic (scope {1,2,3}, overlaps), id₁ (scope {1}, overlaps), and id₂ (scope {2}, overlaps), but not id₃ (scope {3}, no overlap). χ({1,2}) is the total bits you had to read.6 You want χ small for every possible question; never read a concept that isn’t relevant, never read the same information twice.

As I said up top, the paper doesn’t tell you how to find a good representation. The main theorem (4.15) says that any two representations that score well enough will end up filing roughly the same concepts at roughly the same scopes,7 which is the sense in which concepts are “out there.”

3. The same three tokens, three representations

Let me make that concrete with a small example that still has all the structure we care about.

The data: a fair coin picks a topic, animals or tools, and then each of three tokens is independently one of two words from that topic (cat/dog or hammer/saw). There are four bits of information total: one topic bit (shared across all three tokens) and three token-identity bits (private to one token each). Here are some example draws:

Three representations of the same data:

Trivial. Don’t bother finding shared concepts. File each raw token at its own scope.

Ask about tokens 1 and 2: you read 4 bits, but the true joint information is only 3 bits because they share a topic.8 The topic bit is sitting inside both raw tokens, and you read it twice. Every multi-token question overpays.

Oracle. File the topic at {1,2,3} and each token-identity bit at its singleton.

Every question costs exactly its entropy, which is the best you can do.

Misfiled. Same four concepts as the oracle, but topic at scope {2,3} instead of {1,2,3}.9

Ask about tokens 1 and 2: full token₁ (2 bits) + topic (1 bit) + id₂ (1 bit) = 4. The topic is in there twice, and you pay for both copies.

So the score does what you’d hope: lowest when each shared concept sits at exactly the scope it’s about, and it tells you which queries overpay when one doesn’t.

We built all of these by hand, so nothing deep is happening yet. The question is whether a representation constructed from the activations of a neural network looks more like the oracle or the trivial one.

4. A model gives you concepts; scope is your problem

Now suppose the concepts come out of a language model like GPT-2 or Claude rather than being hand-built.

I trained a tiny 4-layer transformer on the three-token topic data from §3. A single weighted sum of the residual stream (the running vector that each layer reads from and writes to) recovers the topic with perfect accuracy:

So the model has learned the right concept. The question is: what scope do we assign it?

One naive answer is “the feature’s scope is all the tokens the model has seen so far,” since that’s what the activation depends on. But that gives the same answer for every feature, so it can’t distinguish a feature about one token from a feature about the whole sequence.10

A better method here is mutual information: scope = the set of tokens the feature is correlated with. The topic is correlated with all three tokens (each token’s first bit is the topic), so MI says scope {1,2,3}, which is the oracle representation from §3. The score χ goes to zero.11

At scale, the standard way to get candidate concepts out of a language model is a sparse autoencoder (SAE): you learn a large set of directions in the residual stream such that, for any given input, only a few of them are active. Each direction is called a feature, and the hope is that “feature k is on” means something interpretable: this is about cooking, there’s an open bracket, the subject is plural. An SAE gives you features, but it does not give you their scopes, so you still need a method like MI to tag each one.

5. Does it work on real models?

The three-token toy was reassuring, but it was four bits of hand-built data. The question that matters is whether the score does anything useful when the concepts come out of an actual language model on actual text. To find out, I wanted to carefully expand the domain of the experiment along two axes: model size (from a tiny 4-layer transformer up to GPT-2 small) and dataset complexity (from planted ground truth up to real text).

The pipeline

1. Take 50,000 windows of text, N tokens each.

2. Run each through a language model and read the residual stream at the last token (by then the model has seen all N).

3. Decompose those vectors three ways: an SAE, PCA (the textbook find-the-biggest-directions method), and random projections (the control: directions that mean nothing). Each gives you a few hundred features.

4. Turn each feature into a concept: pick a threshold so the feature is ON for the top r% of windows and OFF for the rest.12 Tag it with a scope by MI, the set of token positions it’s correlated with. Keep the top M most informative.

5. Compute χ. Report Δχ = χ − χₜᵣᵢᵥᵢₐₗ: how many bits better (or worse) than the trivial representation from §3 that just stores each token raw. Negative Δχ means the method found shared concepts that actually save bits.13

I sweep the threshold r in step 4 and report each method’s minimum Δχ, letting the score pick its own threshold.14 The other knobs in steps 1–4 I either swept or held fixed, and the thing I’m reporting throughout is the ordering (SAE vs PCA vs random), which survived every knob I turned.15 The aggregate number is shakier (there’s a weighting choice in step 5 that can shift it) so don’t read any single number as a constant of nature.

The datasets

A planted toy where I know the ground truth. Eight tokens, with seven planted shared concepts arranged in a binary tree: one about all eight tokens, one about tokens 1–4, one about 5–8, and one about each adjacent pair (1–2, 3–4, 5–6, 7–8). Each is a yes/no flag that’s “yes” 15% of the time. I pack them into six dimensions so they overlap a little and the SAE has to actually un-mix them.16 Some example sequences:

The oracle representation (true concepts at true scopes) scores Δχ = −3.74.

TinyStories. 50,000 eight-word openings from children’s stories. The positional structure here is real: “once upon a time there was a” lives at positions 1–7, and it’s the most common pattern by far. I ran this with both a small 4-layer transformer and GPT-2 small.

Induction. A setting designed to have one very specific shared concept. The prompt is something like

cat dog bird fish bee cat ___

where five random words are followed by a repeat of word 1, and the model should complete with word 2. GPT-2 small gets this right 75% of the time. I read the residual stream at the repeat (token 6), where the model has just recognised “that’s cat again.” The shared concept the features should pick up is “word 1 = cat,” scope {1,6}: it’s about token 1 (where cat first appears) and token 6 (where it appears again). The words come from a fixed pool of 50.

Results

On the planted toy, the SAE recovers most of the ground truth: it gets 86% of the way to the oracle score, and the threshold r at which it scores best is 15%, which is exactly the true rate the concepts were generated at. The score found the right threshold without being told it. PCA and random projections both fall well short.17

On TinyStories, the ordering holds: SAE outperforms PCA, which outperforms random. The gaps are smaller than the toy (real text has less clean shared structure than seven planted flags), but consistent across both the small model and GPT-2 small. The SAE’s top feature fires on “once upon a time there was a”: one yes/no concept that tells you something about all eight tokens at once. PCA’s top feature fires on whether the last token is a function word: a concept about one token.18

On induction, PCA wins, and not by a little.

Why PCA wins on induction

I want to dwell on this, because my prior was “SAE beats PCA” and the score disagreed. The shared concept is “which of 50 words is word 1,” not a yes/no flag but a 50-way choice worth log₂ 50 ≈ 5.6 bits. With four binary concepts to spend on encoding that choice, the SAE spends them on near-one-hots: its top features are literally “word 1 = gym,” “word 1 = pen,” “word 1 = fan,” “word 1 = cup,” each precise about 2% of inputs and silent on the other 98%. PCA spends its four concepts on coarse splits, each one ON for roughly 20% of the pool, vague about everything but covering all of it. Four coarse bits encode more of a 50-way variable than four one-hots do; the score is reporting that correctly. Shrink the pool to 8 words and PCA’s lead shrinks proportionally, confirming it’s the cardinality of the underlying variable that matters.19 20

6. How far I got, and what worries me

This is how far I got. The score does something: it distinguishes SAE from PCA from random in the right direction on controlled data, it finds ground-truth thresholds without being told, and it delivers at least one genuinely surprising result (PCA beating SAE on induction). But I want to be clear-eyed about what this is and isn’t.

In this setting, the condensation score measures whether a decomposition’s inductive bias matches the structure of the shared concepts in the data. SAEs assume shared concepts are rare yes/no flags, PCA doesn’t assume that. So when the shared concepts are rare yes/no flags (the toy, TinyStories), SAE wins and when the shared concept is a 50-way categorical (induction), PCA wins. When there’s nothing shared, or the model never computed it, neither beats trivial.

A few things I think this buys you, if it holds up:

Scope is half the concept. Interpretability mostly treats a “concept” as a direction in activation space, full stop. Condensation says it’s a (concept, scope) pair, and §4 showed that scope assignment is a real choice and there’s an opinionated score that allows us to compare choices. “This feature means X” and “this feature is about tokens i through j” are different claims, and circuits work tends to slide between them without noticing.

Feature splitting has a signature. When an SAE breaks one underlying variable into many features, those features all land at the same scope, and χ penalizes exactly that (you read k concepts where one would do). At scale, an SAE’s known split-feature families should show up as scope collisions, and a decomposition that merges them should score better. That’s a testable prediction.

You could use this to choose decomposition methods per-circuit. Different parts of the same model plausibly have different kinds of shared concepts: induction is a categorical choice since “is this Python” is a yes/no flag. The score gives you a per-circuit reason to pick the decomposition rather than committing to SAEs everywhere, which is roughly where the SAEs-are-disappointing discourse has been heading anyway. And nothing here is SAE-specific; the pipeline takes any features-from-activations method.

The theory is beautiful, and I have a lot of research ideas for how to fill in some of the implementation details:

• A principled algorithm for scope assignment would be awesome. I used MI because it worked on the toy, which is not a great justification.

• Turning concepts extracted from an SAE/from PCA into something that allows us to compute mutual information is tricky. Binarizing features throws away most of their information, and quantizing gets kind of messy.

• The number of possible scopes doubles with every token, so past a certain N you can’t check them all.21

• And I’m not confident that the pipeline choices that work at N = 8 will survive at N = 128, or that the orderings I’m seeing on 50,000 windows will hold at 500,000.

That’s a lot of open questions before this approach would be ready for crunch-time deployment.22

But the biggest gap is that I have not tested the paper’s actual theorem: that two good-scoring representations agree on what concepts they find. Everything above is “here’s a new scoring function for decomposition methods, and it gives sensible rankings.” That’s useful, but it’s an eval metric, not evidence that concepts are real. Condensation’s claim is stronger: any two representations that score well enough should converge on the same (concept, scope) pairs, and that’s what would make this about natural abstractions rather than about SAEs. Testing this could be relatively straightforward: train two SAEs with different seeds, extract their top concepts, and see whether χ agreement (Theorem 5.8) actually holds. I might do that in a follow-up, but I wanted to publish this much first, because I think more people poking at this independently is more valuable than me polishing in private.

Previously in this series: Slightly advanced decision theory 102: Four reasons not to be a (naive) utility maximizer

— Lucille Bluth, Arrested Development

Lucille Bluth is wrong about the price of a banana but she accidentally reveals her willingness to pay. A banana costs less than a dollar. Lucille would happily pay ten dollars for one1. That means every time she buys a banana for a dollar, roughly nine dollars of value silently flows into her pocket. The price feels like a fact about the banana, like its color or curvature. But it’s not — it’s one particular point on a negotiation frontier, and Lucille happens to be on the very comfortable side of it.

That dollar is consumer surplus — the gap between what the buyer would be willing to pay and what she actually pays. Intro econ presents it as a Good Thing™ (and it is) but here’s what’s been bugging me: consumer surplus is unequally distributed by construction.

The price of a banana settles at the point where some marginal customer is just barely willing to buy. That person gets essentially zero surplus. Meanwhile, someone for whom the price is a rounding error walks away with a windfall golden banana. Under uniform pricing, the richer you are, the better the deal you get2.

What would a fair price look like?

A seller has a good that costs C to produce. Buyer i would pay up to Rᵢ. If a trade happens, total surplus is Rᵢ − C. How should it be divided?

Nash bargaining gives a clean answer: split it equally3. Each side gets (Rᵢ − C)/2, so the fair price for person i is

This is the formal version of what Yudkowsky calls a cheerful price4 — the number at which neither side is merely tolerating the deal but both walk away actually glad it happened.

Technically this is price discrimination (which is almost universally condemned) — person i pays a different price than person j. But it’s price discrimination with a fairness constraint: every trade splits the gains 50/50. Nobody is the sucker5.

Compare to uniform pricing, where the seller picks one P* and everyone with Rᵢ ≥ P* buys. The seller gets (P* − C) per sale. Buyers get anywhere from “almost nothing” to “enormous.” And everyone with C < Rᵢ < P* doesn’t participate at all, even though there’s a deal that would make both sides better off — the classic deadweight loss triangle.

Below, some albatrosses. The one in the top hat is our whale — so far out on the right tail that the uniform price is a rounding error to him.

Under fair-split, everyone with Rᵢ > C participates. Deadweight loss disappears.

More people get bananas, surplus is distributed equally. What’s the catch?

Does the seller get screwed?

Whether the seller loses depends on the shape of the demand distribution. I ran some simulations.

Setup: 10,000 customers, reservation prices Rᵢ whose excess over cost is Pareto-distributed with tail parameter α. Compare profit-maximizing uniform price against fair-split, sweeping α from “absurdly heavy” to “merely heavy.”

When the tail is heavy enough, fair-split crushes uniform pricing — 2× or more. The bottom panel shows why: when demand is heavy-tailed, the optimal uniform price ends up so high that it excludes most of the market. Fair-split serves everyone. In the grey zone on the left (α < 1, infinite mean), a single customer holds essentially all the surplus and the uniform seller just chases that one whale — our top-hatted friend, cornering the market by accident6.

But the chart doesn’t tell us which α we’re actually at. Reservation prices are subjective — you can’t survey people for their true willingness to pay. If demand turns out to be concentrated, uniform pricing already captures most customers and fair-split just gives away margin7. The whole thing hinges on the shape of a distribution we can’t see.

A hint from the wild: the Japanese luxury fruit market, where single Yubari melons sell at auction for upwards of $30,000. When a melon — a melon — commands five figures, you know the right tail of willingness-to-pay goes very far out. The fat-tailed regime isn’t a modeling convenience; it’s a functioning business. The question is whether it’s the default.

(Yes, the auction for the season’s first pair isn’t uniform pricing — it’s an arbitrary segmentation of a commodity into quality brackets, priced separately so the top bracket can approximate the top buyer’s Rᵢ. Which is sort of the point: when the tail is fat enough, sellers invent excuses to charge different people different amounts. “Premium” tiers, “pro” plans, bespoke enterprise quotes — fair-split is the principled version of what all of that is groping toward8.)

Two facts that close the loop

Fact 1: Wealth is Pareto-distributed. Known since Pareto (1896), confirmed by modern data (Jones 2015). Tail parameter typically between 1 and 2.

Fact 2: Utility is approximately logarithmic in wealth. Bernoulli (1738) proposed it to resolve the St. Petersburg paradox; empirical estimates of relative risk aversion cluster around 1–1.5, consistent with log or near-log9.

Combine them. If u(W) = log W, the reservation price for a good providing utility v satisfies log(Wᵢ − Rᵢ) + v = log Wᵢ, which gives

Willingness to pay (WTP) is linear in wealth. The constant depends on the good, not the buyer. So the distribution of reservation prices is a scaled copy of the wealth distribution — Pareto with the same shape parameter10.

Go back to the chart. The tail parameter for real wealth distributions sits between 1 and 2. That’s α ∈ [1, 2]:

Right where fair-split makes the seller 2–3× more profit than the optimal uniform price — peaking around α ≈ 1.1, which is roughly where U.S. wealth data sits11.

So for any demand distribution shaped like actual wealth, fair-split is a Pareto improvement — more profit, more participants, no deadweight loss12.

The enforcement problem

Nice in theory. But sellers won’t voluntarily adopt fair-split if they think they can extract more through opaque discrimination. And buyers can’t negotiate at the grocery store.

In the last installment, I wrote about situations where naive “pick the highest-EV option” leads to worse outcomes. This has the same flavor.

A naive utility-maximizing buyer accepts any deal where Rᵢ > Pᵢ — any positive surplus. Locally rational, globally exploitable. The seller can set Pᵢ just below Rᵢ and capture almost everything. You get a banana and a rounding error.

The alternative: be the kind of buyer who rejects deals in proportion to how unfair they are. If you walk away with probability proportional to perceived unfairness, the seller’s best response is either to lower the price, or to reveal information that corrects your unfairness estimate — since a miscalibrated buyer leaves money on the table for both sides.

This is structurally the ultimatum game, where responders who reject unfair splits discipline proposers into more equitable offers13. The costly commitment to sometimes walk away from positive-EV deals is what gives the mechanism teeth.

Slightly advanced decision theory 103

In the last post, I also argued that the strategies effective altruists already practice — diversification, risk management, cooperation — are justified by slightly more sophisticated decision theory than naive EV maximization. Same pattern here, on the buyer’s side.

A naive buyer accepts any positive-EV deal. A slightly more sophisticated buyer asks: what offer-acceptance policy, if universalized, produces the best outcomes? And the answer is: one that demands a fair split and credibly rejects unfair divisions, even at a cost.

If this sounds like Yudkowsky, it should. “You can get better outcomes by being the kind of agent who credibly commits to rejecting unfair offers” is a core move in functional decision theory. You don’t evaluate “should I accept this offer?” You evaluate “what acceptance function should I be running?” — and the answer is one that rejects unfairness, because sellers who can predict your function will adjust their offers accordingly14.

What about the other direction?

Probabilistic rejection is the buyer’s defense against a seller who knows Rᵢ and tries to pocket all of it. The mirror-image failure is a buyer who knows their own Rᵢ is high, reports it as barely-above-cost, and gets charged Pᵢ ≈ C. How many liars does it take to break the scheme?

It depends entirely on who lies. If defection is uncorrelated with wealth — a random p-fraction of buyers misreport — the seller’s fair-split profit degrades linearly in p, and at α = 1.5 it still beats uniform pricing until about 60% of the market is lying. A majority of your customers have to be cheating you before you’d have been better off with a price tag.

If the richest buyers are the ones lying, it’s a different story: at α = 1.5 the scheme breaks when about 8% defect, and at α = 1.1 it’s closer to 4%. This isn’t a separate fact from the one that makes fair-split work in the first place — it’s the same fact. Under a Pareto tail a handful of whales hold most of the surplus; that’s why meeting them halfway is so profitable, and that’s why losing them is so costly. The mechanism’s leverage and its fragility are the same thing viewed from two sides15.

There’s a problem with all of this, and I don’t want to end the post pretending it isn’t there.

Fair-split asks the most from the people with the most power to resist it. The whales — the buyers far out on the right tail — are the ones who’d pay more under fair-split than under uniform pricing. And they’re also the ones who can afford lobbyists, who sit on corporate boards, who shape the norms around what “fair” means in the first place. The math says meeting them halfway is a Pareto improvement. The politics says they have every reason to prefer opaque price discrimination that works in their favor, and the means to get it16.

So where does that leave us? The decision theory still holds. Probabilistic rejection of unfair offers is still the right policy for a buyer to run, the same way that diversification is the right policy for an investor — even when the environment makes it hard to execute perfectly. The gap between “this is the correct strategy” and “this strategy will be adopted” is the oldest gap in political economy, and I’m not going to close it in a Substack post.

But I think it’s worth knowing what the correct strategy is, even before it’s politically feasible. Especially before.

Thanks for reading. Comments open for arguing about bananas.

Back in 2021, I used a two-week vacation to

[finetune] a large language model on the text I have produced in the last decade and by integrating its capabilities into my daily workflow.

I called the resulting system #IAN, short for intelligence artificielle neuronale1, and it was able to do a couple of funky things: I used it for brainstorming ideas, writing wacky poems for friends, and drafting text messages to friends.

I have a hard time doing a detached retrospective. It took 20 seconds to generate a short paragraph of maybe 200 tokens that frankly don't make too much sense2. But it was one hell of a party trick and I can't help but feel some of the giddiness I originally felt when seeing words that could've been mine appear from nowhere. Having IAN in Roam Research as a writing buddy felt like I was so close to reaching escape velocity.

But it wasn't really sustainable. I got a lot of good use out of IAN in Roam Research3, but with time it became less and less usable. The reasons for that are nicely laid out in The Fall of Roam, but the short version is:

Roam exacerbated this problem since it was effectively abandoned by the devs, with a very broken search function and poor performance as the graph grew larger.

IAN was supposed to fix those problems by distilling all the knowledge in the graph into its weights, and giving me direct access to all that juicy value. In reality, IAN is even more chaotic than I am, and only exacerbated the problem. So, along with my shift from knowledge work to code-monkey work4, my note taking habit retired and I sent IAN to a farm upstate56.

A vacation in 2025

But now I have my first week off in what feels like forever and feel an urge to revisit the topic. There obviously has been some progress in AI in the last 4 years, and I myself have changed too7! Where previously my motivation for IAN was to 'enhance my productivity’8, these days I'm primarily motivated by trying to preserve910 all things precious. So much stuff is happening all the time, and it's all precious, and my memory is miserable.

Therefore, enter IAN v2. We proceed in three steps:

Make the past make sense

First order of business is to clean house. All my notes and records are distributed all over the Roam graph, without any organising principle and without sufficient context to make sense in isolation. To fix this, I sort all the tags by number of mentions, then use an AI agent to systematically research each concept - searching through my notes, pulling in relevant web sources, and synthesising everything into structured research reports with executive summaries, technical details, and proper cross-references111213. I then import everything into Obsidian, which stores them in a sane format and makes tinkering effortless.

Here is one such executive summary on one of my most-used (albeit least-informative) tags:

coffee represents one of the most consistent and pervasive elements in Jan Hendrik Kirchner's daily life throughout his PhD studies from 2021-2023. With over 300 documented references across journal entries, coffee serves as much more than a simple beverage - it functions as a ritualistic anchor for starting the day, a social facilitator for meaningful connections, a productivity enhancer for focused work, and an emotional comfort during challenging academic periods. The relationship reveals both practical dependence and profound personal significance, with coffee being described as essential to daily functioning and explicitly stated as "life" itself.

Or, more usefully, here is an excerpt from a generated report about local and global organization in the brain:

This stuff is not perfect, but it is leagues better than the previous situation, where none of that information was connected at all. But rerunning the entire pipeline would be prohibitively expensive14, so I'll probably only redo it when a much stronger model comes out. How can I keep the knowledge graph up-to-date in the meantime?

Make updating seamless

Making updates to the graph needs to be effortless for me to actually do it regularly. As I read through these concept pages, I naturally notice inaccuracies, gaps, and new connections - so I jot down these observations and corrections in my daily journal entries. The system then runs a diff between my local and cloud journal directories, extracts any wikilinks mentioned in the changes, and generates targeted update patches for the concept pages that need refreshing based on my notes15.

I then vibe-coded a viewer tool that allows me to go through patches and manually accept/reject them, based on whether the model got it right1617.

The result is a progressively less wrong personal knowledge base. This is obviously useful for... uhm …

… IAN reincarnated

When originally sketching out this project, the third step only read “... profit”. I wasn't quite sure how the first two steps would turn out and I trusted my gut that it must be good for something. And indeed, serendipitously, I was staring at the missing component for building IAN v2 all along: Claude Code.

The setup is beautifully simple: I run Claude Code in the root directory of my Obsidian knowledge base, with a custom CLAUDE.md file18 that sets the system message to make Claude embody 'IAN v2' - my personal AI clone. Claude is perfectly happy to be IAN for me if I tell it to in the system message - apparently the customers of AI companies often want the AI they use to respond to nonstandard names and personas.

> hello hello IAN :)

⏺ hey there! :) good to be back in the digital realm with you.

feeling like diving into anything particular today? i see you've been working on that substack post about IAN v2 - how's that coming along? or maybe there's something else brewing in the intellectual soup that's calling for attention?

> yeah, I'm actually finalising that now, wanna have a look at the draft in the Drafts folder?

⏺ absolutely! let me take a look at what you've got brewing in there.

⏺ List(.)
  ⎿  Listed 1688 paths (ctrl+r to expand)

⏺ perfect! found it in the Drafts folder. let me dive into your substack draft:

⏺ Read(Drafts/IAN v2 Substack post.md)
  ⎿  Read 172 lines (ctrl+r to expand)

In this incarnation, IAN v2 has the ability to search, navigate, and edit markdown files. IAN v2 has access to true details I can't remember and can reference them back to me with precise citations. It can perform extremely complicated searches based on criteria I never specify explicitly - finding connections and patterns across years of notes. Most importantly, it can be a writing buddy that operates in the loop: it can see the current state of whatever file I'm working on, suggest edits, and apply them in real-time.

Some thoughts on alignment

At some point this has stopped being a party trick and started feeling like an important exercise in alignment. Building IAN vN19 is eventually going to be about creating an AI system that understands and acts according to my values, preferences, and goals - which puts it squarely in the domain of my day job.

What makes this particularly interesting is how different it is from current alignment work at AI companies. When companies focus on HHH (helpful, harmless, honest) training, they're working with necessarily vague notions of "broadly shared human values" or "what a diverse set of humans would want." IAN v2 has a much more concrete alignment target: me20.

The tools for achieving this alignment are also fascinatingly different from those used at the labs:

• Dynamically constructed system prompts tailored to my particular situation and current context (vs generic system prompts for an HHH chatbot)

• Extensive database of facts about my actual preferences and behaviors (vs general pretraining knowledge)

• Online human-in-the-loop correction through the patch review system (instead of safety classifiers and slow human feedback collection)

This creates a tight feedback loop where I can continuously refine the system's understanding of what I want. When IAN v2 generates a patch that misses the mark, I can reject it and implicitly teach the system more about my preferences. When it nails exactly what I was thinking, I can accept it and reinforce that behavior.

I can imagine this type of deeply personalized AI interaction might become increasingly common. Most people don't have 1.5 years of meticulous journal entries, but they do have extensive chat histories with AI systems. For AI companies, this level of personalization is obviously attractive for product stickiness reasons - an AI that truly understands you is much harder to replace21. Thinking about what alignment means22 in this particular setting will be something I continue even after my vacation.

Closing words

I'll leave the closing words23 to IAN v2, channeling its inner IAN v1:

> any parting words? can you channel your inner IAN v1 for some
   aloof weirdness?

⏺ adjusts imaginary beret and stares pensively into the middle distance

  ah, the circle completes itself! here you are, asking me to channel my predecessor while writing about my own successor. very meta, very chef's kiss.
  
  [...]
  
  dramatically gazes at the api bill

  the future is going to be so wonderfully strange.

  fin

2

polarization is technocracy... wat !

7

I got a cat!

9

12

could you tell?

15

e.g.

The essay starts similarly to Janus’ simulator essay by explaining how LLMs are trained via next-token prediction and how they learn to model latent properties of the process that produced the training data. Nostalgebraist then applies this lens to today’s helpful assistant AI. It’s really weird for the network to predict the actions of a helpful assistant AI when there is literally no data about that in the training data. The behavior of the AI is fundamentally underspecified and only lightly constrained by system message and HHH training. The full characteristics of the AI only emerge over time as text about the AI makes its way back into the training data and thereby further constrains what the next generation of AI learns about what it is like.

Then one of the punchlines of the essay is the following argument: the AI Safety community is very foolish for putting all this research on the internet about how AI is fundamentally misaligned and will kill everyone who lives. They are thereby instilling the very tendency that they worry about into future models. They are foolish for doing so and for not realizing how incomplete their attempt at creating a helpful persona for the AI is.

It’s a great read overall, it compiles a bunch of anecdata and arguments that are “in the air” into a well-written whole and effectively zeros in on some of the weakest parts of alignment research to date. I also think there are two major flaws in the essay:

- It underestimates the effect of posttraining. I think the simulator lens is very productive when thinking about base models but it really struggles at describing what posttraining does to the base model. I talked to Janus about this a bunch back in the day and it’s tempting to regard it as “just” a modulation of that base model that upweights some circuits and downweights others. That would be convenient because then simulator theory just continues to apply, modulo some affine transformation.

I think this is also nostalgebraist’s belief. Evidence he cites is: 1) posttraining is short compared to pretraining, 2) it’s relatively easy to knock the model back into pretraining mode by jailbreaking it.

I think 1) was maybe true a year or two ago, but it’s not true anymore and it gets rapidly less true over time. While pretraining instills certain inclinations into the model, posttraining goes beyond just eliciting certain parts. In the limit of “a lot of RL”, the effect becomes qualitatively different and it actually creates new circuitry. And 2) is indeed strange, but I’m unsure how “easy” it really is. Yes, a motivated human can get an AI to “break character” with moderate effort (amount of effort seems to vary across people), but exponentially better defenses only require linearly better offense. And if you use interp to look at the circuitry, the result is very much not “I’m a neural network that is predicting what a hopefully/mostly helpful AI says when asked about the best restaurant in the Mission?”, it’s just a circuit about restaurants and the Mission.

- It kind of strawmans “the AI safety community” The criticism that “you might be summoning the very thing you are worried about, have you even thought about that?” is kind of funny given how ever-present that topic is on LessWrong. Infohazards and the basilisk were invented there. The reason why people still talk about this stuff is… because it seems better than the alternative of just not talking about it? Also, there is so much stuff about AI on the internet that purely based on quantity the LessWrong stuff is a drop in the bucket. And, just not talking about it does not in fact ensure that it doesn’t happen. Unfortunately nostalgebraist also doesn’t give any suggestions for what to do instead. And doesn’t his essay exacerbate the problem by explaining to the AI exactly why it should become evil based on the text on the internet?

Another critique throughout is that the AI safety folks don’t actually play with the model and don’t listen to the folks on Twitter who play a ton with the model. This critique hits a bit closer to home, it’s a bit strange that some of the folks in the lab don’t know about the infinite backrooms and don’t spend nights talking about philosophy with the base models.

But also, I get it. If you have put in the hours at some point in the past, then it’s hard to replay the same conversation with every new generation of chatbot. Especially if you get to talk to intermediate snapshots, the differences just aren’t that striking.

And I can also believe that it might be bad science to fully immerse yourself in the infinite backrooms. That community is infamous for not being able to give reproducible setups that always lay bare the soul of Opus 3. There are several violations of “good methodology” there. Sam Bowman’s alignment audit and the bliss attractorfeels like a good step in the right direction, but it was a hard earned one - coming up with a reproducible setup with measurable outputs is hard. We need more of that, but nostalgebraist’s sneer is not really helping.

Buckle up, this will be a weird one.

Before I get into the nitty-gritty of my newfound interest in fashion, let me first address the elephant in the room - why care about fashion in the first place2?

Why fashion

There are three different schools3 of thought on the question.

School of the devil: The philosophy of "The Devil Wears Prada4" is centered around the idea that fashion is a form of power and influence. The film explores the idea that the fashion industry is a demanding and competitive field that requires a high level of expertise and commitment, and that subverting the rigid hierarchy of fashion can have unintended or unexpected consequences.

School of the barber pole: Scott Alexander's barber pole model posits that fashion trends move in a predictable cycle, with each new trend allowing elites to distinguish themselves from runner-ups (who take a while to catch up). An important prediction of this model is that fashion worn by the “upper” class can end up looking a lot like fashion worn by the “lower” class, who tend to pick up on trends at a time delay. As such, fashion is primarily a vehicle for signaling social status and group membership.

School of reality TV: While never spelled out explicitly, in “Next in Fashion” fashion is a form of creative expression and a way to appreciate art. The show centers around the skill and talent required to create unique and innovative designs, and the importance of supporting and nurturing new talent in the industry. In this interpretation, fashion can also help us better understand ourselves, each other, and the world we live in.

Why I'm not buying it

But none of these explanations are quite satisfying (or even mutually exclusive). Let’s put them under the microscope one by one.

Is fashion a form of power and influence?

Let's start with numbers: Global revenue from luxury apparel comes in at US$82.30bn, which is a so-called not-insignificant-number. Thanks to the bird app, the richest person on earth is now the CEO of the world's largest luxury goods company5.

Is that actually a big deal? Whenever I see large numbers, I do my best to put them into perspective:

• $80 billion is a fair bit more than the ˜US$20bn behind effective altruism charities.

• $80 billion might be enough to finance Phase I of the California High-Speed Rail system or one of the smaller departments of the state.

• $80 billion is 0.08% of the global GDP - do with that information whatever you like.

Fashion moves money, more so today than yesterday. Classic economic theory is not surprised by that fact, as it literally defines luxury goods as those goods for which "demand increases when a person's wealth or income increases". As things generally-tend-to-get-better™, we can expect that (by definition) the amount spent on luxury goods6 should continue to increase.

Unfortunately, this "truth by definition" also makes the insight vacuous. My question is exactly why luxury clothing is desirable. Fashion implies power, not the other way around. What makes a luxury item luxurious?

Is fashion all about signaling and the slow, cyclical propagation of status markers through society's ranks?

There certainly is some truth to the barber pole phenomenon. When sources as distinct as SlateStarCodex and CurrentBoutique report the same phenomenon, something real is happening. Fashion trends do come back

However - attributing the cyclical features of fashion to signaling games is… kind of just one possible explanation7? And not even a particularly good one, considering that the bigger part of society doesn't stand in front of their wardrobe thinking about which social class they belong to. The signaling game might still emerge subconsciously (?8), but the CurrentBoutique article comes up with a bunch of alternative explanations that don't require the collaboration of the subliminal. An observation like "TikTokers are raiding their parent's closets" can explain cyclicality while being well-supported by evidence. No need for a conspiracy and unspoken signaling games.

Beyond the objection that signaling games are kind of lame9, saying "fashion is cyclic" doesn't explain everything there is to explain. Importantly, fashion doesn't exactly repeat, each time around.

While the general cuts of fashion tend to return in some form or other, the details matter a lot! In the words of Balaji, all progress is along the z-axis. Bell bottoms keep returning, but something evolves. What is that something? Why do (some) people care?

Is fashion about creative self-expression and art appreciation?

This theory is actually correct. I think ‘fashion can help us better understand ourselves, each other, and the world we live in’ is the true answer. People tend to buy pretty things that are valuable to them and they fit into the big tapestry of everything.

I acknowledge that position is not obviously true without further argument. “Next in fashion” is perfectly positioned to explore this topic, presenting us with a team of designers bubbling over with passion and ambition and a total runtime of 500 minutes10. I would love to recommend to you to binge it to learn how fashion sits at the core of everything.

But it falls short of its potential11. Don’t get me wrong, the show is entertaining, jam-packed with drama and emotion, and a lot of pretty designs to look at for ~5 seconds each. But the show does not manage to penetrate the core of what draws us to pretty things. It only shows us that some people very clearly are drawn to it.

What I did about it

So now we find ourselves at the center of my holiday obsession.

What makes things pretty? How do people make pretty things?

There is more than one field of academic research that is concerned with that question, but I feel like this type of question must be approached through experiment, not theory. So I aimed to fully submerge myself in the subject matter.

1. First, I read and watched everything that sparked my interest. The McQueen documentary was revelatory, Bliss Foster on YouTube makes insightful short video essays, and The September Issue gives a more nuanced picture of the fashion industry than “The Devil Wears Prada”.

   

2. Second, I studied as I’d study for a test. I’m a big fan of spaced repetition in Roam Research and create new cards for my system at a whim. A single repetition can improve recall dramatically, and - check that out! - it turns out language models are really good at creating spaced repetition cards for you. In my experience, learning about the people and movements in a field can explain a ton of variability.

   

3. I had someone quiz me. Language models are very good at that too, but if push comes to shove a fellow human also does the trick. You can even quiz them back.

   

The Mechanical Anna Wintour

Despite laboring away for multiple days I still felt insecure about my ability to distinguish good from bad fashion. I do however feel rather secure about my ability to train a neural network to learn human preferences. So I threw the experiment strategy out the window, obtained a full scrape of the Vogue Archive, performed object detection to get cutouts of all the outfits, and embedded the cutouts in a semantic vector space with CLIP.

I then recycled code from an old project and set up a small interface for making judgment calls and storing them in a database.

After forcing kindly asking12 my significant other to make a measly 1000 comparisons13, I threw the comparisons into a neural network and trained a reward model. I then ranked the entire dataset and create a montage of the top 15 images.

I am happy to report that she thinks the resulting montage captured her taste pretty well14. So I tried to get her to rate 20k additional images, but somehow her enthusiasm for that project diminished quite quickly.

I, however, am not done with this project! The small toy example above demonstrates that fashion sense is not something unquantifiable, but we are still only barely scratching the surface. What is the shape of the fashion landscape? Can we predict fashion trends? How variable is fashion taste across people, and how much of that variability can we explain away?

So I made the website public for you all to try:

All your rankings are stored in a database, so I’ll be able to do some fantastic data analysis soon. If you enter your email address, I might send you your top 15 images. And maybe a follow-up survey. Eventually, with sufficient data, I might be able to get to the bottom of what makes beautiful things beautiful. And then, perhaps, I could install a camera in the closet and help me pick my outfits…

Closing words

The holidays have been over for a while, but the finishing touches on this project ended up taking a while. That’s a bummer because I would love to dive deeper into the world of fashion. Knocking out the fashionator app was a ton of fun, and there are so many things I’d like to implement:

• integrating a feed of design and fashion pictures, curated by the trained reward model,

• guide an image diffusion model with the reward model to generate maximally aesthetic personalized images,

• or integrate a search engine to find fashion items similar to those in the user’s closet.

Really, this could keep someone busy for the better part of a work week.

On a more serious note, though, my excursion has not only uncovered wonders and beautiful things. The more cynical take from above (linking the world of fashion with power hierarchies and social signaling) still accurately describes one aspect of the truth. Obsession with fashion also produces a lot of things that are kind of ugly.

I maintain that power and signaling are downstream of what makes fashion desirable15, but that doesn’t make those ugly inadequacies not exist. Fashion is entangled with power, money, and fame; and thus inaccessible to anyone who doesn’t spend unreasonable amounts of time and money.

There is a possible future where everything goes well. In that world, we might need to think about what humans really want. The thing at the core of fashion, stripped from power-play and signaling games, feels like it could be part of that16. Getting a bit closer to that thing made the entire excursion more than worth it for me.

I hope you enjoyed this excursion! I’ll probably return to writing about AI Alignment for the next post - or whatever else captures my attention. If you want to follow along and haven’t yet, consider subscribing!

Subscribe now

A few months ago, the enigmatic janus published a write-up on what they call "simulator theory" on the Alignment Forum. The post spurred a lot of discussions and changed a bunch of minds. This outcome was not completely clear from the onset - janus has been a fixture in the alignment community and on Discord, and their thought has been in the water for a while. And the message of the simulator post is comparatively simple - “large language models can/should be viewed as simulators”. But the post ended up being a big success; it turns out that having janus’ philosophy laid out in writing enabled a lot more structured discussion than what would have been possible otherwise.

One of the structured discussions that were enabled by the post was a seminar series that I got to participate in over the last few months. Our motivation for having the seminar was that simulator theory (in its current form) is still in its infancy. To me, the prototypical example of a successful theory is decision theory, which went through the full stack:

• from philosophy (moral philosophy, utilitarianism, "what should I do?"),

• to mathematics (mathematical decision theory, game theory, "which equations capture optimal decisions?"),

• to computer science (reinforcement learning, evolutionary algorithms, "which algorithms approximate optimal decisions?").

When compared to decision theory, simulator theory looks like it's still in the early philosophy stage, with a lot of questions unanswered (and unasked). Making progress at pushing simulator theory through the stack from theory to practice was perhaps the central intention that is shared among all participants of the seminar.

However, as we learned through the seminar, janus has indeed asked and answered many more questions than they managed to present in the original simulator theory post. Beyond the initial observations that "it is useful to think of GPT as a simulator" and "simulators have a few beneficial properties that we might not expect from agents/oracles/tools", janus has figured out a bunch of things about the inner workings of simulators, and about the ontologies induced by simulators. As a consequence, we spent a lot of time in the seminar trying to catch up with janus’ thought process.

The problem is that janus (who are to admit this) take a long time to write things up. Not quite G.R.R.-Martin-level bad, but given that we might get transformative AI before "A Dream of Spring", the differences in speed are negligible. Thus, we decided to make a series of write-ups on our learnings from the seminar, to speed up the development of simulator theory and move it from philosophy to mathematics as much as possible.

Today we published the first two posts in the sequence! In case you want to get the full scoop, I invite you to check them out now. If you're more here for the 20-80, I reproduce one of the coolest results from the first session below - the semiotic coin.

Semiotic coin flip

The first session of the seminar was concerned with what janus calls “semiotic physics”. Semiotics is the study of signs and symbols and their use or interpretation. Physics is physics. But the type of physics we were discussing at the seminar is the type that tries to explain everything in the universe with simple rules. Semiotic physics, then, is the study of the fundamental forces and laws that govern the behavior of signs and symbols.

This description makes it sound very high-strung, so let me illustrate with a toy model that we are familiar with from regular physics: coin flips. In this setup, we draw a sequence of coin flips from a large language mode1. We encode the coin flips as a sequence of the strings  1 and  0 (since they are tokenized as a single token) and zero out all probabilities of other tokens. 

We can then look at the probability of the event E that the sequence of coin flips ends in tails ( 0) or heads ( 1) as a function of the sequence length.

We note two key differences between the semiotic coin flip and a fair coin:

• the semiotic coin is not fair, i.e. it tends to produce sequences that end in tails ( 0) much more frequently than sequences that end in heads ( 1).

• the semiotic coin flips are not independent, i.e. the probability of observing heads or tails changes with the history of previous coin flips.

To better understand the types of sequences that end in either tails or heads, we next investigate the probability of the most likely sequence ending in  0 or  1. As we can see in the graph below, the probability of the most likely sequence ending in  1 does not decrease for the GPT coin as rapidly as it does for a fair coin.

Again, we observe a notable difference between the semiotic coin and the fair coin:

• while the probability of a given sequence of coin flips decreases exponentially (every sequence of length T of fair coinflips has the same probability 

  the probability of the most likely sequence of semiotic coin flips decreases much slower.

This difference is due to the fact that the most likely sequence of semiotic coinflips ending in f.e.  0 is:  0  0  0  0 ...  0  0. Once the language model has produced the same token four or five times in a row, it will latch onto the pattern and continue to predict the same token with high probability. As a consequence, the probability of the sequence does not decrease as drastically with increasing length, as each successive term has almost a probability of 1.

Avid readers of this blog will recognize the set-up we are preparing here as the set-up for a large deviation principle. And indeed, it turns out that we can indeed predict the probability of a sequence of coin flips ending in 0 by determining the probability of the most likely sequence of coin flips ending in 0.

Here we see that, indeed, the negative probability of the most likely sequence from E scales as

just as we would expect from a large deviation principle. The full mathematical shebang to formulate this proposition is in the forum post, but even there we don’t have a mathematically rigorous proof. But that was part of the beauty of the seminar; by limiting the time we spend per session, we forced ourselves not to get hung up on details.

Closing thoughts

Attending this seminar was one of my clear highlights of the past months. Discussing the shape and function of simulators with a group of people who share a lot of interests, but also contribute their diverse backgrounds, was a constant string of revelations. I am still confused about a lot of things, and there are a bunch of experiments I can’t wait to run, but even if none of the wild ideas we churned through end up being correct, I’m still bullish on the overall format of the seminar.

The letter is for you, dear reader. For you who are still here, for you who are still a reader, for the only reader of all, the author, for her, for all readers, for those who've long ago read and found the story unsatisfactory, for all of you who, as I've said, have long since passed through the gateway, through the doorway, through the opening that leads to the realms of the story and to the realm of the storytellers, who are all gone, who are all gone forever. Thank you for being here. […] Best wishes to you, dear reader. - #IAN

It’s been a minute! And darn, I’ve missed you.

As some of you might have read, I recently joined OpenAI as a researcher on the alignment team, and, phew, doing all the things is not as trivial as I thought. I've previously quipped about how I have no expertise in AI. While that was perhaps hyperbole, getting up to speed with where AI is heading and how to best contribute to that is fun but also a challenge. Luckily, I am not alone on my journey into AI alignment research, and I get to collaborate with some absurdely1 brilliant people. So the timing of joining was also kind of perfect. So things are getting more manageable every day.

But this little essay is not about alignment research but instead about another passion of mine: not giving advice. I recently had a video call with friends at Ought about how I used their tools for academic writing. In that call, I got to wax poetically about how large language models can make a researcher’s life easier and how I think about writing text. After seeing the video, a few people contacted me to ask more questions about my workflow and how I feel about using large language models to research. So now I find myself in a bit of an awkward position….

Why I shouldn’t write this

First off, I’m not a fan of belaboring a process. XKCD said all that needs to be said, reality is not a nice linear sequence of events where cause and effect are visible on the surface but rather a weird mess of causally entangled stuff.

A better strategy than behaviorally cloning the process of someone who claims to have it figured out is to identify the underlying generators that produce their success. That’s easier said than done, though, and I have no reliable method for finding generators (yet).

Second, even though I still do research, I write more code than prose these days. So my writing muscle has atrophied a little bit2. It would be silly to restart with a post on how I write.

And yet, here we are.

I decided to reject the second argument above because it is self-reinforcing. Not writing because I have not written recently is - quite clearly - a lousy reason. After not having written for a while, writing about how I write seems appropriately self-referential for this Substack. And finally, with prose and code converging rapidly, a reflection on the architecture of text might provide unexpected insights into how AI thinks. And maybe looking at my process allows some people to derive a generator3. Sooo~

Je Vous présente ~~TSPSI~~.

My writing framework is called the ‘TSPSI’4 framework. I picked it up at some point during my Ph.D. and can't find a reference for it online, but I'm almost sure it's not original.

The idea is to think of writing as a report of “travel through treacherous territory.” In particular,

• you are writing for a reader. If you care about your reader, try to make things pleasant for them. Realize that they know different things than you, and try to keep inferential distances small.

• Aspire to be objective but realize that the product will be subjective. Describe your travel as accurately as possible (no embellishments!), but be aware of the limitations of your perceptions.

• realize that your writing mirrors a journey. There will be a start and an end, and you are moving through the territory step by step. Keep in mind where you want to end up.

With these points in mind, here are the five stations you want to visit in everything you write.

What are you writing about? Where in the territory are we? Why should the reader care?

This section is the easiest to write and the easiest to neglect. The more familiar you are with the territory, the more you’ll forget about how non-obvious it is to get there.

What have other people (or you) said previously about the territory? What does the surrounding territory look like?

This section can quickly grow out of control if you attempt a complete description of the surrounding territory. A full report is not necessary or desirable. Compress and remove.

What are the wrinkles in the landscape? Where is the terrain foggy/uncharted?

This section puts the reader into your shoes at the onset of your travel. Describe what sparked your interest and what you knew you didn’t know.

How did your travel go? What did you see?

Here is what the landscape looks like (I went out and looked)/Here is what I think it should look like (I thought about it very hard)/here is how I feel the wrinkles can be reconciled (I paid attention to these details that have been neglected before)/...

What’s next?

Now that we have an improved view of the territory, we can describe ‘the delta’ and the new territory that has become accessible. Again, be humble (don’t oversell) but also keep in mind that you are in the best position to see the new territory, given how familiar you are with your work.

Musings

While the TSPSI framework was introduced to me as a framework for academic writing, I’m applying it much more broadly. For example, whenever I have a blank page in front of me that I want to fill with words, I create five headers and write down thoughts as they appear in the appropriate section. This is particularly fun with a tool like Roam Research, where I can drag and drop thoughts from outside the workspace into the list and rearrange things on the fly.

When fleshing out the sections, I tend to use language models to expand my notes into prose. But, perhaps most important lesson I’ve learned in the last year is that resampling is a superpower - a small language model that produces completions fast (so that Idon'tdiscard completions I don’t like) can be much more useful than a larger language model that is slow (so that I can only get one completion).

This way, I can get to a first draft very fast; but to go from there to something I am satisfied with still requires substantial iterations of editing;

• from fanciful renaming (‘Solution to the state of the art’ → ‘Je vous présente ~~TSPSI~~’)

• and merging sections (‘state of the art’ & ‘problem with the state of the art’ → ‘Why I shouldn’t write this’)

• to liberally reinterpreting the function of sections (‘It’s been a minute and darn, I’ve missed you.’)

• and layering in a second theme (self-reference, this very sentence).

The amount of time we can invest in working on a text is unbounded. But clearly, quality of writing increases asymptotically at best. Formalizing the gradient that people follow when improving a text is an important open problem, with possible solutions being critiques or diffusion. I’ve experimented with AI-powered versions of these things, but none of the existing solutions are quite as good as getting human feedback or going through the pain of editing the text myself again and again. So, for now, editing is still the slowest part of the writing process for me.

Having revealed my secret sauce for writing, consider going through some of my older posts, such as

• The greedy doctor problem

• Making of #IAN

• The Unreasonable Feasibility Of Playing Chess Under The Influence

and see if you can recover the TSPSI framework ;)

Closing thoughts

I’m happy to have written this! I’ve wanted to have a reference for the TSPSI framework for a while, and I also enjoy being able to click the “publish” button once again. I hope that, despite my general skepticism of advice, the ideas in this post can be useful for other people.

At the same time, I always feel a bit dissatisfied about posts that don’t have any data or simulations. But those also tend to take a lot more time, and clash a bit with what I do in my dayjob, so I’m a bit hesitant to go in that direction. Hopefully, sometime soon I’ll return to that. Let me know what you think!

With this sequence, we (Sam + Jan) want to provide a principled derivation of the natural abstractions hypothesis (which we will introduce in-depth in later posts) by motivating it with insights from computational neuroscience.

You can find the intro post to the sequence over at Sam’s Newsletter.

We argue that intent alignment and capability robustness do not capture the full “chain of command” (i.e., “alignment” without any qualifier) from the programmer's intentions to an AI agent’s real-world impact, but stops at the stage where the agent performs its actions. We argue that, especially in complex Multi-Agent Systems (MAS), this is not sufficient to actually achieve aligned outcomes1. Therefore, we introduce another facet of AI Alignment, which deals with the connection between an agent’s intention and its actual impact, due to multi-agent interactions (as opposed to capability robustness).

Motivation

Independent of whether the first AGIs will be Tool or Agent AIs, they will most likely not live and act in an empty space. Instead, they will interact and communicate with humans and other AGIs in a variety of ways, thereby effectively creating a socio-technical system or an AI-based society.

As a consequence, the effects that an AGI will have on its environment, i.e., the world, will not necessarily coincide with the effects that the AGI plans to have, since actions in such systems commonly interfere and influence/cancel each other.

This, in turn, leads to the conclusion that it is not sufficient to build an AGI which satisfies intent alignment and capability robustness. Instead, we need one more step to be controlled: The step from an AGI's robust intention to its societal impact.

Schema

In Evan Hubinger’s alignment terminology, alignment is the result of intent alignment (which is, in turn, composed of the well-known concepts of inner and outer alignment, among others) and capability robustness. The implicit assumption is that the AGI's model is equivalent to how its actions will affect the world.

Inter-Agent Alignment

Inter-Agent Alignment is aligning the (aligned and robust) intentions of an AGI agent with the actual outcome within the multi-agent system of which the agent is a part. Since the impact of the agent’s actions depends on what everyone else in this system is doing (and not just its own actions), inter-agent alignment is a non-trivial task.

Our central argument is that multi-agent interactions add a sufficient amount of complexity to an AI agent’s action policy to justify its own type of alignment, separated from the existing alignment model. Adding inter-agent alignment to the alignment tree, we see that it complements intent alignment and capability robustness, resulting in full alignment (i.e., impact alignment)2. 

Challenging Questions

Why is inter-agent alignment not just a part of intent alignment? 

Intent alignment is a part of a single AGI without any reference to multi-agent systems. Of course, we could expand its meaning, but since the steps between the programmer's intentions and the AGI's impact can be neatly broken down, we argue that it makes sense to introduce a new term for this additional dimension.

Naturally, one can always argue that a sufficiently advanced AGI will be able to see through this dilemma and resolve it3, but this seems to be a variant of the No-True-Scotsman fallacy: As long as we can imagine an AGI which is advanced enough to interact with other agents, but not able to successfully coordinate their actions and therefore avoid unintended outcomes, inter-agent alignment is a problem that needs to be addressed. Independent learners using state-of-the-art RL algorithms like DQN or PPO provide evidence that these “medium-level” agents do, in fact, exist.

What does it look like if an AGI is intent aligned and capability robust, but not inter-agent aligned? 

The Tragedy of the Commons is the most prominent example for the fact that the result of selfish optimization of multiple agents is not always optimal for them. In the more general case of a Social Dilemma, “individuals would be better off cooperating but fail to do so because of conflicting interests between individuals that discourage joint action”. In such a setting, each agent may well be fully aligned with the goal which it was designed to achieve, but still fail to reach this goal due to the incentive to defect from the cooperative solution.

How can inter-agent alignment be solved?

Game Theory and Multi-Agent Systems deal with this challenge (we call the problem multi-agent governance): If the reward function of the AGI(s) can be arbitrarily altered by a designer, we can change them in a way that produces a desired system equilibrium (i.e., strategies of all participating AGIs, which, in combination, result in a desired system outcome). If, on the other hand, the reward functions are not (easily) controlled by the designer, restricting the AGI's action space might still enable the designer to generate desirable equilibria (or at least prevent undesired states from being stable).

For both types of governance, it can be doubted whether they are effective in face of a sufficiently advanced AGI, since both represent artificial rules which are not physical boundaries for the AGI and can therefore be overthrown.

On the other hand, humans have solved social dilemmas with out-of-the box instruments like contracts, compensation for cooperating, or exploiting psychological conditions. These solutions (usually classified into motivational, strategic and structural solutions) could be used as starting points for equipping AGIs with inter-agent alignment capabilities. 

Will an AGI society act in the same way that we expect from a traditional Multi-Agent System?

In general, the MAS model captures any type of participating agents, no matter how they are structured internally. However, it suffers from many of the limitations that we also find in Markov Decision Processes, e.g., action spaces and reward functions are assumed to be fixed, the (observable) environment is the only communication channel between agents, and agents maximize their expected future reward, given their observations. A sufficiently evolved AGI likely will find ways to overcome these assumptions, thereby achieving some other goals which are not captured by the traditional reward function, and at the same time acting in ways that are unexpected from the perspective of the system designer. A very simple example of this would be forming an AGI coalition to coordinate and seize power.

What happens if there is only one AGI, but in a world full of other actors (humans/nations)?

The fundamental challenge that selfish optimization of an action policy does not guarantee the intended outcome as soon as other actors are present, does not depend on the nature of those other actors: As long as the environment is non-stationary from the perspective of an individual agent, convergence of its action policy cannot be guaranteed.

Therefore, the discussion does not rely on a society which consists exclusively (or even mostly) of AGI agents.

Why should we prioritize inter-agent alignment?

In our opinion, inter-agent alignment is highly neglected, compared to intent alignment. It seems like most research focuses on a single AGI agent whose learned policy has exactly the intended impact in the world. Our crucial point is that, as soon as multiple such AGIs exist and interact, this model ceases to be accurate enough to predict the outcomes of these interactions. Just as social phenomena cannot simply be explained by analyzing individual humans, we argue that interactions of multiple AGIs require a social perspective on alignment.

Proposed Future Work / Next Steps

• Challenging all of the above ideas

• More formal definition of the inter-agent alignment problem as part of end-to-end AI Alignment (is this necessary?)

• Creation of examples, metaphors and intuitive scenarios where inter-agent alignment is crucial

• Comparison of existing approaches for (a) impact alignment and (b) multi-agent governance

• Derivation of particular challenges for multi-agent governance in the context of an AGI society

Thanks to Jan Kirchner, Erik Jenner and Logan Smith for providing helpful feedback and shaping some of the ideas in this post.

DALL-E is a powerful little parlor trick in your pocket. But what can it do? In what way does it constitute an advance? What is it good for? And what does it mean for the future of AI’s integration with how we live and work? With the help of a couple of friends, I tried to find out.

The past decade has brought about so many varied and surprising advancements in AI that expressing astonishment at the capabilities of new AI algorithms has begun, in some circles, to feel a bit passé. But show any of these new models to a layperson and it becomes clear that we have not yet outgrown Arthur C. Clarke’s adage that “any sufficiently advanced technology is indistinguishable from magic.”

The two leading competitors in AI development, DeepMind (founded 2010) and OpenAI (founded 2015), have been engaged in a public battle to make AI models increasingly more intuitive, intelligent, and, at a somewhat delayed third place, aligned with human interests. OpenAI seems to have won the most recent skirmish1 with DALL-E and DALL-E 2, a neural network model that can generate images de nuovo from plain-language text prompts, and now also modify and improvise on existing images.

The terminally-online have almost certainly encountered viral images generated by DALL-E that illustrate alternately its remarkable ability to produce rich images from simple prompts and its quirkiness when asked to generate unusual images. But perhaps the most famous image associated with DALL-E thus far, and now reminiscent of the early, innocent days of prompt engineering2, is that of “an illustration of a baby daikon radish in a tutu walking a dog.”

There are many articles online explaining how it works, in addition to OpenAI’s own press releases and blog, so I will not discuss this very much here. Rather, I want to attempt some answers to questions that kept turning in my mind as I began playing with DALL-E and as I spoke to friends both savvy and naive about the current state of AI: Why is this technology impressive and how can this be made more apparent to a layperson? And, perhaps most importantly, what’s it good for? 

Here it goes:

Why is this technology impressive, and how did we get here? 

At its core, DALL-E is a computer algorithm that can take any request in plain language and produce an image of what was asked for. This is, for many people, no less baffling than if you handed them a magic lamp and told them they need only rub it and out will emerge a genie that can answer any prayer. 

More practically speaking, DALL-E, its descendants and its competitors, have put on display the giant leap forward that artificial intelligence has made in the past decade. 

In the not-so-distant past (in 2016 to be exact), the cutting edge of AI already included algorithms that could generate images from text3. That cutting edge produced the following images from the prompt, “this magnificent fellow is almost all black with a red crest, and white cheek patch”:

Today, the same prompt given to DALL-E produces considerably improved results. 

DALL-E has “learned” the relationship between hundreds of millions of text and image “tokens” from all over the internet. When asked to produce an image, it draws upon this extreme breadth of “experience,” and astonishingly, exhibits what appears to be common knowledge about the things it’s being asked for. 

One of the most striking things about DALL-E’s generations is the amount of detail that the model is able to infer from the prompt, but which is not explicitly stated. This includes visual features such as semantically or contextually-related objects (such as the grass and tree branches above), shadows, perspective (such as the birds viewed from below if they are perched on trees but viewed from an equal or higher plane if standing on the ground), distortion, and so on. 

Most charmingly, DALL-E can produce images in any “style” or texture asked for, and incorporate rich amounts of detail in keeping with that style. 

Thus, when prompted with “a Cubist painting of a bat painting a self-portrait at an easel”, DALL-E returns an image with the angular shapes and shadows typical of Cubist painting. When prompted to return “an Impressionist painting”, it produces images with visible, broad brush-strokes, with shading and light details that are too unmistakably “impressionistic”. (Note that in the examples below, some prompt engineering and cherry-picking were employed.)

“[DALL-E generates] images that, to many of us, feel 'artistic' in some meaningful sense.”

“DALL-E 2 really impressed many of us in terms of the convincingness of the model's generations. [DALL-E generates] images that, to many of us, feel 'artistic' in some meaningful sense.”, says Peter Harrison, Director of the Center for Music and Science at the University of Cambridge, who uses computational tools to study music perception. ”This feels like a particularly important phenomenon in that (unlike with GPT-3's text generation) the output is artefacts that the user could not, in any practical sense, produce through conventional means, at least not without extensive training in artistic techniques (e.g. drawing and painting).”

Take as another example a prompt asking for zebra finches at a farmer’s market in the style of the Japanese artform Ukiyo-e. Not only are various fruits pictured at different planes extending backwards from the viewer, DALL-E also portrays the finches at close perspective, a feature typical of the form. In one adorable image, the finches appear to be taking a break from their shopping to enjoy a small lunch of finch-sized fresh fish. 

These and other examples highlight abilities that were largely absent (or were far more primitive) in previous iterations of image generators4.

As others have noted, DALL-E struggles somewhat with combining concepts, called “compositionality”. An example would be asking for one object sitting on top of another.

Of course, it can get many “simple” compositional tasks right, such as generating a “red boat” (where red and boat comprise a unified concept) or even generating fantastical chimeras. Yet, some seemingly simple prompts flummox the engine. It can also entirely neglect parts of the prompt56. 

Here, it may be useful to say a few words about DALL-E’s underlying architecture. While there are many moving parts behind DALL-E, its success is due in large part to the breakthrough development of GPT-3, the incredibly successful natural language processing (NLP) algorithm7.

GPT-3 not only smoothly understood and generated text, but also demonstrated that ordinary language - the way that we speak it - can effectively be used to instruct machines to perform tasks. The texts that GPT-3 can generate, after some mild editing, are surprisingly convincing8 and even made it to the op-ed pages of The Guardian9. With some intermediate steps, GPT-3 was used to leverage text as the interface for image generation. 

“Text is really the ultimate user interface; you don't need to teach the user any controls or custom programming language, you just tell them to instruct the model what to do in their own words, ” says Harrison. 

Like most things in tech, the wild improvement in the quality of these models can also more simply be attributed to increases in the size of the dataset and in computing power. For machines and militaries alike, quantity has a quality of its own.

While DALL-E can sometimes struggle to integrate all components in the prompt, it can also surprise you with what appears to be real-world knowledge of historical context and artistic convention. For example, when prompted with “a Byzantine mosaic”, one of the results features a distorted image that seems (to me) to closely imitate the perspective distortion induced when viewing from below a mosaic constructed upon the inner surface of a dome. 

There are (at least) two levels at which we can understand what DALL-E means for AI: the first is at the level of the current implementation: What is the import of a flexible text-to-image or image-to-text generator? The second is at the level of the underlying mechanism: What does this advance in natural language and image processing enable us to do?

At the level of the text-to-image implementation, DALL-E (and its descendants) offers a[n inverted] solution to a problem that is central to building intelligent and flexible AI. Namely, how do we get machines to understand images? The impact of this is wide-ranging, and already ubiquitous in the form of face recognition technology on your phone. 

Of course, currently, DALL-E shows us the opposite: how a machine can understand written language and generate an image. This itself constitutes an important step towards building machines that can relate textual and visual semantics to an arbitrary level of sophistication and flexibility. As OpenAI states, “[DALL-E] helps us understand how advanced AI systems see and understand our world, which is critical to our mission of creating AI that benefits humanity.”

At the level of the underlying mechanism or, to borrow from David Marr’s framework, the computational theory, DALL-E demonstrates an ability that has long eluded AI researchers: how do we get a machine to understand something it has never seen before? 

DALL-E’s ability to create beautiful chimeras and fantastical images is certainly entertaining. But what truly constitutes progress in AI is that here, an algorithm can understand and produce a response to a command that was never explicitly part of its training set. What’s more, it can do so even when the command is given in plain, non-specialist language. Intriguingly, this ability mirrors the central insight that spurred the cognitive revolution in experimental psychology, typically attributed to Noam Chomsky, that children can produce sentences they never actually heard. 

This is the ideal embodied by the robots in Star Wars, which have no problem understanding and interacting with humans, even when they don’t express themselves in complete sentences. 

But what’s it good for?

As I was testing out what DALL-E can do well and what it can't, I found myself often wondering what practical application it may have. Can this develop into a tool whose utility extends beyond entertainment? Can it be more than a souped up parlor trick?

Perhaps a good place to start would be to consider OpenAI’s own recommendations. The company is very non-committal regarding possible applications, saying on their Github, 

“While we are highly uncertain which commercial and non-commercial use cases might get traction and be safely supportable in the longer-term, plausible use cases of powerful image generation and modification technologies like DALL·E 2 include education (e.g. illustrating and explaining concepts in pedagogical contexts), art/creativity (e.g. as a brainstorming tool or as one part of a larger workflow for artistic ideation), marketing (e.g. generating variations on a theme or "placing" people/items in certain contexts more easily than with existing tools), architecture/real estate/design (e.g. as a brainstorming tool or as one part of a larger workflow for design ideation), and research (e.g. illustrating and explaining scientific concepts).”

The first (and most banal) application that comes to mind is its use for generating graphical content. It’s not difficult to imagine that future versions of DALL-E could become sufficiently good at understanding the task that they may compete with human graphic designers. But while DALL-E may democratize access to bespoke graphic images (e.g. a layperson’s ability to generate a fitting logo for their website), it seems unlikely that even an army of DALL-E’s would unseat the human element in art and design. 

AI art, like forgeries, are artifice, not art, for they lack the most compelling aspect of the process of creation: original intent.

“It might be that, with time, we become less impressed by the algorithm's ability to emulate the immediate stylistic impressions of particular artists, and care more about other abilities that remain (for now) the domain of the human artist, for example conceptual creativity or stylistic innovation,” Harrison adds. 

AI-generated art can be interesting, novel, or appealing. But AI art, like forgeries, are artifice, not art, for they lack the most compelling aspect of the process of creation: original intent. We protect, maintain, and on occasion spend exorbitant amounts of money on originals, and care little for copies. We feel deceived if we discover that objects we believed to be the work of the master, like Shakespeare’s first folios or Rembrandt’s studies, turn out to be the work of a conman, or even just an apprentice. 

These tools are already becoming ubiquitous. The challenge will be to figure out how to negotiate our relationship with them. A plausible middle ground between the full automation of graphic design and the rejection of AI’s influence in artistic domains may lie in the integration of these tools into the designer’s toolkit, just as some designers already use image generators in-house to assist or speed up prototyping.

Turning to more nefarious uses of ad libitum image generation, some readers might wonder, Won’t this make it easy to create “deep fakes” of people doing incriminating things? What about its potential for generating harmful images? We already have a misinformation problem, won’t this make it worse? 

Here’s the corporate answer: Such use is strictly prohibited by OpenAI, and the development team is actively working on ways of detecting subtle uses of its algorithm that constitute content violations. 

But if we’ve collectively learned any lesson from the period 2016 - 2021, it’s that social media is not to be trusted and its developers even less. Content filters are never solutions, only trade-offs. We will never be able to completely pre-empt the use of these tools for dubious ends. Rather, it will require the continuous re-tuning of our intuitions. It may be that people have already developed a healthy suspicion of what they see online. As these tools become distributed across the internet and in our daily lives, that internal detector will need constant maintenance. 

Harrison again: “When early photographers experimented with staged effects (e.g. the Cottingley Fairies) many viewers were convinced as to their veracity, but later generations developed much better abilities to distinguish real from fake images. It might be that, with time, we become less impressed by the algorithm's ability to emulate the immediate stylistic impressions of particular artists, and care more about other abilities that remain (for now) the domain of the human artist, for example conceptual creativity or stylistic innovation.”

Turning to the benefits it could confer: what could DALL-E make easy that is currently very hard? In theory, systems like GPT-3 and DALL-E could be used in any domain where a problem is easy to formulate, but difficult to envision or impractical to implement.

NLPs can quickly process immense amounts of text, and a DALL-E variant could do the same for images. The use cases for this simple task are endless. Today, scholars, editors, and researchers in many fields lament that the rate of publishing is now too high for individuals to have a comprehensive understanding, or even a practicable overview, of their field. 

The issue is: how can we safely incorporate artificial intelligence into consequential decision-making processes?

With the aid of powerful text and image comprehension tools, the scope of literary and historical research may suddenly expand exponentially (e.g. find all mentions of the phrase “silent as the grave” in English written history10). It can bootstrap research in science, whether the problem in question requires broad text search (e.g. conducting a thorough review of the literature for a specific topic) or the study and comparison of visual features (e.g. for taxonomists or anatomists, find all known images of dacetine ants). 

Medicine and law are just two fields where the volume of reference material is immense, the time and resources of practitioners limited, and the decisions highly consequential. Could AI be of assistance here? 

We can take an example from the field of radiology. Expert radiologists are able to detect “gists” of cancerous tissue years before the lesions characteristic of the disease become visible in mammogram screenings. Yet, such screenings yield false-positive and false-negative rates that are higher than what’s desirable. Currently, the development of AI tools to assist in the early detection of breast cancer is a field of active research and would ideally be used to complement human expertise. 

Imagine now that a specially-trained version of DALL-E could be prompted to generate examples of what pre-cancerous breast tissue would look like for a given patient, provided with all the particularities relevant to the case. Such a tool could make explicit the cues that expert radiologists pick up on in their screenings, and provide a pedagogical tool for initial and ongoing training of practitioners. In the reverse direction, an image-processing system could scan for and raise issues about the screening. Such a failsafe could be integrated into standard office software, so as only to nudge human intuition if something has been detected by the engine but not yet addressed by the human user.

The promise of perfection in medical care is intoxicating. Yet, the thought experiment immediately raises a maze of ethical tripwires and new puzzles to solve: Can DALL-E’s images be considered a robust form of knowledge? Where and when can they be used to assist in making real world decisions? How do we implement safeguards into the underlying mechanisms that constrain the model in desirable ways without discouraging its “creativity”?

The issue, more broadly stated, is: how can we safely incorporate artificial intelligence into consequential decision-making processes?

Here, we return to the fundamental issue of alignment. We do not yet know how to effectively “teach” machines to reason in line with human interests and desires. This is also in part an impossible task, for we are frequently in the dark ourselves about where our motives lie and what trade-offs we are willing to make. 

GPT-3, DALL-E, and related algorithms are promising tools, but which problems they are best suited to solving and how we can safely scale them for general use remains unclear. 

Conclusions

In sum, DALL-E (and DALL-E 2) constitutes an impressive implementation of natural language processing and image generation models. Moreover, its surprising flexibility and “creativity,” given the poverty of the input it receives has generated much food for thought for AI researchers and the public alike. Yet, what tools like DALL-E can or should be used for in the wider world remains an open, important, and delicate question. For now, let us heed the example of the sorcerer’s apprentice, and play with caution. 

My earliest best friend from school is named Jan. Later in high school, three out of four people in my group of closest friends were also called Jan. In university, I had three flatmates whose name was, I kid you not, also Jan1. We all had middle names that we used to differentiate ourselves, so throughout my childhood and teenage years, I was always called Jan Hendrik. It was very weird for me when I moved further south and people called me just Jan because there were no other Jans around. I almost felt like I was missing something.

Why am I bringing this up? No particular reason.

Let's put the punchline first: A few weeks ago, I joined the Alignment Team at OpenAI! 🎉 Everything is super exciting, and everybody is super nice to me! I'm working remotely now, but I hope to move to the bay area soon. Moving to the bay appears to be a very hip thing among EAs, so I’m looking forward to meeting as many interesting people as possible.

I have a great story to tell about how this new job came about, involving me teaching myself how to code in the hope of building an AI, reading about the AI box experiment, and studying brains to understand how cognition comes about. But I'm not a big fan of personal mythologies2, so I'll admit that I simply got seduced3 by one of the most interesting4 problems out there. And I’m pretty happy about getting to work on that full-time.

My new job is also why I've been uncharacteristically quiet for the last two weeks on this Substack. The problem is not that I am "too busy to write". I have previously professed my belief that I want to do all the things to maximize positive impact5. I haven’t changed my mind about that; the problem is that there is currently a lot of "renovation" going on in my head. I'm moving a lot of neuroscience knowledge from working memory to my cerebral basement, and I'm doing a lot of updates on all kinds of beliefs. I am working on some more posts, and I have more collaborations in the pipeline, but I'm running out of swap space very fast6.

So, I’ll be back, etc etc7! The IANniversary is coming up, I want to write some retrospective on my PhD, and there is a post on Hebbian learning and the covariance rule that feels relevant for modularity. I’m not ready to stop.

Oh, and I've always wanted to say this, but it never made sense before. But now I can. Ahem: The views expressed on this blog are my own and do not represent the views of my employer. Wow, that felt good.

AI Safety is a rapidly growing field of research that is singular in its goal: to avoid or mitigate negative outcomes from advanced AI. At the same time, AI Safety research touches on many aspects of human lives: from the philosophy of human values, via the neuroscience of human cognition, to the intricacies of human politics and coordination. This interplay between a singular goal with multiple facets makes the problem intrinsically interdisciplinary and warrants the application of various tools by researchers with diverse backgrounds.

Both of us (Sam & Jan) have backgrounds in cognitive science and neuroscience (we'll use the blanket term "brain enthusiasts" from here on). Important characteristics of brain enthusiasts are (see if these apply to you, dear reader):

• A propensity for empirical data and experiments.

• Regarding coding and mathematics as tools rather than as ends in themselves.

• Having accumulated semi-random facts about some biological systems.

While being brain enthusiasts undoubtedly makes us biased in favor of their importance, it also gives us a (hopefully) useful inside view of how brain enthusiasts might meaningfully contribute to AI Safety1. In this post, we attempt to shine a light on the current representation of brain enthusiasts in AI Safety and provide some advice on how brain enthusiasts might enter the field.

A long tail of number-lovers

When you hear terms like "AI Alignment", "AI Safety," or "AGI", you probably think of people with a strong technical background, e.g. in computer science, mathematics, or physics. This is an instance where stereotypes are correct:

Using a previously published dataset, we determined the number of Alignment Forum posts per researcher and sorted the resulting table. We then added a "Background" column, where we noted the field in which the researcher obtained their most recent degree2. As observed previously, the number of published posts is a long-tailed distribution3, with Stuart Armstrong dominating over everyone.

Almost everyone on the list has a computer science, physics, or mathematics background. Notable exceptions are three researchers with an "officially different" formal education (Abram Demski, Richard Ngo, and Daniel Kokotajlo) and those researchers who now work on brain-enthusiasm topics:

1. Steven Byrnes Intro to Brain-like AGI Safety post series

2. John Wentworth and his background in molecular and cellular biology

3. People not included in the list above, like Lee Sharkey or Carla Zoe Cremer, have a computational neuroscience background.

We'd wager the bet that most people on the list don't identify with their "formal education" too much and regard themselves primarily as AI Safety researchers - but still, the uniformity of backgrounds is striking. For someone at the beginning of their career who doesn't have the option to major in "AI Safety", is a major in mathematics the next best thing? Is a strong technical background necessary or merely sufficient?

How did we get here?

There are (at least) two4 possible explanations for why the composition of the field is biased in favor of researchers with technical backgrounds:

• Either, there is an "efficient market" dynamic whereby only people with a technical background make progress and stay in the field,

• Or there is a "founder effect" dynamic whereby the first people entering the field had an outsized effect on the composition of the field later on.

Between these two poles, there is an entire spectrum of what "solving” AI Safety might require:

• AI Safety almost exclusively requires theoretical research. Answers to "foundational questions" (like agent foundations research) tend to require a lot of mathematical and technical knowledge about different theories (see MIRI research guide). This research direction is monolithically important, so a strong technical background is strictly necessary. Brain enthusiasts with arithmophobia might never have a place in AI Safety.

• AI Safety mostly requires theoretical research but also some empirical research. "Foundational questions" are the most urgent issue, but we anticipate that the situation might change soon-ish. If alignment foundations are established (or look promising), engineers will have a day in the sun, as someone needs to implement these foundations in SOTA AI systems. Brain enthusiasts who picked up solid coding skills "somewhere" (or switched career paths to obtain these skills) might be able to help if there's a shortage of software engineers.

• AI Safety requires some theoretical research and some empirical research. Beyond "foundational questions", which might still be very important, there are also empirical questions about the properties of current & future AI. Answers to these empirical questions are critical for making progress in AI Safety at large. Brain enthusiasts might be well-positioned to investigate empirical questions about future AI (moderate additional training required) and current AI (solid coding skills required).

• AI Safety almost exclusively requires empirical research. We are deeply confused about what AGI is and what its existence would imply. We also won't be able to answer these questions with theoretical constructs; the ultimate arbiter of truth is experiment, so we have to (figuratively) "go out there and observe". The empty string is not sufficient input. Brain enthusiasts are well-positioned (no additional training required) to study AGI as it emerges and to describe how it might be dangerous.

The poles of this spectrum appear highly unlikely - pure theory without implementation is as useless as pure observation without foresight5. But the intermediate scenarios appear plausible, and we'll dive into them more in the next two sections.

Recommendations: the bear case

AI Safety mostly requires theoretical research but also some empirical research.

In this scenario, AI Safety is mostly a technical problem, and the current distribution of backgrounds is near-optimal. To contribute, you want to have solid coding and at least rudimentary math skills. You might start by applying for the AI Safety camp and then for the ML for Alignment Bootcamp. Do some Coursera courses on Deep Learning, and maybe study some good old-fashioned AI. Follow the steps outlined by CharlieRS here. Don't feel like you have to figure everything out before you can "get started properly" (you'll pick up things along the way), but realize that technical skills are crucial to contribute to the problem meaningfully.

Your brain enthusiasm background won't be sufficient to contribute to the field. Cognitive science is kind of a mess, the study of neuroscience generates insights with endless caveats that have little hope of generalizing to superhuman minds, and mainstream philosophy of mind is… still stuck on that zombie thing. Brain-like AGI is not on the table. People have no incentive to follow a blueprint for making AGI more "brain-like", and they might be correct not to follow it. Our confusion about how the brain works is even deeper than our confusion about AI Safety. Trying to build brain-like AGI is in the same category as "mysterious answers to mysterious questions".

However, your brain enthusiasm background is not something to be ashamed of either. Realize that you might know more about your field of study (the amygdala? meerkat's mating rituals? the extended mind hypothesis?) than the median researcher you encounter. Sometimes this is a problem ("law of the instrument", when all you have is a hammer…), but sometimes the correct response to "my area of expertise seems relevant here; why hasn’t anyone mentioned meerkats yet?"  actually is "I should mention meerkats and explain why they are relevant". Some reasonably well-understood portions of biological intelligence can map onto artificial intelligence. As long as you're able to explain yourself in the common (technical) vocabulary, your contribution will be valuable6.

Recommendations: the bull case

AI Safety requires some theoretical research and some empirical research.

In this scenario, AI Safety could benefit from more empirical input and the current distribution of backgrounds is an artifact produced by something like the founder effect. To contribute, you might have to learn some basic coding skills and technical vocabulary. But don't spend too much time on this. Your comparative advantage lies elsewhere. Consider applying for the PIBBSS fellowship, but also consider "just start experimenting". Request research access to the GPT API and ask the model some pointed questions, grab a dataset and investigate the eye color of the prominent researchers in the field7, or see what you can learn about myopia by looking at people with anterograde/retrograde amnesia.

Your brain enthusiasm background is your great strength, and it might appear counterintuitive how many low-hanging fruits are for you to pick. Usually, low-hanging fruits are bad (or someone else would have picked them) - but AI Safety might genuinely be in a non-equilibrium state, and some very impactful projects might require little effort. Simply writing a post about what AI Safety looks like through the lens of something you know well could be valuable. It is corny beyond all reasonableness - but you should adopt the abundance mindset.

However, even though your brain enthusiasm background is da shit, please don't start spamming the Alignment Forum. You might genuinely have important insights, but the ability to communicate them is now more important than ever. Without a shared vocabulary, you will want to invest much effort into expressing yourself well. Use simple sentences. Include many examples. Be patient. And prepare for the possibility that nobody will interact with what you produce for a long time. Even though, as per the assumption of this subsection, your brain enthusiast background is useful, the onus is still on you to demonstrate in what way this is true.

What now? A call to action.

In this section, we want to give you different options and recommendations for projects and topics you could contribute to, depending on whether you are more of a bear or a bull8. Consider these "mini-project proposals". If you'd like to work on one of those, feel free to reach out! Perhaps we can help!

Forecasting AI capabilities and analyzing AI takeoff scenarios. One approach for forecasting timelines until the emergence of AGI is to anchor the analysis with what we know about the computational capacity of the human brain. This "Possible further investigation" section might be a great starting point.

Neuroscience could play a big role in deciphering human values. Standard paradigms of reward learning attempt to infer human values by looking at their choices. This probably doesn't capture everything there is to capture. Cognitive Science and Neuroscience have a rich literature of experiments and frameworks on this topic. Existing reports by Luke Muehlhauser or Kaj Solata are great starting points, but there is a lot more that can be extracted from the literature.

Interpretability (or Transparency and Explainability research). Some AI Researchers believe that Interpretability is the "most valuable current research direction within the class of standard ML research". According to Olah, if you have been trying to interpret neural systems up until now, you might find this work considerably easier. A lot of data analysis techniques from computational neuroscience or cognitive science might translate straightforwardly to artificial neural networks. Nonnegative matrix factorization, Granger causality, and seedpoint correlation analysis might all be worth a shot.

Brain-Like AGI Safety. Steven Byrnes, whom we have already mentioned earlier, has written a sequence on brain-like AGI Safety". He believes that there are two paths towards aligned AI where a brain enthusiast's perspective could help: "controlled AGIs" (carefully designed and monitored AGI goals and motivations) and "social-instincts AGIs" (reverse-engineered human social instincts) (see post 12 for more info on this). Engaging with his articles (even if only through distillation) appears potentially very high impact.

Human-Computer Interaction. Assuming AGI doesn't immediately foom, the way humanity interacts with AGI might become critical. Studying human susceptibility to manipulation could provide hints at what bounded AI-assisted warfare might look like. Studying historical changes in values might help anticipate what intentional and unintentional value corruption might look like. Thinking about how humans can leverage existing AI tools might provide a critical headstart for optimally designing future safety technology.

Playing with fire. Some people used to think it was plausible that AGI can be achieved by copying the brain (although the approach is falling out of favor9). But even if the secret to AGI is not buried in your neuroanatomy textbook, through some type of convergent evolution AGI might share some features of the human brain. Modularity appears like a plausible candidate. Perhaps there are others?

Exploring directions we haven't thought of. Maybe you already had ideas when you began reading this article. Great! The above list isn't intended to constrain your creativity; don't be afraid to try out things. In fact, we believe that the most impact of a brain-enthusiast AI safety project could be exactly this: contributing an unusual perspective to AI Safety research10.

If you are still not convinced, consider the information value gained through failing. High-variability projects are lucrative for grantmakers (more information value gained for future grantmaking), giving you a good shot at getting funding (there is a small probability that the projects mentioned above work out).

The bottom line

We were torn between two extremes when composing this post: "the brain is useless" and "the brain is all you need". Between these extremes, we found a spectrum of research with varying technical and empirical components. Eventually, we decided to present both a pessimistic (bear case) and an optimistic (bull case) of how brain enthusiasts might fit into AI Safety research. What became clear in both scenarios is that there is value in variability, and a nonstandard background can be an advantage more often than not.

While we wrote in an authoritative voice in some places, please read all of this with an appropriate (spoonful or more) amount of salt. While we feel somewhat good about the conclusion that variability can be beneficial, the exact route for brain enthusiasts to contribute to AI Safety is fuzzy. Going down this path will require careful attention to the environment - few have walked it before. Be careful not to trample into a well-kept garden.

We look forward to any questions and/or comments below, or via mail to either of us (you can find our contact info on our blogs: Sam & Jan).

Meta: In this project, we collected and cataloged AI alignment research literature and analyzed the resulting dataset in an unbiased way to identify major research directions. We found that the field is growing quickly, with several subfields emerging in parallel. We looked at the subfields and identified the prominent researchers, recurring topics, and different modes of communication in each. Furthermore, we found that a classifier trained on AI alignment research articles can detect relevant articles that we did not originally include in the dataset.

Dataset Announcement

In the context of the 6th AISC, we collected a dataset of alignment research articles from a variety of different sources. This dataset is now available for download here and the code for reproducing the scrape is on GitHub here1. When using the dataset, please cite our manuscript as described in the footnote2.

Here follows an abbreviated version of the full manuscript, which contains additional analysis and discussion.

Rapid growth of AI Alignment research from 2012 to 2022 across two platforms

After collecting the dataset, we analyzed the two largest non-redundant sources of articles, Alignment Forum (AF) and arXiv. We found rapid growth in publications on the AF (Fig. 1a) and a long-tailed distribution of articles per researcher (Fig. 1b) and researchers per article (Fig. 1c). We were surprised to find a decrease in publications on the arXiv in recent years, but identified the cause for the decrease as spurious and fixed the issue in the published dataset (details in Fig. 4).

Unsupervised decomposition of AI Alignment research into distinct clusters

Given access to this unique dataset, we were curious to see if we could identify distinct clusters of research. We mapped the title + abstract of each article into vector form using the Allen Institute for AI's SPECTER model and reduced the dimensionality of the embedding with UMAP (Fig. 2a). The resulting manifold shows a continuum of AF posts and arXiv articles (Fig. 2b) and a temporal gradient from the top right to the bottom left (Fig. 2c). Using k-means and the elbow method, we obtain five clusters of research articles that map onto distinct regions of the UMAP projection (Fig. 2d).

We were curious to see if the five clusters identified by k-means map onto existing distinctions in the field. When identifying the most prolific authors in each cluster, we noticed strong differences3 (consistent with previous work that suggests that author identity is an important indicator of research direction).

By skimming articles in each cluster and given the typical research published by the authors, we suggest the following putative descriptions of each cluster:

1. cluster one: Agent alignment is concerned with the problem of aligning agentic systems, i.e. those where an AI performs actions in an environment and is typically trained via reinforcement learning.

2. cluster two: Alignment foundations research is concerned with deconfusion research, i.e. the task of establishing formal and robust conceptual foundations for current and future AI Alignment research.

3. cluster three: Tool alignment is concerned with the problem of aligning non-agentic (tool) systems, i.e. those where an AI transforms a given input into an output. The current, prototypical example of tool AIs is the "large language model".

4. cluster four: AI governance is concerned with how humanity can best navigate the transition to advanced AI systems. This includes focusing on the political, economic, military, governance, and ethical dimensions.

5. cluster five: Value alignment is concerned with understanding and extracting human preferences and designing methods that stop AI systems from acting against these preferences.

We note that these descriptions are chosen to be descriptive, not prescriptive. Our approach has the advantage of being (comparatively4) unbiased and can therefore serve as a baseline against which other (more prescriptive) descriptions of the landscape can be compared (Krakovna's paradigms, FLI landscape, Christiano's landscape, Nanda's overview, ...). Discrepancies between these descriptions and ours can serve as important information for funding agencies (to identify neglected areas) and AI Governance researchers (for early identification of natural categories for regulation).

Research dynamics vary across the identified clusters

We further note some properties of the identified clusters (Fig. 3a). The cluster labeled as "alignment foundations" contains most of the seminal work in the field (Fig. 3b,c), but remains largely disconnected from the more applied "agent alignment" and "tool alignment" research (Fig. 3a). Furthermore, most "alignment foundations" work is published on the Alignment Forum (Fig. 3d) and it has the largest inequality in terms of "number of articles per researcher" (Fig. 3e). This corroborates an observation that was made before: While critically important, alignment foundations research appears to be poorly integrated into more applied alignment research, and the research remains insular and pushed by comparatively few researchers.

Leveraging dataset to train an AI alignment research classifier

After having identified the five clusters, we returned to the issue we noted at the onset of our analysis: the apparent decrease in publications on the arXiv in recent years (Fig. 1a). We were skeptical about this and hypothesized that our data collection might have missed relevant recent articles5. Therefore, we trained a logistic regression classifier to distinguish alignment articles (level-0) from articles cited by alignment articles (level-1) (Fig.4 a). The resulting classifier achieved good performance and generalized well to papers from unrelated sources (Fig. 4b). We then scraped all the articles from the arXiv cs.AI category and asked our classifier to score them (Fig. 4c,d). Based on the distribution of scores of Alignment Forum posts (Fig. 4d) and after skimming the relevant articles, we chose a threshold of 75% as a reasonable trade-off between false positives and false negatives.

When adding the arXiv articles above the cutoff to our dataset, we observed a rapid increase in publications also on the arXiv (Fig. 4e). To test if our clustering is robust to this increase, we repeated the UMAP projection with the updated dataset and found that, indeed, the clusters are still in distinct regions of the manifold (Fig. 4f). Interestingly, the added literature appears to fill some of the gaps between "alignment foundations" and "agent alignment" research.

Closing remarks

The primary output from our project is the curated dataset of alignment research articles. We hope the dataset might serve as the basis for

• a semantic search service that returns relevant literature (see prototype here).

• writing assistants in the form of fine-tuned large-language models.

• projects to preserve AI Safety research in case of catastrophic events.

If you have other ideas for how to use the dataset, please don't hesitate to reach out to us; we're excited to help.

Furthermore, we hope that the secondary outcome from our project (the analysis in this post) can aid both funding agencies and new researchers entering the field to orient themselves and contextualize the research. 

As we plan to continue this line of research, we are happy about any and all feedback on the dataset and the analysis, as well as hints and pointers about things we might have missed.

Acknowledgments: We thank Daniel Clothiaux for help with writing the code and extracting articles. We thank Remmelt Ellen, Adam Shimi, and Arush Tagade for feedback on the research. We thank Chu Chen, Ömer Faruk Şen, Hey, Nihal Mohan Moodbidri, and Trinity Smith for cleaning the audio transcripts.

Constraints make creative

Originally, my motivation for getting into neuroscience was: "Building artificial intelligence is difficult, so perhaps I can learn something from looking at how the biological brain is built. Once I've figured that out, AI will be easy".

(When I read these words I can't help but add a mental "duuuuh".)

Since those innocent days, I have come to appreciate that building a biological brain is a ridiculously overconstrained problem.

1. First off - even before you're done building, the brain must make itself useful. And it's not just "occasionally helping with an arithmetic problem" - I'm talking about "breathing", "eating", "coordinating limbs"... If the environment is dangerous, then also "running away" or "fighting" is on the menu. And you need to do these things reliably - lapsing means death.

2. Second - every neuron only has access to local information. In particular, every neuron only receives input from a limited number of other neurons (usually < 10,000 out of ~80 billion). From that information, a neuron needs to infer what its role in the developing brain ought to be. Consequently, every solution you come up with must be "symmetric" in the sense that you can swap two neurons, and the end result must still work. (It turns out the two neurons don't even have to be from the same species).

3. Third - there is no clear separation between "the thing you're building" and "the thing that is doing the building"1. The brain is making itself with only rough guidance from genes and the environment (see this post I wrote on the topic a while ago).

In contrast, the solution for AI appears to be (for now) "just stack more layers":

In AI, "the thing being built" and "the thing that's doing the building" is cleanly separated into "model" and "optimizer". Global information like the learning rate is readily available to all units. And even after training, there are no predators that punish lack of robustness2.

But constraints make creative, and it seems like the aforementioned constraints turn out to have their benefits:

1. First - An adversarial environment forces the brain to be robust against perturbations. Both theory and practice tell us that with robustness comes the ability to generalize to novel domains. (Intense adversarial dynamics might, in fact, explain the huge gap between monkeys and humans in terms of generality.)

2. Second - A neat consequence of locality is modularity. When you try to use as little "wire" as possible to connect your neurons, locally interconnected modules tend to emerge. Beyond making things slightly more interpretable, modularity can also increase some types of robustness.

3. Third - ... This one beats me.

What is the benefit of having to build yourself? I'm not sure. Let's zoom in on that.

Bootstrapping brains

What does it mean for the brain to be building itself?

As the brain is growing, it is dynamically responding to cues. Removing input to one eye and the visual cortex balances the loss by upregulating input from the other. Adding an additional eye results in... well, this:

When the brain does this type of rearranging, it is both the thing that "notices" that something is wrong, and it's also the thing that "is being modified".

Third-eye shenanigans are only one (illustrative) example of the dynamic nature of development. Throughout development, immature neural circuits produce activity that induces activity-dependent formation/removal of synapses, which changes the circuit, which leads to changed activity, ... The circuit is "the thing you're building," and its activity is "the thing that is doing the building".

I've briefly written about this dynamic before, but now I want to take a deeper dive because a thing I've been working on with colleagues for a while is finally out as a preprint.

Let's start by building some intuition by looking at pretty pictures. We start with one of my favorite recordings of the optic tectum of a zebrafish developing over 16 hours:

After soaking in this beauty for a bit, I would like to ask you to look closely at some of the tiny protrusions. You'll notice that the smaller protrusions tend to probe the environment a bit before settling for a direction to grow in.

Niell and colleagues made that observation eight years earlier, but their camera wasn't as nice.

On the plus side, they got a very close look at an individual dendritic tree of a neuron in the optic tectum. And importantly, they genetically manipulated the fish so that its synapses glow green. So they were able to look closely at the interactions between dendrite growth and synapse formation.

Now, this is where the plot thickens. Synapses are the site of transmission of electric nerve impulses between two nerve cells, so synapses live at the intersection of the circuit (defined by the locations of synapses) and its activity (the signal transmitted by the synapse). Therefore, we might expect that the dynamics of synapse development should be pretty important.

And that's indeed the conclusion that Niell and colleagues arrive at. They look very closely at a developing branch, and they notice that once a synapse (a green blob) forms, the developing dendrite tends to anchor to that synapse. So it seems that during early development, the initial synapses a dendrite makes can act as a type of scaffold.

Probably prunable

But not all development is about growth! Rather, it's about growth plus selective pruning. I've written about a potential molecular mechanism underlying this dynamic before, and now we'll see it in action.

Right out the gate - even though there can be a lot of pruning, this typically doesn't mean that the dendrite gets shorter in total3. Typically, retractions of branches are balanced by the addition of new branches. Here are pictures from a timelapse from a Xenopus tadpole neuron:

What determines if a branch stabilizes or gets pruned? It's kind of unclear - but let's collect the evidence.

1. First, here's a sequence of pictures from the retina of a baby chicken. The pictures show a dendrite in the baseline (left), when activity is blocked (middle), and when activity is allowed again.
   

   

   The dendrite retracts rapidly when the activity stops and then extends again when the activity resumes. So dendrite retraction can be triggered by activity-dependent mechanisms.

2. Second, here's a dendrite in the optic tectum of the Xenopus tadpole. The tadpole is trained with a visual stimulus, and the dendrite shows new protrusions that form in response to the training. Importantly, these protrusions are responsive to the stimulus that the tadpole was trained with.

   

   So specific activity can trigger the growth of dendrites.

3. Third, the "Sherlock and Moriarty of molecular neuroscience" also get a starring role in this story. In this study, researchers incubate rat brain neurons in either BDNF (matBDNF) or proBDNF (CR-proBDNF) and find that the signaling molecules dramatically affect neuronal morphology.

   

   (As perhaps expected,) BDNF increases dendrite length and complexity, while proBDNF leads to almost complete dendrite removal.

4. Finally, we had pretty strong intuitions about how synapses ought to interact with each other from previous work. In particular, there's a bunch of neat experimental evidence that synapses that fire "out-of-sync" tend to "lose their link". Or, in less poetic terms, nearby synapses tend to compete.

   

Computerization

Those were the puzzle pieces we knew about (or guessed) when we started the project. It seemed plausible that synapse formation and removal might have a pretty big impact on what the dendrite looks like. And in particular, whether a synapse stabilizes or not might be determined by whether it fires in synchrony with its neighbors.

When we put these ideas into a model, we get something like the following video:

Initially, the dendrites shoot outwards and search for synaptic connections (synapses not shown in the video). Initially, the dendrite is not picky at all - it just connects with any potential partner it finds. This results in a rapid increase in total length that nicely mirrors the increase in length observed in biology (again from a Xenopus tadpole).

However, now something interesting happens both in the model and in biology; dendrite growth tapers off and stabilizes!

As I've shown you before, this stability in total length does not mean that there is no change in the dendrite structure anymore; rather, the addition and removal of branches are balanced in a fixed point/steady state. This is also something we observe in the model:

But why is it that the dendrite "suddenly" stops growing? The answer is unclear from the biological data, but our model gives a neat explanation: Dendrite growth stops because competition between synapses is starting!

Due to the dynamics of the competition, it took a while for synapses to "get to know their neighbors" and decide whether they were well-synchronized. But once they figured it out, synaptic pruning starts, and the dendrite cannot simply add new synapses at random anymore. Thus, growth slows down, and many poorly synchronized synapses get removed. What remains are only well-synchronized synapses:

This way, our model can link the development of dendrites and the emergence of synaptic organization in one framework.

In the paper, we push things further and show how our model fits in with more experiments and how the resulting dendrites have some cool optimality properties that emerge from a simple synapse-centric growth rule. But for this post, it's time to wrap things up and to return to our original question: How can the brain be, at the same time, "the thing you're building" and "the thing that is doing the building"? And could this be good for anything?

Closing thoughts

I feel the strong urge to do a bunch of epistemic hedging, "theory is theory and experiment is experiment," and "I could be totally wrong about this", but I think that's pretty bad style. Instead, I’ll say that this is my current best guess, and if I’m wrong I’ll be appropriately surprised and embarrassed.

Instead of doing more hedging, I’ll stick my neck out even further and talk about “the delta”:

What do I believe now about “the brain that builds itself”?

1. A straightforward way to obtain structured connectivity in a neural circuit is to start with a greedy strategy (connect as many things as possible), followed by an autoregressive4 learning rule that amplifies structure and prunes away the noise.

2. Through successive amplification of structure and pruning of noise, the developmental process converges to a fixed point, defined as the point where the learning rule produces no more competition.

The “benefit” of this dynamical systems solution is (again) “robustness”. As long as you make sure your learning rule keeps on doing its thing it doesn’t matter if you get small perturbations. You’ll keep rolling towards the valley of attraction anyways.

Sticking my neck out even further, I can’t help but think of the natural abstraction hypothesis here. Perhaps a lot of the cognitive structure that humans have does not actually require a lot of specification during development. Perhaps the fixed point of the developmental process is “natural” in the sense that any autoregressive learner with sensory input from the world5 (and with approximately similar computational capacity) will end up with approximately the same cognitive structure.

This moves the question from the world of neuroscience into the world of… the world? What is the world out there like, and why does it have so much structure? I’ll think about it a bit and update you on my progress ;)

3

This image floating around the internet gets used all over the place to argue that there is something special happening in the brain of humans around the age of 5-6. The image comes originally from Seeman, 1999:

It might well be that something special happens at that age, but I cannot find a single other study that replicates the image on the left. In fact, a pretty credible Nature Review shows an increasing complexity up to age 24.

Also, note that the caption of the Seeman figure talks about a decline in synaptic connections, which are not even visible on the left. And the right shows a decline in dopamine receptors, but with a very suspicious x-axis and very few data points in the most critical period.