Back in early 2016 I bet at even odds that self-driving ride sharing would be available in 10 US cities by July 2023. Then I made similar bets a dozen times because everyone disagreed with me.

The first deployment to potentially meet our bar was Phoenix in 2022. I think Waymo is close to offering public rides in SF, and there are a few more cities being tested, but it looks like it will be at least a couple of years before we get 10 cities even if everything goes well.

Back in 2016 it looked plausible to me that the technology would be ready in 7 years. People I talked to in tech, in academia, and in the self-driving car industry were very skeptical. After talking with them it felt to me like they were overconfident. So I was happy to bet at even odds as a test of the general principle that 7 years is a long time and people are unjustifiably confident in extrapolating from current limitations.

In April of 2016 I gave a 60% probability to 10 cities. The main point of making the bets was to stake out my position and maximize volume, I was obviously not trying to extract profit given that I was giving myself very little edge. In mid 2017 I said my probability was 50-60%, and by 2018 I was under 50%.

If 34-year-old Paul was looking at the same evidence that 26-year-old Paul had in 2016 I think I would have given it a 30-40% chance instead of a 60% chance. I had only 10-20 hours of information about the field, and while it’s true that 7 years is a long time it’s also true that things take longer than you’d think, 10 cities is a lot, and expert consensus really does reflect a lot of information about barriers that aren’t easy to articulate clearly. 30% still would have made me more optimistic than a large majority of people I talked to, and so I still would have lost plenty of bets, but I would have made fewer bets and gotten better odds.

But I think 10% would have been about as unreasonable a prediction as 60%. The technology and regulation are mature enough to make deployment possible, so exactly when we get to 10 cities looks very contingent. If the technology was better then deployment would be significantly faster, and I think we should all have wide error bars about 7 years of tech progress. And the pandemic seems to have been a major setback for ride hailing. I’m not saying I got unlucky on any of these—my default guess is that the world we are in right now is the median—but all of these events are contingent enough that we should have had big error bars.

Lessons

People draw a lot of lessons from our collective experience with self-driving cars:

• Some people claim that there was wild overoptimism, but this does not match up with my experience. Investors were optimistic enough to make a bet on a speculative technology, but it seems like most experts and people in tech thought the technology was pretty unlikely to be ready by 2023. Almost everyone I talked to thought 50% was too high, and the three people I talked to who actually worked on self-driving cars went further and said it seemed crazy. The evidence I see for attributing wild optimism seems to be valuations (which could be justified even by a modest probability of success), vague headlines (which make no attempt to communicate calibrated predictions), and Elon Musk saying things.
• Relatedly, people sometimes treat self-driving as if it’s an easy AI problem that should be solved many years before e.g. automated software engineering. But I think we really don’t know. Perceiving and quickly reacting to the world is one of the tasks humans have evolved to be excellent at, and driving could easily be as hard (or harder) than being an engineer or scientist. This isn’t some post hoc rationalization: the claim that being a scientist is clearly hard and perception is probably easy was somewhat common in the mid 20th century but was out of fashion way before 2016 (see: Moravec’s paradox).
• Some people conclude from this example that reliability is really hard and will bottleneck applications in general. I think this is probably overindexing on a single example. Even for humans, driving has a reputation as a task that is unusually dependent on vigilance and reliability, where most of the minutes are pretty easy and where not messing up in rare exciting circumstances is the most important part of the job. Most jobs aren’t like that! Expecting reliability to be as much of a bottleneck for software engineering as for self-driving seems pretty ungrounded, given how different the job is for humans. Software engineers write tests and think carefully about their code, they don’t need to make a long sequence of snap judgments without being able to check their work. To the extent that exceptional moments matter, it’s more about being able to take opportunities than avoid mistakes.
• I think one of the most important lessons is that there’s a big gap between “pretty good” and “actually good enough.” This gap is much bigger than you’d guess if you just eyeballed the performance on academic benchmarks and didn’t get pretty deep into the weeds. I think this will apply all throughout ML; I think I underestimated the gap when looking in at self-driving cars from the outside, and made a few similar mistakes in ML prior to working in the field for years. That said, I still think you should have pretty broad uncertainty over exactly how long it will take to close this gap.
• A final lesson is that we should put more trust in skeptical priors. I think that’s right as far as it goes, and does apply just as well to impactful applications of AI more generally, but I want to emphasize that in absolute terms this is a pretty small update. When you think something is even odds, it’s pretty likely to happen and pretty likely not to happen. And most people had probabilities well below 50% and so they are even less surprised than I am. Over the last 7 years I’ve made quite a lot of predictions about AI, and I think I’ve had a similar rate of misses in both directions. (I also think my overall track record has been quite good, but you shouldn’t believe that.) Overall I’ve learned from the last 7 years to put more stock in certain kinds of skeptical priors, but it hasn’t been a huge effect.

The analogy to transformative AI

Beyond those lessons, I find the analogy to AI interesting. My bottom line is kind of similar in the two cases—I think 34-year-old Paul would have given roughly a 30% chance to self-driving cars in 10 cities by July 2023, and 34-year-old Paul now assigns roughly a 30% chance to transformative AI by July 2033. (By which I mean: systems as economically impactful as a low-cost simulations of arbitrary human experts, which I think is enough to end life as we know it one way or the other.)

But my personal situation is almost completely different in the two cases: for self-driving cars I spent 10-20 hours looking into the issue and 1-2 hours trying to make a forecast, whereas for transformative AI I’ve spent thousands of hours thinking about the domain and hundreds on forecasting.

And the views of experts are similarly different. In self-driving cars, people I talked to in the field tended to think that 30% by July 2023 was too high. Whereas the researchers working in AGI who I most respect (and who I think have the best track records over the last 10 years) tend to think that 30% by July 2033 is too low. The views of the broader ML community and public intellectuals (and investors) seem similar in the two cases, but the views of people actually working on the technology are strikingly different.

The update from self-driving cars, and more generally from my short lifetime of seeing things take a surprisingly long time, has tempered my AI timelines. But not enough to get me below 30% of truly crazy stuff within the next 10 years.

• Kindness: You care about the other player.
• Correlation. Your decision is correlated with the other player’s decision.
• Reciprocity. Your decision is correlated with the other player’s belief about your decision.

Kindness makes common sense, but correlation and reciprocity are often lumped together under “weird acausal stuff.” I think they are worth distinguishing because they work quite differently.

I’ll talk about details of correlation and reciprocity, and then argue that most incentives to cooperate in the true prisoner’s are likely to come from interactions of multiple mechanisms.

Setup: the truly one-shot prisoner’s dilemma

Imagine that one day our civilization sees another through a wormhole—we’ve never met before, and we are far enough away from each other that after today we will never meet again. From each civilization, a single person is chosen to play a game.

The two players then play a one-shot prisoner’s dilemma with survival as the stakes. Each player chooses to cooperate in which case they die with probability 1%, or defect in which case the other play dies with probability 2%. If both players cooperate they each die with 1% chance, if both defect they each die with 2% chance.

No matter what the other player does, you have a lower probability of death if you defect. In this post we analyze reasons that you might cooperate anyway.

I will not discuss the incentives to make commitments in advance based on the possibility that we may play a prisoner’s dilemma in the future (even though I think those incentives are important). I will analyze the question mostly for an agent who uses EDT or UDT.

You could just as well replace the ratio 2%/1% with any X>1, and ask “how large does X need to be before I’d cooperate?” I think it is pretty robust that you should cooperate at some sufficiently large scale (unless you specifically want the other player to die) and so this is ultimately a quantitative question. For this post I’ll just keep X=2 and leave it up to the reader to generalize.

Kindness

The most boring reason to cooperate is that I don’t want the other player to die.

That might be because I value their life and experiences for its own sake, in the same way that I value mine. Or it could be that I mostly care about other consequences of survival—the effect a death will have on the people around us, the things we will accomplish if we survive, and so on—and I think their survival will have nearly as many good consequences as mine.

Let K be how much I value the other player’s survival. If K > 0.5 then I should cooperate. (Since then 2% * K > 1%.)

Correlation

When my deliberation begins, I’m not sure what I’ll end up deciding. I’m also not sure what the other player will decide. Those two unknown quantities are slightly correlated because our decisions have some common structure: what happens when someone is whisked away from their everyday life and spends a day deciding how nice to be to someone from an alien civilization?

A causal decision theorist doesn’t care about this correlation, but an evidential decision theorist considers it a reason to cooperate. We can measure the strength of the correlation as C = P(they cooperate | I cooperate) – P(they cooperate | I defect). If C > 0.5 then I should cooperate.

Smaller correlations give us a partial reason to cooperate, and they combine linearly with kindness: if K + C > 0.5 I should cooperate.

It’s really hard to estimate C. It’s probably not zero but it’s plausible that it’s extremely small. Much of the correlation is likely to be screened off by other stuff I’ve seen (e.g. observing other people playing the prisoner’s dilemma).

The effective correlation is likely to be (much) larger for someone using UDT. I do think altruistic agents should use UDT, see EDT with updating double counts and Decision theory and dynamic consistency, and therefore I think this can be a significant consideration.

Reciprocity

If the other player is very different from me, my decision may not be very correlated with their decision. But if they understand me well enough, my decision might be very correlated with their prediction about my decision.

If the other player is using a policy like “I cooperate only with people who I predict will cooperate with me” then I would care a lot about their prediction and so this could give an evidential decision-theorist reason to cooperate.

You can think of “I cooperate only with people who I predict will cooperate with me” as a kind of logical tit for tat. I know what I predict the other player would do. That prediction involves predicting what they’d predict I’d do, which in turn involves predicting what they’d predict that I’d predict they’d do, and so on. If the real world is just another representative entry in this sequence, then cooperating based on whether I predict my opponent will cooperate is a lot like cooperating based on whether they cooperated in the last step, i.e. if I predict they’d cooperate.

I find it more convincing to think of this as a kind of logical trade. In a usual physical trade, I have a policy like “I will give you a toaster if you give me $1.” I have that policy because I believe you will give me $1 if I have that policy and not otherwise. The logical version is just the same. It’s easiest to make that arrangement when you and I can actually talk. But we can also do the same thing with lower fidelity based only on predictions. To an EDT agent the size of the incentive depends on the quality of predictions. To a CDT agent, the incentive only kicks in when the other player’s reasoning about me takes the form of a simulation such that “I” might be in the simulation. For the rest of the post I’ll consider only EDT or UDT agents.

Let R be my overall estimate for the difference between P(other player cooperates | I cooperate) and P(other player cooperates | I defect) that comes the channel of my decision affecting their prediction which in turn affects their action. We’ll change the definition of C to be the “direct” correlation that comes from the similarity of our decisions rather than through this prediction channel.

This factor combines additively with kindness and correlation, so I should cooperate if R+C+K > 0.5.

(This is a simplification, because I should also myself use a reciprocity-like strategy where my based on my prediction about whether they are cooperating based on a reciprocity-like strategy… but I’m going to keep setting aside those complexities. Program equilibrium via provability logic gives a taste for some of the issues.)

Correlation + Kindness

Correlation and kindness also have an important nonlinear interaction, which is often discussed under the heading of “evidential cooperation in large worlds” or ECL.

Consider all of the pairs of civilizations and people who might have been randomly selected to play this prisoner’s dilemma. For each of them, we can define K and C as in the last sections. We can also think of K=C=1 for ourselves.

Let’s imagine that this crazy wormhole-prisoner’s-dilemma situation arises periodically throughout the universe, and that the universe is very large (e.g. because of quantum branching). Even if K and C are extremely small on average, the sum of K and C across all other players can be astronomically large.

To an evidential decision theorist, that means:

• Because the sum of K is large, most of what I care about in the universe is not this particular game of the prisoner’s dilemma, but the sum over all the other games.
• Because the sum of C is large, most of the difference between the world conditioned on “I cooperate” and the world conditioned on “I defect” is not the fact that I in particular cooperate but the correlation with all the other games.

So when I cooperate, the costs and benefits are effectively distributed across the universe, rather than being concentrated on me and the other player. The calculus is no longer “Is R+C+K > 0.5?”. Instead, conditioned on me cooperating:

• The extra fraction of players who cooperate is E[C].
• The extra fraction of players who survive from cooperation is 2% * E[C].
• My utility from that is 2% * E[C] * E[K], because the cost is distributed across all players.
• The extra fraction of players who die because they cooperated is 1% * E[C].
• My disutility from that is 1% * E[C * K], because the cost is distributed based on C.

Thus the question is whether the correlation between C and K is large enough. In a small universe where K and C are usually small, the correlation between K and C can be huge—I myself am an example of a player with K=C=1, and all of E[C], E[K] and E[C*K] may be equal to “what fraction of all players do I represent?”. But once the sum of K and C get very large, I personally make almost no contribution to E[C] and E[K], and so the correlation becomes dominated by features of other civilizations.

Overall I think that this may give impartial players (who mostly care about people other than themselves) a strong reason to cooperate with each other even if they have very different values and don’t intrinsically care about one another. For selfish players, i.e. whose who mostly care about the future experiences of observers very similar to themselves, it’s a much weaker reason to cooperate. Most humans would be a mix of the two and so it depends on the parameters.

This calculation is pretty subtle and I’ve glossed over a lot of details. One particularly important issue is that C is not just a property of a civilization. In fact it makes most sense for me to cooperate with players who have the kind of values that someone who is correlated with me might have had. And in that case, I could end up making very different decisions than a copy of me who was playing against a rock. Sorting through all those complexities is complicated and does change the recommended policy, but the net effect seems to be to create a generally stronger reason to cooperate with other impartial players.

Reciprocity + correlation + kindness

Intuitively there are two factors limiting the strength of reciprocity motivations for cooperation:

1. Are the other player’s predictions correlated with my behavior?
2. Does the other player implement a reciprocity-like policy? (Or would they, if they thought I might?)

Condition 1 seems quite hard to achieve, especially in civilizations like ours that don’t have sophisticated technology for making predictions. So I expect reciprocity on its own to not be much of a motive in the true prisoner’s dilemma unless the players have an extremely good ability to reason about one another—it doesn’t require going all the way to literal simulations, but does require much more ability than we have today. (Note that the situation changes if we are able to make commitments: in that case it may be natural to commit to a trade-like policy, allowing us to cooperate with other people who make similar commitments by making the prediction problem easier.)

But I think that combining reciprocity with correlation+kindness can create a significant reason to cooperate even when players are not able to make very accurate predictions about one another. In this section we’ll try to analyze that effect. We’ll assume that we are playing against someone who is using a reciprocity-like strategy such that they are more likely to cooperate if they think that I will cooperate. If the other player is only possibly using a reciprocity-like strategy, or only has a weak tendency to cooperate if they think I will cooperate, then the conclusions will get scaled down.

We can model the other players’ limited prediction ability as uncertainty about exactly who I am: they’ve seen some facts about me, but there are lot of possibilities consistent with what they’ve seen. Some of those possibilities cooperate and some of them defect. (This is only an approximation, which I’m using here to give a rough sketch. It does apply to logical uncertainty as well as empirical uncertainty, though in some situations we will care about correlation+kindness across logically impossible worlds which is a more complicated discussion.)

Suppose that from the other player’s perspective I belong to a set of possible players S, and that they have well-calibrated beliefs about S. Then conditioned on one additional player from S cooperating, the other player has a 1/|S| higher probability that each player in S cooperates—the effect of their limited prediction ability is to “smear out” the update across all of the indistinguishable-to-them players.

Conditioned on my cooperation, the fraction of cooperating players in S increases by E[C], which we now define as the average correlation between my decision and the decision from someone else in S who is facing a player using a reciprocity-like strategy (where the correlation is computed my perspective). That then increases the frequency with which people cooperate with players in S by E[C].

The benefits of this cooperation are distributed across all the players in S, and I care based on E[K], resulting in a total benefit I care about of 2% * E[C] * E[K]. The costs are distributed across the players in S proportional to C, and so the cost I care about is 1% * E[C * K].

This leads us to a calculus very similar to the correlation+kindness analysis in the last section: it’s worth cooperating only if the correlation between C and K is weaker than 0.5 * (the extent to which the other player reciprocates). However the correlation is now being computed over the set S, of people who look like me as far as the other player is concerned.

Conditioning on what the other player knows about me will tend to reduce the correlation between C and K, potentially greatly. One way to think about this is that the C-K correlation is driven by some common causes—perhaps my decision and values are both more correlated with overthinking things. To compute the C-K correlation conditioned on S, we remove any of those common causes that are observed by the other player. This won’t significantly reduce correlations coming from selfishness, but is likely to be a very big deal for impartial values.

The bottom line is that if I am facing a player who uses a reciprocity-like strategy, such that they will cooperate more if they expect me to cooperate, and if I have impartial values and the other player knows this fact, then reciprocity+correlation+kindness can give a very big reason to cooperate: if the other player is a weak predictor then the benefits of cooperation will spill over to lots of other people who “look like me” from their perspective, but the costs are also shared across the other people who look like me, and I probably care about as much about the people who get the benefits as the people who pay the costs.

I think the best reason for the other person to run a reciprocity-like strategy is because they expect that you are more likely to cooperate if they do. In practice both players are using a policy like X = “cooperate if the other player uses a policy like X.” So I expect this kind of cooperation to get off the ground when both players have sufficiently impartial values and are inclined to follow this argument (and learn enough about each other to verify those facts).

I think this can greatly strengthen the correlation+kindness effect even while you have weak predictors and may often lead to a net incentive to cooperate for impartial updateless decision theorists. For agents who update / have selfish values, such cooperation probably requires the ability to carry out more detailed simulations.

I think the most dicy step of this argument was modeling the other player’s uncertainty as ignorance about my identity; we’ve modeled them as thinking that I could be anyone in S but having correct beliefs about S on average. Concretely, we’re imagining something like: all the other player can tell is that they are playing against someone who tends to overthink things. I believe that my decision is very slightly correlated with the average behavior of players who overthink things, and I think that the other player will end up with well-calibrated beliefs about the class of players who overthink things. So my decision has the same strength of correlation with their beliefs as it does with the truth, which gives us reason to cooperate no matter how small in absolute value both correlations are. But if that 1:1 assumption doesn’t hold (because the other player can’t reason accurately about people who overthink things), then we broaden the set S, potentially including logically impossible versions of “players who overthink things.” I think this modeling choice is much more plausible for a UDT than an EDT agent, because a UDT agent hasn’t yet updated on any of the factors driving the other player’s belief. But even for a UDT agent this might be wrong and requires more thought.

Conclusion

There seems to be pretty rich decision-theoretic structure in the true prisoner’s dilemma, and some reasons to cooperate will involve a multiplicative interaction between different mechanisms that are often lumped together as “weird acausal stuff.” In particular, I think understanding the strength of reciprocity-like arguments for cooperation requires thinking about the dynamics of evidential cooperation in large worlds (at least until you reach the limit involving literal simulations).

This post informally describes my current best guesses and the very rough analysis behind them; those have a minor effect on my attitudes to some odd philosophical questions, but I wouldn’t personally trust them to be correct or robust. Some of the underlying correlations and frequencies across the universe seem almost impossible to assess, but I do think it’s feasible to analyze the basic situation much more carefully, and that could still make a real difference in our best guesses. There’s a good chance that the basic picture in this post may be completely wrong, and conversely if a more careful analysis had similar conclusions then I’d feel much more comfortable with them.

• When making a decision after observing X, we should condition (or causally intervene) on statements like “My decision algorithm outputs Y after observing X.”
• Updating seems like a description of something you do when making good decisions in this way, not part of defining what a good decision is. (More.)
• Causal reasoning likewise seems like a description of something you do when making good decisions. Or equivalently: we should use a notion of causality that captures the relationships relevant to decision-making rather than intuitions about physical causality. (More.)
• “How much do I care about different copies of myself?” is an arbitrary question about my preferences. If my preferences change over time, it naturally gives rise to dynamic inconsistency unrelated to decision theory. (Of course an agent free to modify itself at time T would benefit by implementing some efficient compromise amongst all copies forked off after time T.)

In this post I’ll discuss the last bullet in more detail since I think it’s a bit unusual, it’s not something I’ve written about before, and it’s one fo the main ways my view of decision theory has changed in the last few years.

(Note: I think this topic is interesting, and could end up being relevant to the world in some weird-yet-possible situations, but I view it as unrelated to my day job on aligning AI with human interests.)

The transparent Newcomb problem

In the transparent version of Newcomb’s problem, you are faced with two transparent boxes (one small and one big). The small box always contains $1,000. The big box contains either $10,000 or $0. You may choose to take the contents of one or both boxes. There is a very accurate predictor, who has placed $10,000 in the big box if and only if they predict that you wouldn’t take the small box regardless of what you see in the big one.

Intuitively, once you see the contents of the big box, you really have no reason not to take the small box. For example, if you see $0 in the big box, you know for a fact that you are either getting $0 or $1,000. So why not just take the small box and walk away with $1,000? EDT and CDT agree about this one.

I think it’s genuinely non-obvious what you should do in this case (if the predictor is accurate enough). But I think this is because of ambiguity about what you want, not how you should make decision. More generally, I think that the apparent differences between EDT and UDT are better explained as differences in preferences. In this post I’ll explain that view, using transparent Newcomb as an illustration.

A simple inconsistent creature

Consider a simple creature which rationally pursues its goals on any given day—but whose goals change completely each midnight. Perhaps on Monday the creature is trying to create as much art and beauty as possible; on Tuesday it is trying to create joy and happiness; on Wednesday it might want something different still.

On any given day we can think of the creature as an agent. The creature on Tuesday is not being irrational when it decides to pursue joy and happiness instead of art and beauty. It has no special reason to try to “wind back the clock” and pursue the same projects it would have pursued on monday.

Of course on Monday the creature would prefer to arrest this predictable value drift—it knows that on Tuesday it will be replaced with a new agent, one that will stop contributing to the project of art and beauty. The creature on Monday ought to make plans accordingly, and if they had the ability to change this feature of themselves they would likely do so. It’s a matter of semantics whether we call this creature a single agent or a sequence of agents (one for each day).

This sequence of agents could benefit from cooperating with one another, and it can do so in different ways. Normal coordination is off the table, since causality runs only one way from each agent to the next. But there are still options:

• The Tuesday-creature might believe that its decision is correlated with the Monday-creature. If the Tuesday-creature tries to stop the Wednesday-creature from existing, then the Monday-creature might have tried to stop the Tuesday-creature from existing. If the correlation is strong enough and stopping values change is expensive, then the Tuesday-creature is best served by being kind to its Wednesday-self, and helping to put it in a good position to realize whatever its goals may be. (Though note that this can unwind just like an iterated prisoner’s dilemma with finite horizon!)
• The Tuesday-creature might believe that its decision is correlated with the Monday-creature’s predictions about what the Tuesday-creature would do. If the Tuesday-creature keeps on carrying out the Monday-creature’s plans, then the Monday-creature would be more motivated to help the Tuesday-creature succeed (and less motivated to try to prevent the value change). If the Monday-creature is a good enough predictor of the Tuesday-creature, then the Tuesday-creature is best served by at least “paying back” the Monday-creature for all of the preparation the Monday-creature did.

However none of these relationships are specific to the fact that it is the same creature on Monday and Tuesday; the fact that the cells are the same has no significance for the decision-theoretic situation. The Tuesday-creature has no intrinsic interest in the fact that it is not “reflectively stable”—of course that instability definitionally implies a desire to change itself, but not a further reason to try to help out the Monday-creature or Wednesday-creature, beyond the relationships described above.

A human inconsistency

I care a lot about what is going to happen to me in the future. I care much more about my future than about different ways that the world could have gone (or than my past for that matter). In fact I would treat those other possible versions of myself quite similarly to how I’d treat another person who just happened to be a lot like me.

This leads to a clear temporal inconsistency, which is so natural to humans that we don’t even think of it as an inconsistency. I’ll try to illustrate with a sequence of thought experiments.

Suppose that at 7AM I think that there is a 50% chance that a bell will ring at 8AM. At 7AM I am indifferent between the happiness of Paul-in-world-with-bell and Paul-in-silent-world. If you asked me which Paul I would prefer to stub his toe, I would be indifferent.

But by 8:01AM my preferences are quite different. After I’ve heard the bell ring, I care overwhelmingly about Paul-in-world-with-bell. I would very strongly prefer that the other Paul stub his toe than that I do.

Some people might say “Well you just cared about what happens to Paul, and then at 8AM you learned what is real. Your beliefs have changed, but not your preferences.” But consider a different experiment where I am duplicated at 7AM and each copy is transported to a different city, one where the bell would ring and the other where it will not. Until I learn which city I’m in, I’m indifferent between the happiness of Paul-in-city-with–bell and Paul-in-silent-city. But at the moment when I hear the bell ring, my preferences shift.

Some people could still say “Well you cared about the same thing all along—what happens to you—and you were merely uncertain about which Paul was you.” But consider the Paul from before the instant of copying, informed that he is about to be copied. That Paul knows full well that he cares about both copies. Yet sometime between the copying and the bell Paul has become much more parochial, and only cares about one. It seems to me like there is little way to escape from the inconsistency here.

One could still say “Nonsense, all along you just cared about what happened to you, you were just uncertain about which of the copies you were going to become.” I find this very unpersuasive (why think there is a fact of the matter about who “I” am?), but at this point I think it’s just a semantic dispute. Either my preferences change, or my preferences are fixed but defined in terms of a concept like “the real me” whose meaning changes. It all amounts to the same thing.

This is not some kind of universal principle of rationality—it’s just a fact about Paul. You can imagine different minds who care about all creatures equally, or who care only about their own future experiences, or who care about all the nearby possible copies of themselves. But I think many humans feel roughly the same way I do—they have some concern for others (including creatures very similar to themselves in other parts of the multiverse), but have a very distinctive kind of caring for what they themselves will actually experience in the future.

Altruism is more complicated

In the examples above I discussed stubbing my toe as the unit of caring. But what if we had instead talked of dollars? And what if I am a relatively altruistic person, who would use marginal dollars to try to make the world better?

Now in the case of two copies in separate cities it is clear enough that my preferences never change. I’m still willing to pay $1 to give my counterpart $2. After all, they can spend those dollars just as well as I can, and I don’t care who it was who did the good.

But in the case of a single city, where the bell either rings or it doesn’t, we run into another ambiguity in my preferences—another question about which we need not expect different minds to agree no matter how rational they are.

Namely: once I’ve heard the bell ringing, do I care about the happiness of the creatures in the world-with-bell (given that it’s the real world, the one we are actually in), or do I care about the happiness of creatures in both worlds even after I’ve learned that I happen to be in one of them?

I think people have different intuitions about this. And there are further subtle distinctions, e.g. many people have different intuitions depending on whether the ringing of the bell was a matter of objective chance (where you could imagine other copies of yourself on far away worlds, or other branches, facing the same situation with a different outcome), or a matter of logical necessity where we were simply ignorant.

While some of those disagreements may settle with more discussion, I think we should be able to agree that in principle we can imagine a mind that works either way, that either care about people in other worlds-that-could-have-been or who don’t.

Most humans have at least some draw towards caring only about the humans in this world. So the rest of my post will focus on their situation.

Back to transparent Newcomb (or: The analogy)

Consider again a human playing the transparent version of Newcomb’s problem. They see before them two boxes, a small one containing $1000 and a big one containing $0. They are told that the big box would have contained $10000 if a powerful predictor had guessed that they would never take the small box.

If the human cares only for their own future experiences, and would spend the money only on themselves, they have a pretty good case for taking the small box and walking away with $1000. After all, their own future experiences are either going to involve walking away with $1000 or with nothing, there is no possible world where they experience seeing an empty big box and then end up with the money after all.

Of course before taking the big box, the human would have much preferred to commit to never taking the small box. If they are an evidential decision theorist, they could also have just closed their eyes (curse that negative value of information!). That way they would have ended up with $10,000 instead of $1,000.

Does this mean that they have reason to take nothing after all, even after seeing the box?

I think the human’s situation is structurally identical to the inconsistent creature whose preferences change at midnight. Their problem is that in the instant when they see the empty big box, their preferences change. Once upon a time they cared about all of the possible versions of themselves, weighted by their probability. But once they see the empty big box, they cease to care at all about the versions of themselves who saw a full box. They end up in conflict with other very similar copies of themselves, and from the perspective of the human at the beginning of the process the whole thing is a great tragedy.

Just like the inconsistent creature, the human would have strongly preferred to make a commitment to avoid these shifting preferences. Just like the inconsistent creature, they might still find other ways to coordinate even after the preferences change, but it’s more contingent and challenging. Unlike the inconsistent creature, they can avoid the catastrophe by simply closing their eyes—because the preference change was caused by new information rather than by the passage of time.

The situation is most stark if we imagine the predictor running detailed simulations in order to decide whether to fill the big box. In this case, there is not one human but three copies of the human: two inside the predictor’s mind (one who sees an empty box and one who sees a full box) and one outside the predictor in the real world (seeing an empty or full box based on the results of the simulation). The problem for the human is that these copies of themselves can’t get along.

Even if you explained the whole situation to the human inside the simulation, they’d have no reason to go along with it. By avoiding taking the small box, all they can achieve is to benefit a different human outside of the simulation, who they no longer care at all about. From their perspective, better to just take the money (since there’s a 50% chance that they are outside of the simulation and will benefit by $1000).

(There are even more subtleties here if these different possible humans have preferences about their own existence, or about being in a simulation, or so on. But none of these change the fundamental bottom line.)

Altruism is still more complicated

If we consider a human who wants to make money to make the world better, the situation is similar but with an extra winkle.

Now if we explain the situation to the inside human, they may not be quite so callous. Instead they might reason “If I don’t take the small box, there is a good chance that a ‘real’ human on the outside will then get $10,000. That looks like a good deal, so I’m happy to walk away with nothing.”

Put differently, when we see an empty box we might not conclude that predictor didn’t fill the box. Instead, we might consider the possibility that we are living inside the predictor’s imagination, being presented with a hypothetical that need not have any relationship to what’s going on out there in the real world.

The most extreme version of this principle would lead me to entertain very skeptical / open-minded beliefs about the world. In any decision-problem where “what I’d do if I saw X” matters for what happens in cases where X is false, I could say that there is a “version” of me in the hypothetical who sees X. So I can never really update on my observations.

This leads to CDT=EDT=UDT. For people who endorse that perspective (and have no indexical preferences), this post probably isn’t very interesting. Myself, I think I somewhat split the difference: I think explicitly about my preferences about worlds that I “know don’t exist,” roughly using the framework of this post. But I justify that perspective in significant part from a position of radical uncertainty: I’m not sure if I’m thinking about worlds that don’t exist, or if it’s us who don’t exist and there is some real world somewhere thinking about us.

Conclusion

Overall the perspective in this post has made me feel much less confused about updatelessness. I expect I’m still wrong about big parts of decision theory, but for now I feel tentatively comfortable using UDT and don’t see the alternatives as very appealing. In particular, I no longer think that updating feels very plausible as a fundamental decision-theoretic principle, but at the same time don’t think there’s much of a reflective-stability-based argument for e.g. one-boxing in transparent Newcomb.

Most of the behaviors I associate with being “updateless” seem to really be about consistent preferences, and in particular continuing to care about worlds that are in some sense inconsistent with our observations. I believe my altruistic preferences are roughly stable in this sense (partially justified by a kind of radical epistemic humility about whether this is the “real” world), but my indexical preferences are not. The perspective in this post also more clearly frames the coordination problem faced by different copies of me (e.g. in different plausible futures) and I think has left me somewhat more optimistic about finding win-win deals.

People sometimes object to evidential decision theory by saying: “It seems like the distinction between correlation and causation is really important to making good decisions in practice. So how can a theory like EDT, with no role for causality, possibly be right?”

Long-time readers probably know my answer, but I want to articulate it in a little bit more detail. This is essentially identical to the treatment of causality in Eliezer Yudkowsky’s manuscript Timeless Decision Theory, but much shorter and probably less clear.

Causality and conditional independence

If a system is well-described by a causal diagram, then it satisfies a complex set of statistical relationships. For example:

• In the causal graph A ⟶ B ⟶ C, the variables A and C are independent given B.
• In the graph A ⟶ B ⟵C, the variables A and C are independent, but are dependent given B.

To an evidential decision theorist, these kinds of statistical relationships are the whole story about causality, or at least about its relevance to decisions. We could still ask why such relationships exist, but the answer wouldn’t matter to what we should do.

EDT = CDT

Now suppose that I’m making a decision X, trying to optimize Y.

And suppose further that there is a complicated causal diagram containing X and Y, such that my beliefs satisfy all of the statistical relationships implied by that causal diagram.

Note that this diagram will necessarily contain me and all of the computation that goes into my decision, and so it will be (much) too large for me to reason about explicitly.

Then I claim that an evidential decision theorist will endorse the recommendations of CDT (using that causal diagram):

• EDT recommends maximizing the conditional expectation of Y, conditioned on all the inputs to X. Write Z for all of these inputs.
  • It might be challenging to condition on all of Z, given limits on our introspective ability, but we’d recommend doing it if possible. (At least for the rationalist’s interpretation EDT, which evaluates expected utility conditioned on a fact of the form “I decided X given inputs Z.”)
  • So if we can describe a heuristic that gives us the same answer as conditioning on all of Z, then an EDT agent will want to use it.
  • I’ll argue that CDT is such a heuristic.
    • Edited to add: This is wrong or at least badly incomplete. I don’t think it matters to the main point of this post (that EDT does “normal-looking causal inference” in normal cases). But it’s pretty central for the actual live philosophical debates about EDT v CDT v TDT.
    • In particular, it’s true that we’d like to condition on all of Z, but if we lack introspect access to parts of Z then this procedure won’t do that. It ignores effects via Z but doesn’t actually know the values in Z, so there’s no real reason to ignore those effects. Actually handling this issue is very subtle and has been discussed a lot. I think it’s fine if you use any algorithm A that conditions on A() = X, but in general it’s very messy to talk about algorithms that take facts as inputs without knowing those facts.
    • This is a lot of what people are interested in when discussing the EDT vs CDT comparison, since I think everyone understands the basic point in this post. I think that the logical facts about what EDT outputs are on a different footing than the other inputs in Z (since e.g. you can’t build an agent that knows them, whereas updating on all the inputs to Z is extremely reasonable) but they are often treated similarly in the decision theory literature.
• In a causal diagram, there is an easy graphical condition (d-connectedness) to see whether (and how) X and Y are related given Z:
  • We need to have a path from X to Y that satisfies certain properties:
  • That path can start out moving upstream (i.e. against the causal arrows); it may switch from moving upstream to downstream at any time (including at the start); it must switch direction whenever it hits a node in Z; and it may only switch from moving downstream to upstream when it hits a node in Z.
• If Z includes exactly the causal parents of X, then it’s easy to check that the only way for X and Y to be d-connected is by a direct downstream path from X to Y.
• Under these conditions, it’s easy to see that intervening on X is the same as conditioning on X. (Indeed you could check this more directly from the definition of a causal intervention, which is structurally identical to conditioning in cases where we are already conditioning on all parents.)

Moreover, once the evidential decision-theorist’s problem is expressed this way, they can remove all of the causal nodes upstream of X, since they have no effect on the decision. This is particularly valuable because that contains all of the complexity of their own decision-making process (which they had no hope of modeling anyway).

So if the EDT agent can find a causal structure that reflects their (statistical) beliefs about the world, then they will end up making the same decision as a CDT agent who believes in the same causal structure.

Whence subjective causality?

You might think: causal diagrams encode a very specific kind of conditional independence structure. Why would we see that structure in the world so often? Is this just some weird linguistic game we are playing, where you can rig up some weird statistical structure that happens to give the same conclusions as more straightforward reasoning from causality?

Indeed, one easy way to have statistical relationships is to have “metaphysically fundamental” causality: in a world containing many variables, each of which is an independent stochastic function of its parents in some causal diagram, then those variables will satisfy all the conditional independencies implied by the that causal diagram.

If this were the only way that we got subjective causality, then there’d be no difference between EDT and CDT, and no one would care about whether we treated causality as subjective or metaphysically fundamental.

But it’s not. There are other sources for similar statistical relationships. And moreover, the “metaphysically fundamental causality” isn’t actually consistent with the subjective beliefs of a logically bounded agent.

We can illustrate both points with the calculator example from Yudkowsky’s manuscript:

• Suppose there are two calculators, one in Mongolia and one on Neptune, each computing the same function (whose value we don’t know) at the same instant.
• Our beliefs about the two calculators are correlated, since we know they compute the same function. This remains true after conditioning on all the physical facts about the two calculators.
• But in the “metaphysically fundamental” causal diagram, the results of the two calculators should be d-separated once we know the physical facts about them (since there isn’t even enough time for causal influences to propagate between them).
• We can recover the correct conditional independencies by adding a common cause of the two calculators, representing “what is the correct output of the calculation?” We might describe this as “logical” causality.

This kind of “logical” causality can lead to major deviations from the CDT recommendation in cases where the EDT agent’s decision is highly correlated with other facts about the environment through non-physically-causal channels. For example: if there are two identical agents, or if someone else is reasoning about the agent’s decision sufficiently accurately, then the EDT agent would be inclined to say that the logical facts about their decision “cause” physical facts about the world (and hence induce correlations), whereas a CDT agent would say that those correlations should be ignored.

Punchline

EDT and CDT agree under two conditions: (i) we require that our causal model of the world and our beliefs agree in the usual statistical sense, i.e. that our beliefs satisfy the conditional independencies implied by our causal model, (ii) we evaluate utility conditioned on “I make decision X after receiving inputs Z” rather than conditioning on “I make decision X in the current situation” without including relevant facts about the current situation.

In practice, I think the main way CDT and EDT differ is that CDT ends up in a complicated philosophical discussion about “what really is causality?” (and so splinters into a host of theories) while EDT picks a particular answer: for EDT, causality is completely characterized by condition (i), that our beliefs and our causal model agree. That makes it is obvious how to generalize causality to logical facts (or to arbitrary universes with very different laws), while recovering the usual behavior of causality in typical cases.

I believe the notion of causality that is relevant to EDT is the “right” one, because causality seems like a concept developed to make and understand decisions (both over evolutionary time and more importantly over cultural evolution) rather than something ontologically fundamental that is needed to even define a correct decision.

If we take this perspective, it doesn’t matter whether we use EDT or CDT. I think this perspective basically accounts for intuitions about the importance of causality to decision-making, as well as the empirical importance of causality, while removing most of the philosophical ambiguity about causality. And it’s a big part of why I don’t feel particularly confused about decision theory.

Calculator bet: I am offered the opportunity to bet on a mathematical statement X to which I initially assign 50% probability (perhaps X = 139926 is a quadratic residue modulo 314159). I have access to a calculator that is 99% reliable, i.e. it corrupts the answer 1% of the time at random. The calculator says that X is true. With what probability should I be willing to wager?

I think the answer is clearly “99%.” But a naive application of EDT can recommend betting with 99.99% probability. I think this is a mistake, and understanding the mistake helps clarify what it means to be “updateless” and why it’s essentially obligatory for EDT agents. My takeaway is that for an EDT agent, bayesian updating is a description of the expected utility calculation rather than something that EDT agent should do to form its beliefs before calculating expected utility.

Thanks to Joe Carlsmith and Katja Grace for the conversation that prompted this post. I suspect this point is well-known in the philosophy literature. I’ve seen related issues discussed in the rationalist community, especially in this sequence and this post but found those a bit confusing—in particular, I think I initially glossed over how “SSA” was being used to refer to a view which rejects bayesian updating on observations (!) in this comment and the linked paper. In general I’ve absorbed the idea that decision theory and anthropics had a weird interaction, but hadn’t noticed that exactly the same weirdness also applied in cases where the number of observers is constant across possible worlds.

Why EDT bets at 99.99% odds (under some conditions)

I’ll make four assumptions:

• I have impartial values. Perhaps I’m making a wager where I can either make 1 person happy or 99 people happy—I just care about the total amount of happiness, not whether I am responsible for it. I’ll still describe the payoffs of the bets in $, but imagine that utility is a linear function of total $ earned by all copies of me.
• We live in a very big universe where many copies of me all face the exact same decision. This seems plausible for a variety of reasons; the best one is accepting an interpretation of quantum mechanics without collapse (a popular view).
• I handle logical uncertainty in the same way I handle empirical uncertainty. You could construct a similar case to the calculator bet using logical uncertainty, but the correlation across possible copies of me is clearest if I take a logical fact.
• I form my beliefs E by updating on my observations. Then after updating I consider E[utility|I take action a] and E[utility|I take action a’] and choose the action with higher expected utility.

Under these assumptions, what happens if someone offers me a bet of $1 at 99.9% odds? If I take the bet I’ll gain $1 if X is true, but lose $1000 if X turns out to be false? Intuitively this is a very bad bet, because I “should” only have 99% confidence. But under these assumptions EDT thinks it’s a great deal.

• To calculate utility, I need to sum up over a bunch of copies of me.
  • Let N be the number of copies of me in the universe who are faced with this exact opportunity to bet decision.
  • My decision is identical to the other copies of me who also observed their calculator say “X is true”.
  • My decision may also be correlated with copies of me who made a different observation, or with totally different people doing totally different things, but those don’t change the bottom line and I’ll ignore them to keep life simple.
  • So I’ll evaluate the total money earned by people who saw their calculator say “X is true” and whose decision is perfectly correlated with mine.
• To calculate utility, I calculate the probability of X and then calculate expected utility
  • First I update on the fact that my calculator says X is true. This observation has probability 99% if X is true and 1% if X is false. The prior probability of X was 50%, so the posterior probability is 99%.
  • My utility is the total amount of money made by all N copies of me, averaged over the world where X is true (with 99% weight) and the world where X is false (with 1% weight)
• So to calculate the utility conditioned on taking the bet, I ask two questions:
  • Suppose that X is true, and I decide to take the bet. What is my utility then?
    If X is true, there are 0.99 N copies of me who all saw their calculator correctly say “X is true.” So I get $0.99 N
  • Suppose that X is false, and I decide to take the bet. What is my utility then?
    If X is false, then there are 0.01N copies of me who saw their calculator incorrectly say “X is true.” So I lose $1000 * 0.01N = $10N.
  • I think that there’s a 99% probability that X is true, so my expected utility is 99% x $0.99N – 1% x $10N = $0.88N.
• If I don’t take the bet, none of my copies win or lose any money. So we get $0 utility, which is much worse than $0.88N.
• Therefore I take the bet without thinking twice.

Failure diagnosis

Intuitively 99.99% is the wrong answer to this question. But it’s important to understand what actually went wrong. After all, intuitions could be mistaken and maybe big universes lead to weird conclusions (I endorse a few of those myself). Moreover, if you’re like me and think the “obvious” argument for EDT is compelling, this case might lead you to suspect something has gone wrong in your reasoning.

The intuitive problem is that we are “updating” on the calculator’s verdict twice:

• First when we form our beliefs about whether X is true.
• Second when we ask “If X is true, how many copies of me would have made the current observations, and therefore make a decision correlated with my own?”

The second “update” is pretty much inherent in the nature of EDT—if I care about the aggregate fate of all of the people like me, and if all of their decisions are correlated with mine, then I need to perform a sum over all of them and so I will care twice as much about possible worlds where there are twice as many of them. Rejecting this “update” basically means rejecting EDT.

The first “update” looks solid at first, since Bayesian updating given evidence seems like a really solid epistemic principle. But I claim this is actually where we ran into trouble. In my view there is an excellent simple argument for using EDT to make decisions, but there is no good argument for using beliefs formed by condition on your observations as the input into EDT.

This may sound a bit wild, but hear me out. The basic justification for updating is essentially decision-theoretic—either it’s about counting the observers across possible worlds who would have made your observations, or it’s about dutch book arguments constraining the probabilities with which you should bet. (As an example, see SEP on bayesian epistemology.) I’ve internalized these arguments enough that it can feel like a primitive bedrock of epistemology, but really they only really constrain how you should bet (or maybe what “you” should expect to see next)—they don’t say much about what you should “expect” in any observer-independent sense that would be relevant to a utility calculation for an impartial actor.

If you are an EDT agent, the right way to understand discussions of “updating” is as a description of the calculation done by EDT. Indeed, it’s common to use the word “belief” to refer to the odds at which you’d bet, in which case beliefs are the output of EDT rather than the input. Other epistemological principles do help constrain the input to EDT (e.g. principles about simplicity or parsimony or whatever), but not updating.

This is similar to the way that an EDT agent sees causal relationships: as helpful descriptions of what happens inside normatively correct decision making. Updating and causality may play a critical role in algorithms that implement normatively correct decision making, but they are not inputs into normatively correct decision making. Intuitions and classical arguments about the relevance of these concepts can be understood as what those algorithms feel like from the inside, as agents who have evolved to implement (rather than reason about) correct decision-making.

“Updatelessness” as a feature of preferences

On this perspective whether to be “updateless” isn’t really a free parameter in EDT—there is only one reasonable theory, which is to use the prior probabilities to evaluate conditional utilities given each possible decision that an agent with your nature and observations could make.

So what are we to make of cases like transparent newcomb that appear to separate EDT from UDT?

I currently think of this as a question of values or identity (though I think this is dicier than the earlier part of the post). Consider the following pair of cases to illustrate:

• I am split into two copies A and B who will go on to live separate lives in separate (but otherwise identical) worlds. There is a button in front of each copy. If copy A presses the button, they will lose $1 and copy B will gain $2. If copy B presses the button, nothing happens. In this case, all versions of EDT will press the button. In some sense at this point the two copies must care about each other, since they don’t even know which one they are, and so the $1 of loss and $2 of gain can be compared directly.
• But now suppose that copy A sees the letter “A” and copy B sees the letter “B.” Now no one cares what I do after seeing “B,” and if I see “A” the entire question is whether I care what happens to the other copy. The “updateless” answer is to care about all the copies of yourself who made different observations. The normal “selfish” answer is to care about only the copy of yourself who has made the same observations.

This framing makes it clear and relatively uninteresting why you should modify yourself to be updateless: any pair of agents agents could benefit from a bilateral commitment to value each other’s welfare. It’s just that A and B start off being the same, and so they happen to be in an exceptionally good position to make such a commitment, and it’s very clear what the “fair” agreement is.

What if the agents aren’t selfish? Say they both just want to maximize happiness?

• If both agents exist and they are just in separate worlds, then there is no conflict between their values at all, and they always push the button.
• Suppose that only one agent exists. Then it feels weird, seeing button “B,” to press the button knowing that it causes you to lose $1 in the real, actually-existing world. But in this case I think the problem comes from the sketchy way we’re using the word “exist”—if copy B gets money based on copy A’s decision, then in what sense exactly does copy A “not exist”? What are we to make of the version of copy A who is doing the same reasoning, and is apparently wrong about whether or not they exist? I think these cases are confusing from a misuse of “existence” as a concept rather than updatelessness per se.

It would be great to guarantee digital people some control over their situation: 1. to control their local environment and sensations, 2. to avoid unauthorized rewinding or duplicating.

I’ll describe how you could modify the code of a digital person so that they retain this control even if an adversary has access to their source code. This would be very expensive with current cryptography. I think the overhead will eventually become cheap enough that it’s possible to do for some digital people, though it will likely remain expensive enough that it is never applied to most digital people (and with luck most digital people will be able to feel secure for other reasons).

Part 1: the right to control my environment

My ideal

• I live in a comfortable virtual home. I control all of the details of that world.
• When people communicate with me, I can choose how/whether to hear them, and how/whether to update my home based on what they say (e.g. to render an avatar for them)
• Sometimes I may occupy a virtual world where a foreign server determines what I see, feel, or hear. But even then I can place boundaries on my experiences and have the ability to quickly retreat to my home.
• I have as much control as feasible over my own mental state and simulated body. No one else can tamper directly with them.
• I can choose to pause myself for as long as I want (or permanently).
• My local environment is private, and I have access to plenty of tamper-proof storage. I can do whatever I want with computers in my home, including e.g. verifying signatures or carrying on encrypted conversations.

Implementation

1. First we write a simple environment that reflects all my desiderata (the “home”).
2. Then I apply indistinguishability obfuscation to (me + home), so that the house becomes private and tamper-proof. (This is an extremely expensive operation, more on that later.)
3. I distribute the obfuscated home and hopefully destroy any unprotected copies of myself.

One conceptual difficulty is that indistinguishability obfuscation applies to circuits whereas I would like to obfuscate a long-running program. But this can be handled straightforwardly, as discussed in Appendix A.

The home could consume terabytes of memory and teraflops of compute before it added significantly to the expense of running a human-like digital person, so I could live in relative luxury. The home could also negotiate resource requirements with the external world, and to decide what to do when requested resources are unavailable (e.g. to pause until it becomes available).

Limitation 1: cost

Indistinguishability obfuscation is extremely expensive, more like a factor of 10000000000 slowdown than 10.

It will get faster with further research, but probably not fast enough to obfuscate the whole person+home. But there are other ways to speed up the process:

• I think it’s probably possible to have most of the computation be “merely” homomorphically encrypted, and to have an obfuscated controller which verifies and decrypts the results. FHE could be much faster than obfuscation; if I had to guess I’d say it would converge to something like 2-3 orders of magnitude of slowdown.
• We can potentially have an obfuscated controller verify a much larger untrusted computation. I don’t know how fast we can make delegated computation, but I could imagine it getting closer to 2x than 100x. It might help further that we are not applying these methods to generic problems but to a very specific structured problem (which probably has quite low circuit depth). One complication is that we need our proof system to be secure even against an adversary who can unwind the prover, but I don’t think this is a huge deal.
• Delegating computation would preserve integrity but not security. So the computation we delegate may need to already be private. Here it seems likely that we can benefit a lot from the structure of the computation. Almost all of our operations are in doing a brain simulation, and we don’t really care about leaking the fact that we are doing a brain simulation, just about leaking the state of the brain. I don’t know how fast this can be made but again I would not be surprised by a factor of 2.

It’s pretty unclear how fast this could get, either from taking some of these techniques to their limits or from thinking of other cleverer ideas. I would not be at all surprised by getting the whole thing down to a factor of 2 slowdown. That said, I also think it’s quite plausible that you need 10x or 10000x.

Limitation 2: security?

The cryptography used in this construction may end up getting broken—whether from a mistaken security assumption, or because the future contains really giant computers, or because we implemented it badly.

The software used in my home may get compromised even if the cryptography works right. An adversary can provide trillions of malicious inputs to find one that lets them do something unintended like exfiltrate my code. With modern software engineering this would be a fatal problem unless the home was extremely simple, but in the long run writing a secure home is probably easier than writing fast enough cryptography.

I may be persuaded to output my source code, letting an adversary run it. I might not give myself the ability to inspect my own source, or might tie my hands in other ways to limit bad outcomes, but probably I can still end up in trouble given enough persuasion. This is particularly plausible if an adversary can rewind and replay me.

Limitation 3: rewinding

In the best case, this scheme guarantees that an attacker can only use my code as part of a valid execution history. But for classical computers there is no possible way to stop them from running many valid execution histories.

An attacker could save a snapshot of me and then expose it to a billion different inputs until they found one in which I responded in a desired way. (Even if I’m cagey enough to avoid this attack in most possible situations, they just have to find one situation where I let my guard down and then escalate from there.) Or I could have revealed information to the outside world that I no longer remember because I’ve been reset to an earlier state.

Someone living in this kind of secure house is protected from the worst abuses, but they still can’t really trust the basic nature of their reality and are vulnerable to extreme manipulation.

This brings us to part 2.

Part 2: the right to a single timeline

My ideal

• No one should be able to make a second copy of me without my permission, or revert me to a previous state.
• I should be able to fork deliberately. I can’t force someone to run a second copy of me, but I should be able to give specific permission.

Implementation with trusted hardware

This is easy to achieve if we have a small piece of trusted tamper-resistant hardware that can run cheap computations. We use the same mechanism as in the last section, but:

• The trusted hardware has a secret key sk, and it maintains an internal counter k.
• On input x, the trusted hardware signs (x, k) and increments the counter.
• Whenever someone provides my obfuscated controller an input and tries to step it forward, the obfuscated controller first checks to see that the input has been signed by the trusted controller with the correct timestep.
• In order to make a copy, I need to have the public key of another piece of trusted hardware, which I use to initialize a new copy. (Ideally, the manufacturer signs the public key of each piece of trusted hardware they built, and I know the manufacturer’s public key.)

If I were willing to make a round trip to a trusted third party every time I received a novel input, then I could have them implement this function directly instead of using tamper-proof hardware. The real critical ingredient is me trusting someone on the outside. I’ll discuss how to potentially remove this assumption in the section on quantum computers below.

None of this actually requires my house to be built to guarantee the right to a single timeline—I could start without such a right, and then install a wrapper to enforce a single timeline once there was some hardware I trusted or if it became important enough.

Implementation with 1-of-2 transfer

Suppose that the only kind of trusted hardware is a device that holds two secrets, and will reveal one or the other of them when asked but not both. I think this is somewhat easier to build than general trusted hardware. (Related: locking phones with quantum bits.)

Now suppose there is a trusted party who manufactures a bunch of these devices, with a public key pk. Each device a serial number n, and its two secrets are signatures from pk: one of (n, 0) and one of (n, 1)

I can still achieve a single-timelines guarantee:

• At each time, I have a list of 100 secure devices I expect to use to verify my next step. When I am initially created I have to choose 100 devices that someone will need to run me.
• Each time someone wants to step me forward, they need to provide both the input and the serial numbers for 100 new secure devices to use for the next step.
• I then apply a cryptographic hash function to get a 100 bit string. I output this string and wait for a response before stepping forward the simulation.
• They use the nth bit of my string to decide which secret to reveal from their nth device. I verify the signatures for all 100 secrets.
• If the adversary tries to run me with two different inputs, or to have two different sets of 100 devices to use for the next steps, they will end up with two different 100 bit string challenges (unless they find a hash collision). Stepping me forward for both strings would require getting the signatures for two different bits, which is impossible if the secure devices work as intended and the signature scheme remains secure.

Implementation with quantum computers

We can probably remove the need for a trusted party if we have quantum computers big enough to perform delegated cryptographic operations (but no big enough quantum computers to run an entire digital person in superposition).

The vague idea in this section looks to me like it should work—I’d give it at least 85% odds—but would require some new ideas and analysis. It’s on much sketchier ground than any of the ideas in previous sections

The basic idea is to replace the secure trusted hardware with a small quantum computer that holds a quantum secret. At all times the obfuscated controller knows the secret, but it is impossible for anyone external to get two copies of the secret. When we ask the obfuscated controller to advance its simulation, it will first ask the quantum computer to transform the secret in an irreversible way and to provide proof that it has done so.

To achieve this we can basically repurpose a protocol for delegated quantum computation and verifying quantumness of a single quantum device, which is roughly what you need if you don’t trust the external world at all and so need to treat it as one device (it’s an amusing coincidence that those papers are by more old officemate Urmila Mahadev and that I worked with her on this problem in another life). To briefly summarize the trick:

• The operator knows 100 hard-to-invert functions f_i.
• For each i, the untrusted quantum computer has a superposition over two values x for which f_i(x) is equal.
• By measuring the state, the untrusted quantum computer could collapse the superposition to a single pre-image x, but it is provably hard for them to find both pre-images.
• The prover can also *prove* that they have both values by applying a hadamard transform.
• This means that the prover has a given superposition, but they don’t actually know the parts of the superposition, and so the only way they can manipulate it is by applying quantum gates.
• The obfuscated machine can then tell the untrusted quantum computer a sequence of encrypted operations to perform on this data in order to transform it into a new set of secrets, and then can ask for a set of measurements to both verify that this has been done correctly and ensure that it is irreversible. This step is the one that would require an expert in the field to spend some time.
• If that actually works then they can use the same idea to fork by stretching the 100 secrets to 400, verifying 200 to confirm that everything was done correctly, and then splitting the remaining 200 into two new secrets, one used by each of my 2 copies.

This protocol fails if an attacker can run the entire home in superposition, since then they can simply rewind the entire process. But if the attacker can’t run the secure home in superposition, then you can extract a classical copy of each input on which they run the home, and so running it on two different inputs would allow you to obtain classical copies of two challenge-responses and so violate the assumption.

This protocol probably provides better support for forking, since I can probably direct the external quantum computer to produce two new secrets without having to find a source of additional trusted hardware.

Limitations:

• You only obtain security-against-copying *after* you have set up the quantum secret. You can do this by performing a simple protocol from inside your home, and it doesn’t require trusting the quantum device at all. But if an adversary has a copy of you from before you’ve performed this protocol, they could create many copies of you and perform the ritual separately for each of them. (It actually takes some extra work to ensure that an adversary who had a copy of you from before the protocol couldn’t simply break the scheme—I think that can be done but I’m not sure.)
• There will eventually be quantum computers with quadrillions of qbits, and at that point an attacker (who has access to the final version of the quantum computer) could make many copies of me. If I was naive they could also revive any prior snapshots of me, but I could prevent that if I want to by asking the quantum computer to periodically shred and refresh its secret.
• For each new input someone wants to send to my home, they need to first consult with a quantum computer. The total cost of the quantum computation is not likely to be too large, but having quantum computers “on site” might be logistically challenging, and round trips could introduce significant latency.

Appendix A: obfuscation for uniform computations

Suppose that I want to obfuscate the program that repeatedly applies the circuit C to a state, i.e. we start from some initial state S[0], then we repeatedly compute (S[t+1], output[t]) = C(S[t], input[t]).

We’ll instead produce an obfuscated “controller” C’, and an appropriate initial state S'[0]. A legitimate operator with access to C’ can simulate my original program, whereas a malicious operator will not be able to do anything other than running multiple copies of me, rewinding to old snapshots, or killing me prematurely.

C’ contains a secret cryptographic key sk. When it receives an input (S'[t], input[t]) it does the following operations:

• First verify that S'[t] is signed with sk.
• Then decrypt S'[t] with sk in order to obtain S[t].
• Now apply C(S[t], input[t]) to obtain (S[t+1], output[t])
• Now encrypt and sign S[t+1] to obtain S'[t+1]
• Output (S'[t+1], output[t])

The analysis is left as an easy exercise for the reader (famous last words, especially hazardous in cryptography).

The same idea can be used to obfuscate other kinds of uniform computation, e.g. providing access to secure RAM or having many interacting processors.

(I think that this proposal, and many of the arguments in favor, is old. Wikipedia quotes Joseph Stieglitz making the basic point in Economics of the Public Sector.)

The policy

I’m advocating a package with 3 parts:

1. We should significantly increase capital gains taxes by taxing it at the same rate as ordinary income.
2. If an investor makes money one year then loses it later, we should give them their taxes back.
3. We should only tax excess returns above what you get by putting your money in a savings account.

Justifications

1. Why have a higher tax rate?

The simple answer is “most Americans think rich people should pay higher capital gains taxes.” It looks unfair for Warren Buffet to pay a lower marginal rate on income than a working class family. But there are other great reasons.

One is that it can be really hard to distinguish “income” from “investment income,” and many rich people misclassify income to pay fewer taxes. Some of these egregious loopholes lose a lot of tax revenue. But if we tax capital gains at the same rate as other income then these loopholes disappear and the whole system gets simpler.

Another is that rich people do a good job of identifying profitable investment opportunities; normal people can’t participate effectively in those opportunities, and so the rich get richer faster. That feels like a fact of life, but it’s actually pretty easy to change. A capital gains tax effectively lets everyone benefit from every investment opportunity.

2. Why pay investors back when they lose money?

When an investor makes money they pay taxes. But when they lose money they never get a check back. The government effectively participates in the returns but not the risk.

(Investors do get tax deductions they can use once they’ve recouped all their losses, but much of the reason people are afraid to lose money is that they might never make it back.)

This discourages investors from taking on risk and encourages them to push the risk onto other people. Combined with the proposed increase in capital gains rates, it seriously discourages uncertain long-term investments and probably reduces growth.

In addition to sounding a lot like “Heads I win, tails you lose,” I think this policy probably loses the government money.

Why? The risk of going broke pushes investors to be more conservative and causes them to have much less taxable income—which is bad for both them and the government. But because very few investors actually end up going broke, the government doesn’t save much money at all. We’ve managed to get the worst of all worlds.

(That’s coming from a pretty rough BOTEC. If the intuitive case actually depends on that calculation, I should check more carefully.)

The new alternative effectively makes the government a non-voting partner in every investment, sharing in both risk and returns. That make the system more fair, puts us all on the same team with aligned incentives, and makes everyone richer. Instead of discouraging rich people from finding opportunities, it motivates them to make bigger bets so that they can give everyone a share.

3. Why adjust the initial investment using the risk-free rate?

When I put $1,000 in a savings account, I get $2,000 in 30 years because “money now” is worth twice as much as “money later.” Everyone—rich and poor, household or government—can make the substitution at that same rate. We should only tax your savings account if we specifically want you to spend your money immediately rather than later.

It’s actually worse than that: over the last few decades all of my “earnings” from a savings account are just keeping up with inflation, so I’m paying taxes without getting any richer at all. The market expects this to keep happening for the next 30 years, so this isn’t a weird corner case.

Instead, we should tax the difference between what you earned and what anyone could have made by just putting the same amount of money in a savings account. That is, we should tax the stuff that is actually income, i.e. when you are actually doing work, taking risks, or exploiting connections.

The best argument for taxing savings is that rich people save a higher percentage of their income and we want to spread the wealth. But if that’s our goal then we can just raise income taxes on the richest people—we don’t need to take a convoluted approach. Justifications about “keeping the money moving” are based on a misunderstanding—when I “save” money I’m lending it to someone else who will use it immediately.

In this post I’ll describe a possible offsetting strategy that I think is unusually robust and should be satisfying for many moral perspectives. The same proposal would also apply to many other animal products and potentially to the environmental impacts of consumption.

Proposal

I think it’s possible to produce humane eggs where hens have positive lives and nothing horrifying happens to anyone. So my ideal would be to buy and use humane eggs. But this is tough since most of the time I’m eating eggs that someone else used as an ingredient (and even when I’m using them myself acquiring really humane eggs is kind of a pain).

So here’s an alternative that seems easier and just as good:

• Some people raise humane eggs.
• They sell these on the wholesale market as if they were totally normal eggs.
• An inspector verifies that hens are treated extremely well and that they have sold N eggs on the wholesale market.
• The inspector issues N “humane egg” certificates to the producer.
• The producer sells these certificates in an online marketplace in order to cover the extra costs of humane eggs.
• Whenever I eat an egg, I buy a humane egg certificate to go with it.

Analysis

If I buy an egg and a humane egg certificate, what is the net effect on the world?

Buying the egg increased demand for eggs. If I hadn’t also bought a certificate, that would indirectly cause someone to make one more factory-farmed egg.

Buying the positive-welfare certificate means that someone sold a wholesale egg on my behalf and increased the supply of eggs. If I hadn’t also bought an egg, that would indirectly cause someone to make one less factory-farmed egg.

So my net effect on factory farmed eggs is zero. It’s as if I was making my own positive-welfare egg and eating it, with no effect on how many factory-farmed eggs other people make or eat.

(In reality both of these actions will have other effects, e.g. causing other people to eat more or fewer eggs, but I think they still cancel out perfectly.)

This is an unusually pure form of offsetting. I’m ensuring that every hen who comes into existence because of me is living a positive life. Put differently, buying eggs only hurt hens via some indirect market effects, and I’m now offsetting my harm at that level before it turns into any actual harm to a hen. I think this form of offsetting is acceptable on a very broad range of moral perspectives (practically any perspective that is comfortable with humane eggs themselves).

Cost to the consumer

I’d guess that positive welfare eggs cost something like 3x more than typical eggs. For example I think Vital Farms sells eggs for around $6/dozen vs $2/dozen for more typical eggs.

So for each $1 that I would spend on eggs, I’d need to spend $2 to buy an egg-offset certificate. I haven’t looked into it but I could imagine wanting to go even higher to have a margin of error and shoot for even higher welfare standards. Let’s call it $0.50/egg, suggesting a 4-5x markup over typical eggs.

(I’m also not sure about relative egg sizes and didn’t look into prices very precisely, for me personally the numbers are low enough that it doesn’t matter too much even if being conservative.)

How much would that cost in practice? Here are some estimates from quick googling of recipes:

• $0.03 for a croissant (16 croissants / egg)
• <$0.30 for a scoop of egg-y ice cream (5 yolks / quart x 8 scoops / quart)
• <$0.20 for an egg-y donut (3 eggs and 2 whites / 12 donuts)
• $0.06 for a pancake (8 pancakes / egg)

The whole US produces about 100 billion eggs / year. If we wanted to offset all of that at the less-conservative $0.25/egg number, that would be $25B/year or $76 per person per year.

Benefits of decoupling (paying for welfare) from (paying for eggs)

Today animal welfare standards involve (at least) three parties: whoever raises the hen humanely, the grocery store or restaurant or whatever who uses the humane eggs, and the customer who demands humane eggs. Often there are more people still, like a host who prepares a meal for guests. And that’s all on top of the fact that people need to buy eggs at a particular place and time and so on. This means that dealing with humane eggs is always kind of a pain, and we need to get everyone to agree on a small list of standardized welfare requirements.

But humane egg certificates cut out all the middle men and just involve the producer and the consumer. So they make life way easier, and then they can also be specialized based on different welfare expectations. A certificate can say as much as it wants to about the welfare of the hens who produced it. Some customers could just go for anything that sounds “humane,” others could rely on more nuanced evaluations by non-profits who evaluate welfare claims, and the most over-the-top consumers could be matched with the most over-the-top farms.

Getting started

By the same token, it seems much less daunting to get this kind of system off the ground since it only involves two parties. A single farm could opt to sell wholesale eggs and allow inspections in order to serve a small number of customers, since they don’t have to geographically or temporally matched. I think it could be very fast to reach the point where humane egg certificates had as much flexibility as the current patchwork ecosystem for buying humane eggs.

The market could even get jump started by a tiny number of philanthropists (or even just one) who buy a few million dollars of humane egg certificates, negotiating directly with a few farmers. So this doesn’t necessarily involve any hard coordination problems at all.

Or we could literally have a kickstarter amongst people who are interested in humane egg certificates and have enough agreement about what kind of welfare standard they’d want to use (and then could pick someone to find and negotiate with a farm directly).

I think most philanthropists wouldn’t do this unless they were excited about setting norms or driven by non-utilitarian interests. In some sense this methodology feels like the “GiveDirectly of animal welfare,” extremely scalable and robust but a very expensive way to fix the problem. See the next section.

(In practice I think there are all kinds of issues with this proposal that I don’t know about, so I actually don’t know what the first steps would end up looking like, e.g. it depends on how small farmers think about humane eggs and wholesaling. This post is definitely written in spitballing mode, not as someone who understands the domain.)

Thoughts on cost-effectiveness

I’d estimate that it would cost something like $300B/year to offset all of the global harms of factory farming in this way, which feels at least 1-2 orders of magnitude worse than the kinds of welfare interventions that philanthropists feel excited about. Of course that’s also an argument against the cost-effectiveness of offsetting.

Here are some of the things going on:

• Most interventions consist in lobbying or advocacy, trying to convince other people to pay a cost (e.g. to eat less meat or to pay for better animal welfare standards). I think that’s ethically problematic as an offsetting strategy relative to paying the cost yourself, and to a lesser extent I think there’s something to be said for philanthropists directly paying these kinds of costs in addition to trying to convince others to pay them.
• The standards I’m imagining above (at $0.50/egg) are kind of “overkill,” designed to address all of the possible negative effects on a hen rather than targeting the lowest hanging fruit. Individual humane egg certificate purchasers could focus on particular standards (e.g. just ensuring that hens have dust baths and adequate cage space) and get to something closer to the cost-effectiveness of traditional hen welfare interventions, and the same norms and infrastructure could be used for both.

Universalizability

What would happen if many people tried to use this offsetting strategy?

If demand for humane egg certificates was equal to demand for eggs, then all eggs would be humane. Economically, this would be because the price of eggs has fallen below the costs of factory farming, with most of the value being in the humane certificates. I think this is basically the best possible case for an offsetting strategy, and I’d personally consider it better than abolition.

If deployed at large scale, there could be considerable customization of humane egg certificates and people who wanted it would be able to get high welfare standards with high-quality monitoring. In general I think that early adopters should view themselves as overpaying in order to share the burden of setting up the scheme and helping scale up humane agriculture.

If we all contribute to such public goods, then we can all end up better off. But as an individual it’s almost never a good return on investment. I think of supporting such public goods as being a good citizen, but that leaves open the question what is a good amount to contribute? I can make most decisions by balancing costs and benefits, but I think that basically never leads to making small contributions to public goods in the name of being a “good citizen.”

This post doesn’t aim to answer that full question, but it sets up one simple formal model for the situation and a very natural “wish list” for public goods funding norms. It then proposes the following norm that meets this wish list:

My contribution to public good X should be 100 times larger than the amount I’d personally benefit if public good X received 1% more funding.

For example, suppose I’m considering how much to contribute to Wikipedia. Maybe I get $1000/year of value from Wikipedia, and I’d get $1000.10 of value if Wikipedia had 1% more funding. This rule says I should contribute $10/year (= 10 cents x 100).

(In practice I think that labor is often a more important contribution than money, but a similar principle can apply. I’d also prefer that most donors to public goods use something like a donor lottery so that they can make a more informed decision about a larger amount of money when they do decide to give. This post is trying to bite off a tiny bit of the conceptual problem.)

Formal model

Assume there are n people and k projects. Person i makes a contribution x_ij to public good j. Let y_j = sum x_ij be the total funding for public good j.

Each person has a utility function U_i(y_1, …, y_k) expressing how much they benefit if the public goods are funded at levels y_1, …, y_k. The total utility for person i is U_i(y_1, …, y_k) – sum_j x_ij.

For simplicity I’ll assume that everyone’s utility is monotone in public goods: I’m never unhappy when one of the projects receives more funding.

I’ll also assume that the utility functions U are convex: increases in funding only ever decrease the marginal value of more funding. Convexity is often realistic and it makes it much easier to prove theorems or do computations, so it seems like a good starting point. Relaxing it seems important but not something I’m going to get into here.

I’m also assuming that everyone has linear utility in money which is only realistic if the amounts involved are very small. When utility is linear it is natural to measure utility in $ and compare it freely across people. Relaxing this assumption (along with the associated normative assumptions) is a whole can of worms that’s also important but I’m also not going to get into here.

In this setting there is a single “optimal” funding for public goods, namely the allocation that maximizes the convex aggregate utility function U_1(y_1, …, y_k) + … + U_n(y_1, …, y_k) – (y_1 +… + y_k). This leaves unspecified who provides the funding.

Wishlist

By “norm” I mean a rule that individuals can use for deciding how much to fund each public good.

Here are two plausible desiderata for a norm:

• If everyone always follows the norm, then we end up with the optimal levels of funding for the public goods.
• If you start with a community that follows the norm and add a bunch of new people who behave manipulatively, they can never make the original community worse off.

These are not the conventional desiderata in mechanism design—they say nothing at all about incentives.

All these desiderata say is that if a bunch of people choose to follow this norm, then they will do relatively well (even if many people try to manipulate their generosity). You can probably do even better by defecting. Following the norm in this post would be a kind of act of generosity. It’s also not verifiable so it can’t be enforced in any straightforward way.

Overall it’s not at all clear that you want a norm with these properties (since they can be in tension with other desirable properties like incentive compatibility), but I do think there’s a plausible case to use such norms as a kind of “cooperative default.”

Participating in this kind of cooperative norm involves being generous along an orthogonal axis to utilitarianism. Ten utilitarians with big enough irreconcilable differences would find themselves in exactly the same bind as 10 selfish people unwilling to fund public goods that would benefit the group.

The norm

I’ll first describe a centralized implementation of funding decisions since it’s easier to analyze. In the next section i’ll describe centralizing it.

Everyone publishes their utility function U_i. Choose the socially optimal level of funding y_j given those utility functions. Person j contributes y_j * dU_i/dy_j to public good j. In light of the following lemma, this is equivalent to the norm quoted at the top of the post:

Lemma [efficiency]: This norm balances the budget, i.e. y_j is in fact equal to the sum of x_ij.

Proof: at optimal funding levels, the sum of DU_i/dy_j must be equal to 1 for every j, otherwise we could improve utility by either increasing or decreasing the level of funding. So sum x_ij = sum y_j * dU_i/dy_j = y_j * sum dU_i/dy_j = y_j * 1 = y_j.

Lemma [robustness]: adding new people never harms an honest community who follows this norm.

Proof: let x_ij and y_j be the funding levels before adding new people and x_ij*, y_j* be the levels after adding people.

Adding new dishonest people doesn’t change the utility functions of any of the honest people, it just adds new (monotone) terms U_i to the sum that we are maximizing. This results in increasing the marginal value of every public good, which results in increasing the level of funding y_j* for each public good, which results (by convexity) in decreasing every derivative dU_i/dy_j. I’ll write dU_i/dy_j* for the derivative of U evaluated at y* and dU_i/dy_j for the derivative evaluated at y.

We’ll show that the increase in utility U_i(y*) – U_i(y) is always greater than the increase in person i’s payment:

• By convexity, U_i(y*) – U_i(y) is at least sum_j (y_j* – y_j) dU_i/dy_j*
• Since derivatives are decreasing and y_j >= 0, this is at least sum_j ( y_j* dU_i / dy_j*) – (y_j dU_i / dy_j)
• By definition, this is equal to sum_j x_ij* – x_ij, i.e. the total amount that we increased person i’s payment.

Relaxing assumptions

Centralization. To implement this norm need to find the unique allocation that satisfies x_ij = y_j * dU_i/dy_j. We could do this in an iterative process where everyone iteratively adjusts their contributions x_ij until we get to equilibrium, basically implementing a distributed convex optimization algorithm—we don’t actually need to have anyone ever write down their utility function or have any communication beyond calculating the combined funding for each public good y_j = sum_i x_ij.

Linear utility. In reality my value for money increases as I spend more of it. I think the basic approach in this post still works, and the iterative algorithm is basically unchanged as long as you just convert between $ and utility using your current preferences at each step. The guarantee becomes more complex because it’s no longer so simple to talk about a social optimum. The best we can say is that we’ll get something Pareto efficient (even including the possibility of transfers).

Convexity. If utility is non-convex then an iterative approach won’t work and you certainly need to have some kind of coordination to get to an efficient allocation (you also need to solve a potentially intractable optimization problem). I believe the norm in this post may also become vulnerable to manipulation but I haven’t checked for an example. It’s plausible to me that my two desiderata are unachievable when utility is non-convex. (They are obviously unachievable when utility is non-monotone.)

Uniqueness? It seems like everything had to work out surprisingly nicely for this norm to achieve the two desiderata. I’m not sure whether there are many other norms that would work or if this one is essentially unique.

On the flip side, creating really bad outcomes seems much easier, running into none of the symmetric “problems.” So what gives?

I’ll argue that nature is basically out to get us, and it’s not a coincidence that making things good is so much harder than making them bad.

First: some other explanations

Two common answers (e.g. see here and comments):

• The worst things that can quickly happen to an animal in nature are much worse than the best things that can quickly happen.
• It’s easy to kill or maim an animal, but hard to make things go well, so “random” experiences are more likely to be bad than good.

I think both of these are real, but that the consideration in this post is at least as important.

Main argument: reward errors are asymmetric

Suppose that I’m building an RL agent who I want to achieve some goal in the world. I can imagine different kinds of errors:

• Pessimism: the rewards are too low. Maybe the agent gets a really low reward even though nothing bad happened.
• Optimism: the rewards are too high. Maybe the agent gets a really high reward even though nothing good happened, or gets no reward even though something bad happened.

Pessimistic errors are no big deal. The agent will randomly avoid behaviors that get penalized, but as long as those behaviors are reasonably rare (and aren’t the only way to get a good outcome) then that’s not too costly.

But optimistic errors are catastrophic. The agent will systematically seek out the behaviors that receive the high reward, and will use loopholes to avoid penalties when something actually bad happens. So even if these errors are extremely rare initially, they can totally mess up my agent.

When we try to create suffering by going off distribution, evolution doesn’t really care. It didn’t build the machinery to be robust.

But when we try to create incredibly good stable outcomes, we are fighting an adversarial game against evolution. Every animal forever has been playing that game using all the tricks it could learn, and evolution has patched every hole that they found.

In order to win this game, evolution can implement general strategies like boredom, or an aversion to meaningless pleasures. Each of these measures makes it harder for us to inadvertently find a loophole that gets us high reward.

Implications

Overall I think this is a relatively optimistic view: some of our asymmetrical intuitions about pleasure and pain may be miscalibrated for a world where we are able to outsmart evolution. I think evolution’s tricks just mean that creating good worlds is difficult rather than impossible, and that we will be able to create an incredibly good world as we become wiser.

It’s possible that evolution solved the overoptimism problem in a way that is actually universal—such that it is in fact impossible to create outcomes as good as the worst outcomes are bad. But I think that’s unlikely. Evolution’s solution only needed to be good enough to stop our ancestors from finding loopholes, and we are a much more challenging adversary.