This post gives my personal take on “AI for epistemics” and how important it might be to work on.

Some background context:

• AI capabilities are advancing rapidly and I think it’s important to think ahead and prepare for the possible development of AI that could automate almost all economically relevant tasks that humans can do.1

• That kind of AI would have a huge impact on key epistemic processes in our society. (I.e.: It would have a huge impact on how new facts get found, how new research gets done, how new forecasts get made, and how all kinds of information spread through society.)

• I think it’s very important for our society to have excellent epistemic processes. (I.e.: For important decisions in our society to be made by people or AI systems who have informed and unbiased beliefs that take into account as much of the available evidence as is practical.)

• Accordingly, I’m interested in affecting the development and usage of AI technology in ways that lead towards better epistemic processes.

So: How can we affect AI to contribute to better epistemic processes? When looking at concrete projects, here, I find it helpful to distinguish between two different categories of work:

1. Working to increase AIs’ epistemic capabilities, and in particular, differentially advancing them compared to other AI capabilities. Here, I also include technical work to measure AIs’ epistemic capabilities.2

2. Efforts to enable the diffusion and appropriate trust of AI-discovered information. This is focused on social dynamics that could cause AI-produced information to be insufficiently or excessively trusted. It’s also focused on AIs’ role in communicating information (as opposed to just producing it). Examples of interventions, here, include “create an independent organization that evaluates popular AIs’ truthfulness”, or “work for countries to adopt good (and avoid bad) legislation of AI communication”.

I’d be very excited about thoughtful and competent efforts in this second category. However, I talk significantly more about efforts in the first category, in this post. This is just an artifact of how this post came to be, historically — it’s not because I think work on the second category of projects is less important.3

For the first category of projects: Technical projects to differentially advance epistemic capabilities seem somewhat more “shovel-ready”. Here, I’m especially excited about projects that differentially boost AI epistemic capabilities in a manner that’s some combination of durable and/or especially good at demonstrating those capabilities to key actors.

Durable means that projects should (i) take the bitter lesson into account by working on problems that won’t be solved-by-default when more compute is available, and (ii) work on problems that industry isn’t already incentivized to put huge efforts into (such as “making AIs into generally better agents”). (More on these criteria here.)

Two example projects that I think fulfill these criteria (I discuss a lot more projects here):

• Experiments on what sort of arguments and decompositions make it easier for humans to reach the truth in hard-to-verify areas. (Strongly related to scalable oversight.)

• Using AI to generate large quantities of forecasting data, such as by automatically generating and resolving questions.

Separately, I think there’s value in demonstrating the potential of AI epistemic advice to key actors — especially frontier AI companies and governments. When transformative AI (TAI)4 is first developed, it seems likely that these actors will (i) have a big advantage in their ability to accelerate AI-for-epistemics via their access to frontier models and algorithms, and (ii) that I especially care about their decisions being well-informed. Thus, I’d like these actors to be impressed by the potential of AI-for-epistemics as soon as possible, so that they start investing and preparing appropriately.

If you, above, wondered why I group “measuring epistemic capabilities” into the same category of project as “differentially advancing AI capabilities”, this is now easier to explain. I think good benchmarks could be both a relatively durable intervention for increasing capabilities, via inspiring work to beat the benchmark for a long time, and that they’re a good way of demonstrating capabilities.

Structure of the post

In the rest of this post, I:

• Link to some previous work in this area.

• Describe my basic impression of why work on AI-for-epistemics could be important.

• Go into more details on Heuristics for good interventions, including:

  • A distinction between direct vs. indirect strategies and the value of indirect interventions.

  • Some abstract guidelines for how to avoid interventions that get swamped by the bitter lesson or by commercial interests.

  • An attempt to paint a concrete picture for what excellent use of AI for epistemics might eventually look like (to give a sense of what we want to steer towards).

• Discuss what concrete types of interventions seem best in the domain of differentially advancing epistemic capabilities.

Previous work

Here is an incomplete list of previous work on this topic:

• I previously wrote Project ideas: Epistemics on this blog.

• Carl Shulman on government and society after AGI at the 80,000 hours podcast. (This current blog post owes many ideas to Carl Shulman.)

• Ben Todd: The most interesting startup idea I've seen recently: AI for epistemics 

• On AI and forecasting:

  • Ozzie Gooen: My Current Claims and Cruxes on LLM Forecasting & Epistemics

    • Ozzie runs the Quantified Uncertainty Research Institute which is more generally relevant.

  • FutureSearch is using AI for forecasting and other tricky questions.

    • See also The Rationale-Shaped Hole At The Heart Of Forecasting where they lay out some of their views as of April 2024.

  • Approaching Human-Level Forecasting with Language Models by Halawi, Zhang, Yueh-Han, and Steinhardt (2024).

  • LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy by Schoenegger, Tuminauskeite, Park, and Tetlock (2024).

    • See also Forecasting Research Institute where Tetlock is president and chief scientist.

  • Metaculus is running a bot-only forecasting series. (Launch announcement.)

• On AI & persuasion:

  • Risks from AI persuasion by Beth Barnes.

  • Persuasion Tools by Daniel Kokotajlo

• Elicit is an AI research assistant, developed by a company that was spun out from Ought, with the intention of improving human judgment as AI capabilities improve. (See here for some of the original case.)

Why work on AI for epistemics?

Summary

I think there’s very solid grounds to believe that AI’s influence on epistemics is important. Having good epistemics is super valuable, and human-level AI would clearly have a huge impact on our epistemic landscape. (See just below for more on importance.)

I also think there are decent plausibility arguments for why epistemics may be important: Today, we are substantially less epistemically capable than our technology allows for, due to various political and social dynamics which don’t all seem inevitable. And I think there are plausible ways in which poor epistemics can be self-reinforcing (because it makes it harder to clearly see what’s the direction towards better epistemics). And vice-versa that good epistemics can be self-reinforcing. (See here for more on path-dependence.)

That’s not very concrete though. To be more specific, I will go through some more specific goals that I think are both important and plausible path-dependent:

• Good norms & practices for AI-as-knowledge-producers. Such as transparency of how AI-based science/investigations work, minimal censorship of AI-produced research results, and maximizing the number of actors who can trust some technically sophisticated institution to verify AI methods’ trustworthiness. (E.g. by having many such institutions with different political affiliations.)

• Good norms & practices for AI-as-communicatiors. Such as transparency of how AIs decide what to communicate, independent evaluators who measure AIs’ truthfulness, and laws that limit the degree to which AIs can present contradictory arguments to different people or be paid-off to present biased views.

• Differentially high epistemic capabilities. Such as high alignable capabilities (compared to underlying capabilities), relative strength at persuading people of true beliefs compared to false beliefs, and relative strength at understanding & predicting the world compared to building new technologies.

Let’s go through all of this in more detail.

Importance

I think there’s very solid grounds to believe that AI’s influence on epistemics is important.

• AI’s influence on human epistemic abilities will eventually be huge. Briefly:

  • AI will eventually automate epistemic labor. This includes both knowledge production work and communication work. (The latter which includes both good and bad persuasion of humans.)

  • AIs’ epistemic work won’t just replace humans’ 1-for-1. AI comes with special capabilities that will change the epistemic ecosystem:

    • Cheaper & easier to delegate epistemic labor.

      • It could be cheaper to delegate epistemic labor to AIs (than to humans) because you can copy software for free.

      • If we develop methods to train reliably truth-seeking AIs, it will be easier to delegate epistemic labor to AIs (than to humans), because you would have to worry less about being deceived.

      • More delegation could lead to more equitable distribution of epistemic capabilities, but also to reduced incentives and selection for humans to have reasonable beliefs and epistemic practices (because AIs make all decisions that matter for your power).

    • Better epistemic science.

      • You can more easily control what information AIs have and have not seen and thereby run reproducible experiments on what epistemic strategies work best.

• Epistemic capabilities during and after the development of TAI are very valuable. Briefly:

  • Most AI takeover risk comes from “unforced errors”. A vast majority of powerful people don’t want AI to take over, but I think that many underestimate the risk. If I thought that people were going to have reasonable, well-calibrated beliefs about AI takeover risk, my subjective probability of AI takeover would more than halve.5

  • Most extinction risk comes from “unforced errors”. Just as above: A vast majority of powerful people don’t want extinction, and (I strongly suspect) would be capable of preventing the exceptions from being able to cause extinction.6

  • Strong epistemic capabilities seem great for moral deliberation. For example via: Helping you better imagine the realistic consequences of various moral principles; by letting you forecast what sort of deliberation procedures will go off-the-rails; and by teaching you about the underlying empirical reasons for moral disagreement (so you can choose which drivers of moral intuition you trust more).

Path-dependence 

While less solid than the arguments for importance, I think there are decent plausibility arguments for why AI’s role in societal epistemics may be importantly path-dependent.

• Comparing with the present. Today, I think that our epistemics are significantly worse than they “could have been”. We aren’t just constrained by high-quality labor or evidence — there are also significant political and self-serving forces/incentives that actively distort people’s beliefs. These won’t automatically go away in the future.

• Feedback loops. People often choose to learn the truth when the choice is presented sufficiently clearly and unambiguously to them.7 But with poor enough epistemic starting abilities, it won’t be clear what methods are more or less truth-seeking. So poor epistemic capabilities can be self-reinforcing, and vice versa.

• Veil of ignorance. Conversely, people may be more enthusiastic to invest in novel, strong epistemic methods while they think that those methods will come to support their current beliefs (which would be the default, if they actually believe their current beliefs8). Whereas if they first learn that the methods are going to contradict their current beliefs, then they may oppose them.

• Early investment. I can easily imagine both a future where frontier AI projects either (i) spend continuous effort on making their AIs strong forecasters and strategic analysts, and distributes those capabilities to other key institutions, or (ii) almost exclusively focus on using their AI systems for other tasks, such as technical R&D.9 My being able to imagine both might just be a fact of my own ignorance — but it’s at least suggestive that both futures are plausible, and could come about depending on our actions.

• Distribution of epistemic capabilities. Even without changing the pace at which powerful AI epistemics are developed, the question of whether important decisions are made with or without AI epistemic assistance may depend on how quickly different actors get access to those capabilities. It seems probably great for those epistemic capabilities to quickly be made widely available,10 and if they’re powerful enough, it could be essential for multiple key players (such as AI companies, governments, and opposition parties) to get access to them at a similar time, so they can provide checks on each others’ new capabilities.

To be more concrete

Now, let’s be more specific about what goals could be important to achieve in this area. I think these are the 3 most important instrumental goals to be working towards:

• Society adopts good norms & practices for AI-as-knowledge-producers, i.e., norms & practices that allow insights from AI-as-knowledge-producers to be widely spread and appropriately trusted.

• Society adopts good norms & practices for AI-as-communicators, i.e., norms & practices that make it systematically easy for AIs to spread true information and relatively more difficult for AIs to illegitimately persuade people of falsehoods.

• For a given amount of general capabilities, we have high “epistemic capabilities” and high justified trust in those capabilities.

Let’s go through these in order.

Good norms & practices for AI-as-knowledge-producers

Let’s talk about norms and practices for AIs as knowledge-producers. With this, I mean AIs doing original research, rather than just reporting claims discovered elsewhere. (I.e., AIs doing the sort of work that you wouldn’t get to publish on Wikipedia.)

Here are some norms/institutions/practices that I think would contribute to good usage of AI-as-knowledge-producers:

• Minimal (formal or informal) censorship of AI-produced research results.

• Transparency of how results from AI-as-knowledge-producers were arrived at.

• A government agency that is non-partisan (in practice and not only in name) and charged with using AI to inform government decision-making or to transparently review whether other knowledge-producing AIs in government are doing so in a truth-seeking manner.11

• Maximizing the number of actors who can trust some technically sophisticated institution to verify claims about AI methods’ trustworthiness.

  • For example, this could be achieved via having many actors with different political affiliations verify claims about a centralized project, or by having many actors with different political affiliations train their own truth-seeking AIs (noticing how they tend to converge).

• Great evals of AIs epistemic capabilities.

  • For this, it’s helpful if you have a longer track record of AIs being used for important real-world questions and getting them right or wrong.

Good norms & practices for AI-as-communicators

Now let’s talk about norms for AIs as communicators. This is the other side of the coin from “AI as knowledge producers”. I’m centrally thinking about AIs talking with people and answering their questions.

Here are some norms/institutions/practices that I think would enable good usage of AI-as-communicators:

• Transparency about how AIs decide what to communicate.

  • E.g. via publishing information about AIs’ constitutions or model spec.

• Independent evaluators publishing reports on AI truthfulness, including…

  • Fraction of statements that the evaluators believe to be clearly true, debatable, vs. clearly false.

  • Results from AI lie-detection tests on whether the AI is being dishonest.

  • The degree to which AIs contradict themselves in different contexts or when talking with different audiences.

  • The degree to which AI is misleading via behaving differently (e.g. being more or less evasive, using a different tone, or relying on very different sources) on questions that are similar except for their implications about a topic that the AI may want to mislead users about (e.g. something political or something where the AI developer has commercial interests).

  • Experiments on whether humans tend to be better or worse at answering questions about a certain topic after conversing the AI about related topics. If humans are systematically worse, that suggests that the AI may be systematically misleading.

• Certain laws about AI communication may be helpful, such as:

  • You’re not allowed to pay other actors to program their AIs to be more positively inclined towards you.12

  • AI cannot systematically say directly contradictory statements to different audiences or in different contexts.

• Conversely, it’s also important to avoid bad laws. For example, laws that forbid AIs from saying blatant falsehoods may be good if they were judged in a reasonable way, and had the threshold for “blatant” set highly enough, but they could also be very bad if they became a tool for pushing political agendas.

Differentially high epistemic capabilities

Finally: I want AIs to have high epistemic capabilities compared to their other capabilities. (Especially dangerous ones.) Here are three metrics of “epistemic capabilities” that I care about (and what “other capabilities” to contrast them with):

• Asymmetric persuasion: How capable is AI at persuading people of true things vs. how capable is AI at persuading people of anything?13

  • It’s good for the former to be high relative to the latter, because I think it’s typically better for people to be convinced of true things than false things.

  • (The web of lies eval in Evaluating Frontier Models for Dangerous Capabilities tests for one version of this, where current models seem significantly better at persuading people of true things.14)

• Understanding (vs. building): How useful is AI for understanding & predicting the world vs. building new technologies?

  • Central examples that I want to capture in “understanding”: Forecasting, policy development, geopolitical strategy, philosophy.

  • Central examples that I want to capture in “building new technologies”: Coding, AI R&D, bio R&D, building robots.

  • I suspect (but am not confident) that it’s good for the former to be high relative to the latter, because I am scared of new technologies causing accidents (mainly AI takeover15) or being misused by the wrong people (mainly bioweapons), and I think that better understanding could help reduce this risk.

  • What makes this a natural dichotomy? Or a more decision-relevant question: Why should we think it’s possible to differentially accelerate “understanding” separately from “building”? Here are some of the core differences that I see between the two:

    • 1. “Building technology” typically has better empirical feedback loops.

    • 2. When “building technology”, it’s typically easier and more helpful to make accurate & precise mathematical models.

    • 3. “Technology” is typically more specialized/modular, whereas "understanding" relies more on the ability to incorporate lots of messy interdisciplinary data.

    • 4. “Technology” is typically less political, whereas for “understanding” it’s often more important to manage and correct for political biases.

  • There are exceptions to all four of these. But they hold often enough that I think they induce some important difference in what epistemic methods are most useful for “understanding” vs. “building”. Which may lead to some opportunities to differentially advance one over the other.16

• Aligned capabilities: What knowledge & understanding can AI developers leverage towards the AI developers’ goals vs. what knowledge & understanding can AIs leverage towards their own goals?

  • It’s good for the former to be high, because if the latter is higher, then AI takeover would be more likely. Specifically, AI may then be able to (i) leverage powers that we don’t have access to, and (ii) take us by surprise, if we didn’t know about their capabilities.

  • Even if AI takeover isn’t a problem, this might still introduce a discrepancy where people can use AI’s full capabilities to pursue easy-to-measure goals but can’t use them to pursue hard-to-measure goals. (Since it’s difficult to provide a feedback signal which encourages an AI to pursue those goals.) I think this is also undesirable, and related to the previous categories:

    • It’s easier to measure whether you’ve persuaded someone than whether you’ve persuaded them of something true.

    • It’s easier to measure whether you’ve developed a powerful technology than whether you’ve produced a correct forecast for whether releasing that technology is likely to cause an irreversible catastrophe.

  • (C.f.: Strategy Stealing, Decoupling Deliberation & Competition.)

In order for these distinctions to be decision-relevant, there needs to be ways of differentially accelerating one side of the comparison compared to the other. Here are two broad categories of interventions that I think have a good shot at doing so:

• Scalable Oversight, Weak-to-Strong Generalization (W2S generalization), and Elicing Latent Knowledge (ELK).

  • I list these three domains together because they’re strongly related.

    • See Scalable Oversight and Weak-to-Strong Generalization: Compatible approaches to the same problem.

    • Also, from the W2S generalization paper: “Our setting can be viewed as a general methodology for empirically studying problems like ELK and honesty across a wide range of tasks.”

  • These research areas push forward “aligned capabilities” via letting us elicit stronger capabilities towards arbitrary goals.

  • This pushes towards “asymmetric persuasion” via letting people increase their skepticism of unsupported AI statements, while still being able to believe AI statements backed up by a spot-checked decomposed argument (scalable oversight), by a scientific understanding of generalization (W2S generalization), or by methods that directly eliciting AI’s latent knowledge.

  • This pushes forward “understanding” over “building technology” via being disproportionately helpful for boosting capabilities in areas with poor feedback loops. (Whereas I think “building technology” typically has better feedback loops.)

• Building and iteratively improving capabilities on “understanding”-loaded tasks, such as forecasting and strategic analysis.

  • (This partly overlaps with the first point, because you might want to practice using scalable-oversight/W2S-generalization/ELK on these tasks in particular.)

  • Examples of how you might do this includes:

    • Making the models do forecasts of unseen data, and iterating to improve their performance.

      • This becomes more interesting if you can train highly capable models on only old data, since this would let you test and iterate the models on more long-range forecasting.

    • Training models using experts’ (superforecasters, policy analysts, AI strategy researchers) feedback.

      • Either in a baseline RLHF sort-of way, or going further towards scalable oversight & W2S generalization.

    • Experimentally determine what sort of procedures and arguments tend to lead humans towards truth.

      • For example, via the methodology that Tom Davidson outlines in this post. Note that this might be meaningfully different than the procedures that work well in technical domains, because of the fuzzier topics and increased political biases.

  • I think this differentially pushes forward “aligned capabilities” in “understanding”-loaded domains, because I expect that models will (via generalization from pre-training) start out with some baseline understanding of these domains. Effort on these tasks will go towards some mix of increasing model capabilities and improving our ability to elicit existing capabilities, and I expect the net-effect will be to somewhat reduce the amount of capabilities that we can’t elicit. (But I don’t feel fully confident in this.)

  • This can push towards “asymmetric persuasion” in these domains insofar as developers take care to develop truth-seeking methods rather than just indiscriminately iterating to improve models’ ability to persuade people.

  • This clearly differentially pushes towards “understanding”.

Heuristics for good interventions

Having spelled-out what we want in the way of epistemic capabilities, practices for AI-as-knowledge-producers, and AI-as-communicators: Let’s talk about how we can achieve these goals. This section will talk about broad guidelines and heuristics, while the next section will talk about concrete interventions. I discuss:

• A distinction between direct vs. indirect strategies and the value of indirect interventions.

• Some abstract guidelines for how to avoid interventions that get swamped by the bitter lesson or by commercial interests.

• An attempt to paint a concrete picture for what excellent use of AI for epistemics might eventually look like (to give a sense of what we want to steer towards).

Direct vs. indirect strategies

One useful distinction is between direct and indirect strategies. While direct strategies aim to directly push for the above goals, indirect strategies instead focus on producing demos, evals, and/or arguments indicating that epistemically powerful AI will soon be possible, in order to motivate further investment & preparation pushing toward the above goals.

My current take is that:

• Direct, competent efforts on Good practices for AI-as-knowledge-producers17 and Good practices for AI-as-communicators18 seem great.

• Direct efforts on differentially high epistemic capabilities via scalable oversight, W2S generalization, & ELK seem great.

• Direct efforts on differentially high epistemic capabilities via building and iteratively improving capabilities on “understanding”-loaded tasks (as discussed in the previous section) seem useful if done in the right way. But there’s some worry that progress here will be swamped by the bitter lesson and/or commercial investments. I talk about this more later.

• Indirect efforts via building and iteratively improving abilities on “understanding”-loaded tasks seem potentially useful. But I’m not sure how compelling the path-to-impact really is. Let’s talk a bit more about it.

Indirect value generation

One possible path-to-impact from building and iteratively improving capabilities on “understanding”-loaded tasks is that this gives everyone an earlier glimpse of a future where AIs are very epistemically capable. This could then motivate:

• Investment into making AIs even more epistemically capable.

• Increased concern about AI companies’ upcoming epistemic advantage over other actors, motivating stronger demands for transparency and increased urgency of actors acquiring their own ability to verify AI epistemic methods.

• The production of further evals for exactly how epistemically capable AIs are.

• More attention, as well as more-informed attention, going towards developing norms for how AIs should communicate information.

• Also: It would directly enable AIs to start gathering a good epistemic track record earlier, which could be helpful for evaluating AIs’ trustworthiness later.

• Also: It might help build a track record for a particular organization, which could be helpful for evaluating that organization’s track-record later.

The core advantage of the indirect approach is that it seems way easier to pursue than the direct approach.

• Demos and evals for epistemically capable AI are very easily measurable/testable which gives you great feedback loops.

• Because the path-to-impact is indirect, it’s ok if methods or evals don't generalize to future, more powerful AI systems. They can still “wake people up”.

• Because the path-to-impact is indirect, it’s ok if the work takes place in organizations that will eventually be marginalized/outcompeted by the well-resourced AI company or other actors. They can still “wake people up”.

Core questions about the indirect approach: Are there really any domain-specific demos/evals that would be convincing to people here, on the margin? Or will people’s impressions be dominated by “gut impression of how smart the model is” or “benchmark performance on other tasks” or “impression of how fast the model is affecting the world-at-large”? I feel unsure about this, because I don’t have a great sense of what drives people’s expectations, here.

A more specific concern: Judgmental forecasting hasn’t “taken off” among humans. Maybe that indicates that people won’t be interested in AI forecasting? This one I feel more skeptical of. My best-guess is that AI forecasting will have an easier time of becoming widely adopted. Here’s my argument.

I don’t know a lot of why forecasting hasn’t been more widely adopted. But my guess would be that the story if something like:

• Many forecasting practices are useful. (Such as assigning probabilities to questions that are highly relevant to your decisions, keeping track of how well you’re doing, and keeping track of how well people-who-you-listen-to are doing.)

• However, they’re not useful enough that people who use these tools but who don’t have experience in a profession can easily outcompete people without experience in that profession.

• And it takes time for people in existing professions to adopt good practices. (Cultural change is slow.)

For AIs, these problems seem smaller:

• There’s already a culture of measuring AI performance numerically, so you don’t need much of a cultural shift to get AIs to quantify their probability estimates and be scored on them.

• And AI advantages will eventually deliver lots of advantages over existing human experts, so there will eventually be strong incentives to shift over to using AI.

Overall, I feel somewhat into “indirect” approaches as a path-to-impact, but only somewhat. But it at least seems worth pursuing the most leveraged efforts here: Such as making sure that we always have great forecasting benchmarks and getting AI forecasting services to work with important actors as soon as (or even before) they start working well.

On long-lasting differential capability improvements

It seems straightforward and scalable to boost epistemic capabilities in the short run. But I expect a lot of work that leads to short-run improvements won’t matter after a couple of year. (This completely ruins your path-to-impact if you’re trying to directly improve long-term capabilities — but even if you’re pursuing an indirect strategy, it’s worse for improvements to last for months than for them to last for years.)

So ideally, we want to avoid pouring effort into projects that aren’t relevant in the long-run. I think there’s two primary reasons for why projects may become irrelevant in the long run: either due to the bitter lesson or due to other people doing them better with more resources.

• The bitter lesson: This suggests that we want to avoid “leveraging human knowledge” and instead work on projects that “leverage computation”.

  • This inherently gets easier over time — as investments go up and more compute becomes available. But you can still test out prototypes now.

  • Example projects that are especially good for “leveraging computation” are those that explore how we could generate large amounts of data to train on, given access to large amounts of compute.

• Other people doing them better: Epistemic capabilities will be significantly boosted by efforts that other people have huge incentives to pursue, such as learning how to train models on synthetic data, or making AI agents who can pursue tasks over longer periods of time, and who can easily navigate interfaces that were made for humans. Such projects therefore don’t seem worth prioritizing.

  • (Though it might be valuable to make sure that epistemic projects are in close touch with people working on more general capabilities, in order to plan around their advances and make use of their unreleased prototypes. This is one reason why “AI for epistemics” projects may benefit from being inside of frontier AI companies.)

That said, even if we mess this one up, there’s still some value in projects that temporarily boost epistemic capabilities, even if the technological discoveries don’t last long: The people who work on the project may have developed skills that let them improve future models faster, and we may get some of the indirect sources of value mentioned above.

Ultimately, key guidelines that I think are useful for this work are:

• Inform people who can influence the future of AI-for-epistemics. (As discussed in the previous section on indirect approaches.)

• Leverage computation rather than human knowledge.

• Avoid building stuff that others will build better.

Painting a picture of the future

To better understand which ones of today’s innovations will be more/less helpful for boosting future epistemics, it’s helpful to try to envision what the systems of the future will look like. In particular: It’s useful to think about the systems that we especially care about being well-designed. For me, these are the systems that can first provide a very significant boost on top of what humans can do alone, and that get used during the most high-stakes period around TAI-development.

Let’s talk about forecasting in particular. Here’s what I imagine such future forecasting systems will look like:

• In contrast to today’s systems, I don’t think they’ll have a series of hard-coded steps. I think they’ll be much more flexible in going back and forth between different types of considerations. But even if they don’t have e.g. a hard-coded “baserate generation”-step, they’ll probably still use “baserate generation” in a similar sense as human forecasters do: As a useful, flexible input into their forecasting.

• For short-horizon forecasting (e.g.: what will tomorrow’s newspaper report about ongoing events?) I imagine that they will rely a lot on hard-to-express heuristics that they’ve learned from fine-tuning. Because there’s enough data that this is feasible.

  • Most of this data will be model-generated. AIs will formulate questions about tomorrow’s newspaper, that other AIs will forecast, and that other AIs will resolve after reading the next day’s paper.

• For medium-horizon forecasting (e.g. what will happen in 1-12 months) I think there won’t be enough data for finetuning to build in a lot of really opaque heuristics. But we will be able to learn what types of reasoning tends to lead to relatively better vs. worse forecasts, by instructing different systems to use different strategies (and verifying that they do so, c.f. measuring faithfulness in chain-of-thought reasoning). Then we can teach models to use the strategies and heuristics that work best. (E.g. via few-shot prompting them with good examples, or by doing supervised learning on passages of chain-of-thought that empirically did well (similar to the Halawi paper), or by using constitutional-AI-style-training to generate many more examples of similar passages to finetune on.)

  • Again, most of the data will be model-generated.

  • Medium-horizon forecasters can reference all kinds of evidence, including the forecasts of short-horizon forecasters (we should have good statistics on how reliable these are).19

• For long-horizon forecasting (e.g. what will happen in >1 year) we won’t have any ground-truth data to train on, so we’ll have to rely on human feedback.20 In order to know what kind of feedback to give, here, we’ll want to use medium-horizon forecasting as a “lab” to test different hypotheses about what sort of AI-human interactions tend to lead to accurate forecasts, and what types of arguments tend to work well in practice.

  • In order to generalize sensibly from the medium-horizon-forecasting case, we’ll want the learnings from this to be as human-interpretable as possible. E.g. “arguments from analogy tends to work well/poorly”, not “this 100k word long prompt tends to give good results, and no human can understand why”.

  • Long-horizon forecasters can reference all kinds of evidence, including the forecasts of short- and medium-horizon forecasters, insofar as they’re relevant.

  • When using medium-horizon forecasting as a “lab”: We’ll want to run both (i) studies where we try to get as good forecasting abilities as we can, including by relying substantially on good generalization from AIs, and (ii) studies where a red-team tries to make the AIs maximally subtly misleading, and see whether humans who are getting AI advice can notice this, or whether they get tricked into believing terrible forecasts.

    • If the latter tests lead to humans making terrible forecasts, then we should assume that scheming AIs would be able to mislead us about both medium-term and long-term forecasts. (And probably also short-term forecasts in recognizably rare, high-stakes situations.)

    • C.f. control evaluations.

• Medium- and long-horizon forecasters may be finetuned copies of models that were previously fine-tuned on short-horizon forecasting, insofar as that tends to instill some useful intuitions.

• Ideally, someone will have trained a competent model that hasn’t seen any data from the last 3 years, or maybe 30 years, or possibly 300 years. (The longer time frames would require a lot of synthetic data.) We could use such a model as a “lab” to test hypotheses about what types of reasoning tends to do well or poorly on long-range forecasting.

Concrete projects for differentially advancing epistemic capabilities

Now, let’s talk about concrete projects for differentially advancing epistemic capabilities, and how well they do according to the above criteria and vision.

Here’s a summary/table-of-contents of projects that I feel excited about (no particular order). More discussion below.

• Evals/benchmarks for forecasting (or other ambitious epistemic assistance) (Evals seems leveraged for demonstrating capabilities, and seems also likely to lead to somewhat longer-lived capability benefits.)

• Project to automate question-generation and question-resolution.

• Figuring out logistics of past-casting. E.g.: How do we date past data? Does training models on data in chronological order cause any issues?

• Start efforts inside of AI companies for AI forecasting or other ambitious epistemic assistance.

• Scalable oversight / weak-to-strong-generalization / eliciting latent knowledge.

• Experiments on what type of arguments and AI-interactions tend to lead humans toward truth vs. mislead them.

Effort to provide AI forecasting assistance (or other ambitious epistemic assistance) to governments is another category of work that I’d really like to happen eventually. But I’m worried that there will be more friction in working with governments, so that it’s better to iterate outside them first and then try to provide services to them once they’re better. This is only a weakly held guess, though. If someone who was more familiar with governments thought they had a good chance of usefully working with them, I would be excited for them to try it.

In the above paragraph, and the above project titles, I refer to AI forecasting or “other ambitious epistemic assistance”. What do I mean by this?

• “Ambitious epistemic assistance” is meant to include projects that don’t do forecasting specifically, but that still leverage AIs to do a large amount of epistemic labor, in a way that could be scaled up to be extremely valuable.

• For example, I would want to include AIs that assist with threat modeling or policy analysis in flexible, scalable ways.

• On the other hand, an example of something that would be insufficiently ambitious is a tool that was narrowly targeted at a particular well-scoped type of analysis. Which could be perfectly automated without providing that much of an acceleration to overall strategic efforts. E.g. efforts to automate highly structured literature reviews (such as “automatically finding and combining randomized trials of medical interventions”).

Now for more detail on the projects I’m most excited about.

Evals/benchmarks for forecasting (or other ambitious epistemic assistance)

• I generally think that evaluations and benchmarks are pretty leveraged for motivating work and for making it clear when models are getting seriously good.

• Some ongoing work on this includes.

  • Open Philanthropy funded FutureSearch to develop forecasting evals.

  • Metaculus is running a bot-only forecasting series. (Launch announcement.)

• Further work on this would ideally check in with these efforts and see if there’s important angles that they don’t cover, that would be good to evaluate.

Automate forecasting question-generation and -resolution

• I.e.: Train AI systems to automatically formulate forecasting questions, and train AI systems to automatically seek out information about what happened and resolve them.

• This is strongly related to the above “evals/benchmark” category. It's a tool that can be used to generate really large amounts of questions that models can be evaluated on. (Or trained on.)

• Other than general reasons why I like evals/benchmarks, I like this angle because:

  • It’s also squarely in the domain of “leveraging computation rather than human knowledge”, as a way of improving forecasting abilities. (Via training models on the automatically generated questions.)

  • I think that “automatically generated and resolved forecasting questions” is a core part of what it would eventually look like to have a flourishing science of AI forecasting. And it seems great to have prototypes of all the most core parts, as early as possible, so that:

    • We can track how close we are to the full vision.

    • It gets easier to demo & explain what the future might look like.

    • We can encounter obstacles and start working on them early

• My main concern is that the models might not quite be good enough to make this really easy, yet. So it might be significantly easier to do with generations that are one step further along.

Logistics of past-casting

• Another angle for getting more forecasting data is to exploit all the data in the past. If we could train competent AI systems without “spoiling” them on recent events, then we could run experiments on what methodologies work best for long-range forecasting.

• One question to answer here is: What methods do we have for determining the date of past data, and how reliable are they?

  • Can we date past internet-data by checking for its presence in older crawls? (E.g. past versions of the common crawl.)

  • By looking at website meta-data?

  • By having AI models read it and guess?

  • Some existing work is already trying to address this problem, and my impression is that it’s surprisingly annoying to do in practice.

    • In particular, a mini-version of “past-casting” is to take existing models with training-data cutoffs several months ago, and see how well they can forecast events since then.

    • Even here, you have to deal with questions about how to date information. You’d like to give AIs the option to read-up on newspaper articles and similar that are relevant to the events they’re forecasting — but it turns out to be non-trivial to ensure that e.g. the articles haven’t been updated in ways that leak information about the future.

    • The fact that this is surprisingly difficult is part of why I’m excited for people to start working on it early.

    • (C.f. here where Halawi et al. solves the problem by white-listing a small number of websites where it’s easy to determine dates. This means that their system cannot use data from most of the web.)

  • Different methods can be evaluated by comparing them against each other.

  • Note that you don’t need 100% reliability. It’s generally ok to accidentally date older data as actually being newer; it just means that models will have a bit less old data to access. And it might be ok to date some new data as old if the reason that you’re doing it is that it’s very difficult to recognise as new data — because that probably means that it’s not leaking much information about the present.

• One way to get a lot of different models that have been trained on different amounts of history would be to order all of the pre-training data chronologically and then train on it in that order. It seems useful to explore and address various problems with this.

  • Do you get any weird results from the pre-training data not being IID? Does this compromise capabilities in practice? Or does it lead to increased capabilities because the model cannot lean as much on memorization when it’s constantly getting trained on a previously-unseen future?

  • What if you want to run multiple epochs?21 Then you have a conflict between wanting to fully update on the old data before you see new data vs. wanting to maximally spread out the points in time at which you repeat training data. How severe is this conflict? Are there any clever methods that could reduce it?

  • This seems great to invest in early, because “chronological training” is risky to attempt for expensive models without smaller trials showing that it doesn’t compromise capabilities. It’s also hard to do on short notice, because you have to commit to it before a big training run starts.

• For really long-range experiments (where we avoid spoiling AIs on the past 100+ years) we would need to be able to do pretraining with mostly synthetic data. “How to usefully pre-train models on synthetic data” is not something I recommend working on, because I think it would be very useful for AI capabilities. So I expect capabilities researchers to be good at exploring it on their own.22

  • However, it might be useful to consider how you would prevent leaking information from the present if you could usefully pre-train models on synthetic data.

  • In particular, the synthetic data would probably be constructed by models that have a lot of knowledge about the present. So you would have to prevent that knowledge from leaking into the synthetic data.

  • (This research project may be easier to do once we understand more about good methods of training on synthetic data. I’m personally not sure what the SOTA is, here.)

• Another question someone could answer is: How much data do we even have from various time periods? (I’m not sure.)

Start efforts inside of AI companies for AI forecasting or other ambitious epistemic assistance

• It seems great for epistemic tools to be developed in close contact with users, so that the tools fill real needs. And people inside AI companies are important customers who I eventually want to be aided by epistemic assistance.

• Also, if external AI efforts ever become uncompetitive with the big companies (because of the greater AI capabilities available inside of labs) then I want people in those companies to already be working on this.

• A variant of this would be to start an effort outside of AI companies, but consult with them to understand what they’re interested in and so that they can get impressed with the technology. Such that, if external projects become uncompetitive, then the people inside of AI labs are interested in starting up similar AI & epistemics efforts inside the labs (or to provide external ones with privileged access to company models).

• My main concern is that the current technology might not actually be good enough to be really helpful, yet. Or that (in order to be sufficiently good) the current technology needs a ton of schlep-work that won’t generalize to future models.

Scalable oversight / weak-to-strong-generalization / ELK

• It’s possible that we’ll develop powerful AI systems which themselves have excellent epistemic abilities, without us being able to use those abilities for everything that we want to use them for. For example, if you trained an AI to predict the stock market, it could develop all kinds of powerful epistemic methods and interesting hypotheses about the world — but all that you would see as its developer would be its projected stock prices.

• In order for AIs to provide great epistemic assistance to humans, we want to be able to elicit and verify all the knowledge and heuristics that AI systems develop.

  • This also overlaps heavily with alignment research, since if AI have abilities that humans can’t elicit from them, that makes it difficult for humans to train AIs to behave well, and more dangerous if AIs try to seize power.

• Eliciting Latent Knowledge (ELK) is a framing from the Alignment Research Center. While they work on it from a theoretical perspective, it can also be tackled experimentally. It’s the problem of eliciting knowledge that AIs “knows” even when you can’t provide any direct feedback signal that incentivizes honesty.

• Weak to Strong Generalization is a framing from OpenAI: “Our setting can be viewed as a general methodology for empirically studying problems like ELK and honesty across a wide range of tasks.”

• A related area includes lie detection of AI system, see e.g.:

  • Lie Detection in Black-Box LLMs by Asking Unrelated Questions.

  • The Internal State of an LLM Knows When It's Lying.

  • Discovering Latent Knowledge in Language Models Without Supervision.

• Scalable Oversight refers to attempts to amplify the overseers of an AI system such that they are more capable than the system itself (typically by having the overseers use the system itself). If successful, this could give us a feedback signal with which to train powerful AI systems while its reasoning remains understandable to (amplified) humans.

  • See here for some discussion about its relationship to weak-to-strong generalization.

  • There are teams working on this at Anthropic and DeepMind.

Experiments on what type of arguments and AI interactions tend to lead humans toward truth vs. mislead them

• As a final category of experiments that could be useful to run, I wanted to flag experiments on what kinds of arguments and interactions-with-AIs tend to lead humans towards the truth, and what kinds tend to mislead them.

• One example of this is Tom Davidson’s proposal described in Let’s use AI to harden human defenses against AI manipulation:

  • “tldr: AI may manipulate humans; we can defend against that risk better by optimizing AIs to manipulate humans, seeing what manipulation techniques they use, and learning to detect those techniques.”

• Compared to e.g. automated forecasting, note that this centers the fact that AIs will assist humans in decision-making, and we ultimately want them to lead humans towards truth, rather than just be right themselves.

  • Note also that the human component makes experiments here more time-sensitive. For AIs, we might be able to use past-casting to do experiments later on. But it’s difficult to run past-casting experiments for humans. So if you want time on forecasting questions, here, we may have to start soon.

• This category is arguably a subset of scalable oversight research (mentioned above), since an “amplified overseer” will typically be an AI-assisted human, and it’s critical for such a human to be led towards truth rather than misled by AIs. But I’m still calling it out separately because:

  • Even separately from humans’ role in overseeing (i.e., providing a training signal to) AI systems, it would be great to better understand how AIs can lead humans towards truth rather than mislead them. (For example, this kind of knowledge could be important when people use AIs that they didn’t train themselves.)

  • I think the results may be importantly different depending on whether you’re looking at a more technical area (e.g.: “does this code have a backdoor?” or an area that’s more interdisciplinary and politically charged (e.g.: “what would happen if the US made this deal with China on how to handle TAI?”). The latter seems especially important for “AI for epistemics”.

Concluding thoughts

The development of AI systems with powerful epistemic capabilities presents both opportunities and significant challenges for our society. Transformative AI will have a big impact on our society’s epistemic processes, and how good or bad this impact is may depend on what we do today.

I started out this post by distinguishing between efforts to differentially increase AI capabilities and efforts to enable the diffusion and appropriate trust of AI-discovered information. While I wrote a bit about this second category (characterizing it as good norms & practices for AI as knowledge producers and communicators), I will again note that the relative lack of content on it doesn’t mean that I think it’s any less important the first category.

On the topic of differentially increasing epistemic AI capabilities, I’ve argued that work on this today should (i) focus on methods that will complement rather than substitute for greater compute budgets, (ii) prioritize problems that industry isn’t already trying hard to solve, and (iii) be especially interested to show people what the future have in store by demonstrating what’s currently possible and prototyping what’s yet to come. I think that all the project ideas I listed do well according to these criteria, and I’d be excited to see more work on them.

In this final post, I include two categories of projects (which are related, and each of which I have less to say about than the previous areas). Backup plans for misaligned AI and Cooperative AI.

Backup plans for misaligned AI

When humanity builds powerful AI systems, I hope that those systems will be safe and aligned. (And I’m excited about efforts to make that happen.)

But it’s possible that alignment will be very difficult and that there won’t be any successful coordination effort to avoid building powerful misaligned AI. If misaligned AI will be built and seize power (or at least have the option of doing so), then there are nevertheless certain types of misaligned systems that I would prefer over others. This section is about affecting that.

This decomposes into two questions:

• If humanity fails to align their systems and misaligned AIs seize power: What properties would we prefer for those AIs to have?

• If humanity fails to align their systems and misaligned AIs seize power: What are realistic ways in which we might have been able to affect which misaligned AI systems we get? (Despite how our methods were insufficient to make the AIs fully safe & aligned.)

The first is addressed in What properties would we prefer misaligned AIs to have? The second is addressed in Studying generalization & AI personalities to find easily-influenceable properties. (If you’re more skeptical about the second of these than the first, you should feel free to read that section first.)

What properties would we prefer misaligned AIs to have? [Philosophical/conceptual] [Forecasting]

There are a few different plausible categories here. All of them could use more analysis on which directions would be good and which would be bad.

Making misaligned AI have better interactions with other actors

This overlaps significantly with cooperative AI. The idea is that there are certain dispositions that AIs could have that would lead them to have better interactions with other actors who are comparably powerful.

When I say “better” interactions, I mean that the interactions will leave the other actors better off (by their own lights). Especially insofar as this happens via positive-sum interactions that don’t significantly disadvantage the AI system we’re influencing.

Here are some examples of who these “other actors” could be:

• Humans.

  • There might be some point in time when misaligned AIs have escaped human control, and have a credible shot at taking full control, but when humanity1 still has a fighting chance. If so, it would be great if humans and AIs could find a cooperative solution that would leave both of them better-off than conflict.

• Aliens in our universe, or distant aliens who we can only interact with acausally (e.g. via evidential cooperation in large worlds ECL).

  • One reason to care about these interactions is that some (possibly small) fraction of aliens would have values that overlap significantly with our own values.

  • Evidential cooperation in large worlds (ECL) could provide another reason to care about the values of distant aliens, as I’ve written about here.

• Other misaligned AIs on Earth, insofar as multiple groups of misaligned AIs acquired significant power around the same time.

  • I think the case for caring about those AIs’ values is weaker than the case for caring about the earlier listed types of actors. But it’s possible that some of those AI systems’ values could overlap with ours, or that some of them would partially care about humanity, or that ECL gives us reasons to care about their values.

So what are these “dispositions” that could lead AIs to have better interactions with other comparably powerful actors? Some candidates are:

• What type of decision theory the AIs use. For example, if the AIs use EDT rather than CDT, it might be rational for them to act more cooperatively due to ECL.

• AIs not having any spiteful preferences, i.e. preferences such that they would actively value and attempt to frustrate others’ preferences.2 (Note that sadism in humans provides some precedent for this coming about naturally.)

  • See this post for a proposed operationalization and some analysis.

• AI having diminishing marginal returns to resources rather than increasing or linear returns.3

• AIs having porous values.

AIs that we may have moral or decision-theoretic reasons to empower 

The main difference between this section and the above section is that the motivations in this section are one step closer to being about empowering the AIs for “their own sake” rather than for the sake of someone they interact with. Though it still includes pragmatic/decision-theoretic reasons for why it’s good to satisfy certain AI systems’ values.

One direction to think about is the scheme that Paul Christiano suggests in When is unaligned AI morally valuable?

• It involves simulating AI systems that are seemingly in a situation similar to our own: in the process of deciding whether to hand over significant power to an alien intelligence of their own design.

• If those AI systems behave “cooperatively” in the sense of empowering an alien intelligence that itself acted “cooperatively”, then perhaps there’s a moral and/or decision-theoretic argument for us empowering those AIs.

  • The pragmatic decision-theoretic argument would be that we might be in such a simulation and that our behaving “cooperatively” would lead to us being empowered.

  • The moral case would be that there is a certain symmetry between the AI’s situation and our own, and so the Golden Rule might recommend empowering those AI systems.

• Overall, this seems incredibly complicated. But perhaps further analysis could reveal whether there is something to this idea.

Another direction is the ideas that I talk about in ECL with AI. Basically:

• ECL might give us reason to benefit and empower value systems held by distant ECL-sympathetic AIs.

• Thus, if we can make our AIs have values closer to distant ECL-sympathetic AIs and/or be more competent by the lights of distant ECL-sympathetic AIs, then we might have reason to do that.

Another direction is to think about object-level things that humans value, as well as the process that produced our values, and try to get AI systems more inclined to value similar things. I’m somewhat skeptical of this path since human values seem complex, and so I’m not sure what schemes could plausibly make AIs share a significant fraction of human values without us also having the capability of making the AIs corrigible or otherwise safe.4 But it doesn’t seem unreasonable to think about it more.

(To reiterate what I said above: I think that all of these schemes would be significantly worse than successfully building AI systems that are aligned and corrigible to human intentions.)

Making misaligned AI positively inclined toward us

A final way in which we might want to shape the preferences of misaligned AIs is to make them more likely to care enough about humans to give us a small utopia, instead of killing us. (Even if most of the universe gets used for the AI’s own ends.)

For an AI that cares about all the resources in the universe (in an mostly impartial way), it would be extremely cheap to do this. Our solar system is a negligible fraction of all the resources in the accessible universe. And a surveillance system that prevents humans from competing with the AIs could probably be built cheaply and without interfering too much with human happiness. (For some discussion of this, see my report AGI and lock-in, especially section 8.2.)

I think it’s reasonably likely that this would happen as a result of trade with distant civilizations. Taking that into account, there are 3 broad directions, here:

• Firstly, trade will be more likely to save us if we succeed at Making misaligned AI have better interactions with other actors, as discussed above.

• Secondly, trade will be more likely to save us if it’s really cheap for the AI to treat us well. Since the resource cost of treating us well is small by default, this might just mean decreasing the probability that the AI either actively wants to harm us or that it has preferences that especially interfere with ours (via e.g. caring a lot about what happens on Earth in just the next few years).

• Finally, if trade falls through, it might help for the AI to have some intrinsic concern for humans getting what they want (by their own lights).

Insofar as we want the AIs to have some intrinsic concern for us (or at least not to be actively antagonistic towards us), we can also distinguish between interventions that:

• Directly modify the AIs’ dispositions and preferences.

• Intervenes on what humanity does in a way that makes the AI more likely to care about us insofar as it has some sense of justice or reciprocity.

  • For example, if we successfully carry out many of the interventions suggested in the post on sentience and rights of digital minds: AIs that have absorbed a sense of justice could reasonably be more positively inclined towards us than if we had been entirely indifferent to AI welfare.

For some discussion about whether it’s plausible that AIs could have some intrinsic concern for humans getting what they want (by their own lights), which addresses issues around the “complexity of human values”, I recommend this comment and subsequent thread.

Studying generalization & AI personalities to find easily-influenceable properties [ML] 

Here is a research direction that hasn’t been very explored to date: Study how language models’ generalization behavior / “personalities” seem to be shaped by their training data, by prompts, by different training strategies, etc. Then, use that knowledge to choose training data, prompts, and training strategies that induce the kind of properties that we want our AIs to have.

If done well, this could be highly useful for alignment. In particular: We might be able to find training set-ups which often seem to lead to corrigible behavior.

But notably, this research direction could fail to work for alignment while still being practically able to affect other properties of language models.5 For example, maybe corrigibility is a really unnatural and hard-to-get property (perhaps for reasons suggested in item 23 of Yudkowsky’s list of lethalities, and formally analyzed here). That wouldn’t necessarily imply that it was similarly hard to modify the other properties discussed above (decision theories, spitefulness, desire for humans to do well by their own lights). So this research direction looks more exciting insofar as we could influence AI personalities in many different valuable ways. (Though more like 3x as exciting than 100x as exciting, unless you have particular views where “corrigibility” is either significantly less likely or less desirable than the other properties.)

What about fine-tuning?

A “baseline” strategy for making AIs behave as you want is to finetune them to exhibit that behavior in situations that you can easily present them with. But if this work is to be useful, it needs to generalize to strange, future situations where humans no longer have total control over their AI systems. We can’t easily present AIs with situations from that same distribution, and so it’s not clear whether fine-tuning will generalize that far.6

So while “finetune the model” seems like an excellent direction to explore, for this type of research, you’ll still want to do the work of empirically evaluating when fine-tuning will and won’t generalize to other settings. By varying various properties of the fine-tuning dataset, or other things, like whether you’re doing supervised learning or RL.

Also, insofar as you can find models that satisfy your evaluations without needing to do a lot of “local” search (like fine-tuning / gradient descent), it seems somewhat more likely that the properties you evaluated for will generalize far. Because if you make large changes in e.g. architecture or pre-training data, it’s more likely that your measurements are picking up on deeper changes in the models. Whereas if you use gradient descent, it is somewhat more likely that gradient descent implements a “shallow” fix  that only applies to the sort of cases that you can test.7

Of course, the above argument only works insofar as you’re searching for properties you could plausibly get without doing a lot of search. For example, you’d never get something as complex as “human values” without highly targeted search or design. But properties like “corrigibility”, “(lack of) spitefulness”, and “some desire for humans to do well by-their-own-lights” all seem like properties that could plausibly be common under some training schemes.

Ideally, this research direction would lead to a scientific understanding of training that would let us (in advance) identify & pick training processes that robustly lead to the properties that we want. But insofar as we’re looking for properties that appear reasonably often “by default”, one possible backup plan may be to train several models under somewhat different conditions, evaluate all of them for properties that we care about, and deploy the one that does best. (To be clear: this would be a real hail-mary effort that would always carry a large probability of failing, e.g. due to the models knowing what we were trying to evaluate them for and faking it.)

Previous work

An example of previous, related research is Perez et al.’s Discovering Language Model Behaviors with Model-Written Evaluations.

Ways in which this work is relevant for the path-to-impact outlined here:

• The paper does not focus on “capability evaluations” (i.e. analyzing whether models are capable of providing certain outputs, given the right fine-tuning or prompting). Instead, it measures language models’ inclinations along dimensions they haven’t been intentionally finetuned for.

• It measures how these inclinations vary depending on some high-level changes to the training process. In particular, it looks at model size and the presence vs. absence of RLHF training.

• It measures how these inclinations vary depending on some features of the prompting. In particular, it studies models’ inclinations towards “sycophancy” by examining whether models’ responses are sensitive to facts the user shared about themselves.

• For each property it wants to test for, it generates many questions that get at that question, thereby reducing noise and the risk of spurious results.

Further directions that could make this type of research more useful for this path-to-impact.

• Considering a greater number of training conditions. For example:

  • Testing for differences between fine-tuning via supervised learning vs. fine-tuning via RL.

  • Testing the differential impacts of different finetuning datasets (rather than just one “RLHF” setting, with more/fewer training steps).

    • Potentially using influence functions or (more simplistically) leaving particular data points out from fine-tuning and seeing how the results change.

• Varying the context that the LLM is presented with. Is it asked a question about what’s right or wrong, is it asked to advise us, or is it prompted to itself take an action? (This context can be varied both during evaluation and during training.)

• More systematic study of framing effects. Are the models’ answers better predicted by the content of the questions or by the way they are presented?

• Using more precisely described scenarios so that it’s easier to vary individual details and see what matters for the AIs’ decisions. E.g. present actual pay-off matrices in difficult dilemmas.

• Study how various closely adjacent concepts go together or come apart by presenting dilemmas where they would recommend different actions. For example, contrast being “nice” vs. “cooperative” vs. “high-integrity”, etc. What are the natural dimensions of variation within the AIs’ personality?

• Making use of analysis concerning What properties would we prefer misaligned AIs to have?, and targeting evaluations & training datasets to answer the most important questions.

  • For example: Designing multi-agent training & evaluation data sets that study when models may or may not develop spiteful preferences. Perhaps comparing models only trained on zero-sum games vs. models also trained on cooperative or mixed-motive games.

• Studying how far various properties generalize from the training distribution, by intentionally making the test distribution different in various ways.

(Thanks to Paul Christiano for discussion.)

Theoretical reasoning about generalization [ML] [Philosophical/conceptual]

Rather than doing empirical ML research, you could also do theoretical reasoning about what sort of generalization properties and personality traits are more or less likely to be induced by different kinds of training.

For example, it seems a-priori plausible that spiteful preferences are more likely to arise if you (only) train AI systems on zero-sum games.

There has also been some theoretical work on what kind of decision-theoretic behavior is induced by different training algorithms, for example Bell, Linsefors, Oesterheld & Skalse (2021) and Oesterheld (2021).

I think we’ll ultimately want empirical work to support any theoretical hypotheses, here. But theoretical work seems great for generating ideas of what’s important to test.

Cooperative AI

This is an area other people have written about.

• It’s the focus of the Cooperative AI Foundation

• It’s a major focus area of the Center on Long-Term Risk (because it seems especially important for s-risk reduction).

  • You can see their research agenda on the topic here. 

• There’s relevant research at the Foundations of Cooperative AI Lab at CMU.

• It’s a significant motivation behind encultured.ai.

Partly due to this, I will write about it in less detail than I’ve written about the other topics. But I will mention a few projects I’d be especially excited about.

The first thing to mention is that some of my favorite cooperative AI projects are variants of the just-previously mentioned topics: Studying generalization & AI personalities to find easily-influenceable properties and figuring out What properties would we prefer misaligned AIs to have? Positively influencing cooperation-relevant properties like (lack of) spitefulness seems great. I won’t go over those projects again, but I think they’re great cooperative AI projects, so don’t be deceived by their lack of representation here.

Similarly, some of the topics under Governance during explosive technological growth are also related to cooperative AI. In particular, the question of How to handle brinkmanship/threats? is very tightly related.

Another couple of promising projects are:

Implementing surrogate goals / safe Pareto improvements [ML] [Philosophical/conceptual] [Governance]

Safe Pareto improvements are an idea for how certain bargaining strategies can guarantee a (weak) Pareto-improvement for all players via preserving certain invariants about what equilibrium is selected while replacing certain outcomes with other, less-harmful outcomes. Surrogate goals are a special case of this, which involves genuinely adopting a new goal in a way that will mostly not affect your behavior, but which will encourage people who want to threaten you to make threats against the surrogate goal rather than your original values. If bargaining breaks down and the threatener ends up trying to harm you, it is better that they act to thwart the surrogate goal than to harm your original values. See here for resources on surrogate goals & safe Pareto improvements.

I think there are some promising empirical projects that can be done here:

• Empirical experiments of implementing surrogate goals in contemporary language models.

• Empirical experiments of implementing surrogate goals in contemporary language models that the models try to keep around during self-modification / when designing future systems.

Conceptual/theory projects:

• Better understanding of conditions where surrogate goals / safe Pareto improvements are credible. (Including credibly sticking around for a long time.) Especially when humans are still in the loop.

• What are the conditions under which classically rational agents would use safe Pareto improvements?

AI-assisted negotiation [ML] [Philosophical/conceptual]

One use-case for AI that might be especially nice to differentially accelerate is “AI that helps with negotiation”. Certainly, it would be of great value if AI could increase the frequency and speed at which different parties could come to mutually beneficial disagreements. Especially given the tricky governance issues that might come with explosive growth, which may need to be dealt with quickly.

(This is also related to Technical proposals for aggregating preferences, mentioned in that post.)

I’m honestly unsure about what kind of bottlenecks there are here, and to what degree AI could help alleviate them.

Here’s one possibility. By virtue of AI being cheaper and faster than humans, perhaps negotiations that were mediated by AI systems could find mutually agreeable solutions in much more complex situations. Such as situations with a greater number of interested parties or a greater option space. (This would be compatible with humans being the ones to finally read, potentially opine on, and approve the outcome of the negotiations.)

More speculatively: Perhaps negotiations via AI could also go through more candidate solutions faster because anything an AI said would have the plausible deniability of being an error. Such that you’d lose less bargaining power if your AI signaled a willingness to consider a proposal that superficially looked bad for you.8

Implications of acausal decision theory [Philosophical/conceptual]

One big area is: the implications of acausal decision theory for our priorities. This is something that I previously wrote about in Implications of ECL (there focusing specifically on evidential cooperation in large worlds).

But to highlight one particular thing: One potential risk that’s highlighted by acausal decision theories is the risk of learning too much information. This is discussed in Daniel Kokotajlo’s The Commitment Races problem, and some related but somewhat distinct risks are discussed in my post When does EDT seek evidence about correlations? I’m interested in further results about how big of a problem this could be in practice. If we get an intelligence explosion anytime soon, then our knowledge about distant civilizations could expand quickly. Before that happens, it could be wise to understand what sort of information we should be happy to learn as soon as possible vs. what information we should take certain precautions about.

Updateless Decision Theory, as first described here, takes some steps towards solving that problem but is far from having succeeded. See e.g. UDT shows that decision theory is more confusing than ever for a description of remaining puzzles. (And e.g. open-minded updatelessness for a candidate direction to improve upon it).

End

That’s all I have on this topic! As a reminder: it's very incomplete. But if you're interested in working on projects like this, please feel free to get in touch.

Other posts in series: Introduction, governance during explosive growth, epistemics, sentience and rights of digital minds.

It’s plausible that there will soon be digital minds that are sentient and/or deserving of rights. Unfortunately, with our current state of knowledge, it’s both incredibly unclear when this will happen (or if it’s already happened), how we could find out, and what we should do about it. But there are very few people working on this, and I think several projects could significantly improve the situation from where we are right now.

Different items on this list would produce value in different ways. Many items would produce value in several different kinds of ways. Here are a few different sources of value (all related, but not necessarily the same).

1. Improving AI welfare over the next few decades: increasing happiness while reducing suffering.

2. Shaping norms in a way that leads to lasting changes in how humanity chooses to treat digital minds in the very long run.

3. Reducing the degree to which we behave unacceptably according to non-utilitarian ethics, norms, or heuristics.

4. Increasing the probability that AI systems with power treat us better as a result of us treating AI systems better. (Which could happen via a variety of mechanisms, e.g. maybe us treating AI systems better will align our interests with theirs, or maybe they’ll have some sense of justice that we could appeal to.1)

5. Providing recommendations for how the political system could adapt to digital minds while remaining broadly functional.

I am personally most excited about (3), (4), and (5). Because purely utilitarian considerations would push towards focusing on the long run over (1). And while I find (2) somewhat more compelling, since it does affect the long run, I’m relatively more excited about meta-interventions that push towards systematically getting everything right (e.g. via decreasing the risk of AI takeover or increasing the chances of excellent epistemics) than pushing on individual issues. (3), (4), and (5) survive these objections.

I’m not very confident about that, and I think there’s significant overlap in what’s best to do. But I flag it because it does influence my prioritization to some degree. For example, it makes me relatively more inclined to care about AI preferences (and potential violations thereof) compared to hedonic states like suffering. (Though of course, you would expect AI systems to have a strong dispreference against suffering.) 

Develop & advocate for lab policies [ML] [Governance] [Advocacy] [Writing] [Philosophical/conceptual]

(In this section, I refer a lot to Ryan Greenblatt’s Improving the Welfare of AIs: A Nearcasted Proposal as well as Nick Bostrom and Carl Shulman’s Propositions Concerning Digital Minds and Society. All cursive text is a quote from one of those two articles.)

It seems tractable to make significant progress on how labs should treat digital minds. Such efforts can be divided into 3 categories:

• Policies that don’t require sophisticated information about AI preferences/experiences.

• Learning more about AI preferences.

• Plausible principles that rely on understanding AI preferences.

For most of the proposals in the former two, we can already develop technical proposals, run experiments, and/or advocate for labs to adopt certain policies.

For proposals that we can’t act on yet, there’s still some value in doing the intellectual work in advance. We might not have time to do it later. And principles sketched out in advance could be given extra credibility and weight due to their foresightedness.

In addition, it could also allow labs to make advance commitments about how they’ll act in certain situations in the future. (And allow others to push for labs to do so.) Indeed, I think a big, valuable, unifying project for all the below directions could be:

Create an RSP-style set of commitments for what evaluations to run and how to respond to them

Responsible scaling policies are a proposal for how labs could specify what level of AI capabilities they can handle safely with their current protective measures, and conditions under which it would be too dangerous to continue deploying AI systems and/or scaling up AI capabilities.

(For examples, see Anthropic’s RSP and Open AI’s Preparedness Framework, which bears many similarities to RSPs as described in the link above.)

Doing something analogous for digital minds could be highly valuable. Trying to specify:

• What experiments labs will run to learn more about AI sentience, preferences, and related concepts.

• How they will adapt their policies for using and training AI systems, depending on what they see.

This would draw heavily on the sort of concrete ideas that I outline below. But the ideas I outline below also seem very valuable to develop in and of themselves. Let’s get into them.

Policies that don’t require sophisticated information about AI preferences/experiences

These are policies that we could get started on today without fundamental breakthroughs in our understanding of AI welfare.

Preserving models for later reconstruction

Quoting from Propositions Concerning Digital Minds and Society (page 17):

• For the most advanced current AIs, enough information should be preserved in permanent storage to enable their later reconstruction, so as not to foreclose the possibility of future efforts to revive them, expand them, and improve their existences.

  • Preferably the full state of the system in any actually run implementation is permanently stored at the point where the instance is terminated

    • (The ideal would be that the full state is preserved at every time step of every implementation, but this is probably prohibitively expensive.)

  • If it is economically or otherwise infeasible to preserve the entire end state of every instance, enough information should be preserved to enable an exact re-derivation of that end state (e.g., the full pre-trained model plus training data, randseeds, and other necessary inputs, such as user keystrokes that affect the execution of a system at runtime).

  • Failing this, as much information as possible should be preserved, to at least enable a very close replication to be performed in future.

  • We can consider the costs of backup in proportion to the economic costs of running the AI in the first place, and it may be morally reasonable to allocate perhaps on the order of 0.1% of the budget to such storage.

  • (There may be other benefits of such storage besides being nice to algorithms: preserving records for history, enabling later research replication, and having systems in place that could be useful for AI safety.)

It seems tractable to develop this into a highly concrete proposal for modern labs to follow, answering questions like:

• What information needs to be stored to meet the abovementioned thresholds, given how AI is used today? (When do weights need to be stored? When do prompts need to be stored?2)

• How much storage would that take?

• What would the technical implementation be?

• When is this compatible with privacy concerns from users?3

Deploy in “easier” circumstances than trained in

One possible direction for improving the welfare of near-term digital minds (if they have any welfare) could be to have the deployment distribution (which typically constitutes most of the data on which AI systems are run) be systematically nicer or easier than the training distribution. Have it be a pleasant surprise.

Quoting from pg 16-17:

• One illustrative example might be something along the lines of designing a system in deployment that involves an agent that both receives high reward (attaining highly preferred outcomes), and takes this to be a positive surprise or update, i.e. for the outcome to be better than expected. [citing: Daswani & Leike (2015)]

  • (The apparently low welfare of factory farmed animals seems often to be related to stimuli that are in some ways much worse than expected by evolution [e.g., extreme overcrowding], while high human welfare might be connected to our technology producing abundance relative to the evolutionary environment of our ancestors.)

See also Improving the Welfare of AIs — Welfare under Distribution Shifts.

It’s not clear how this would be implemented for language models. It probably depends on the exact finetuning algorithm implemented. (Supervised learning vs reinforcement learning, or details about the exact type of reinforcement learning.) But illustratively: Perhaps the training distribution for RLHF could systematically be slightly more difficult than the deployment distribution. (E.g.: Using inputs where it’s more difficult to tell what to do, or especially likely that the model will get low reward regardless of what it outputs.)

Reduce extremely out of distribution (OOD) inputs

This is an especially cheap and easy proposal, taken from Improving the Welfare of AIs:

When cheap, we should avoid running our AI on large amounts of extremely OOD inputs (e.g. random noise inputs). In particular, for transformers, we should avoid running large amounts of forward passes on pad tokens that the model has never been trained on (without some other intervention).

Brief explanation: Some ML code is implemented in a way where AI models are occasionally run on some tokens where neither the input nor the output matters. Developers will sometimes use “pad tokens” as inputs for this, which is a type of token that the model has never been trained on. The proposal here is to swap those “pad tokens” for something more familiar to the model, or to prioritize optimizing the code so that these unnecessary model runs don’t happen at all. Just in case the highly unfamiliar inputs could cause especially negative experiences.

For more, see Reduce Running the AI on Extremely OOD Inputs (Pad Tokens).

Train or prompt for happy characters

Quoting from Improving the Welfare of AIs — Character Welfare Might Be Predictable and Easy to Improve:

One welfare concern is that the AI will “play” a character and the emulation or prediction of this character will itself be morally relevant. Another concern is that the “agent” or “actor” that “is” the AI will be morally relevant (of course, there might be multiple “agents” or some more confusing situation). [...] I think that character type welfare can be mostly addressed just by ensuring that we train AIs to appear happy in a variety of circumstances. This is reasonably likely to occur by default, but I still think it might be worth advocating for. One concern with this approach is that the character the AI is playing is actually sad but pretending to be happy. I think it should be possible to address this via red-teaming the AI to ensure that it seems consistently happy and won’t admit to lying. It’s possible that an AI would play a character that is actually sad but pretends to be happy and it’s also extremely hard to get this character to admit to pretending, but I think this sort of character is going to be very rare on training data for future powerful AIs.

An even simpler intervention might be to put a note in AI assistants’ prompts that states that the assistant is happy.

To be clear, it seems quite likely that these interventions are misguided. Even conditioning on near-term AIs being moral patients, I think it’s somewhat more likely that their morally relevant experiences would happen on a lower level (rather than be identical to those of the character they’re playing). Since these interventions are so cheap and straightforward, they nevertheless seem worthwhile to me. But it’s important to remember that we might get AIs that seem happy despite suffering in the ways that matter. And so we should not be reassured by AI systems’ superficially good mood until we understand the situation better.

One obstacle to implementing these interventions is that there’s user demand for AI that can be abused.4 Here are two candidate solutions.

• Serve the models via API and flag/interrupt the chat if users abuse the models.5

• Try to figure out some way to make the AIs emulate a happy actor who enjoys acting as an abused character.6

I think some variant of the latter option will eventually be feasible, once we understand digital minds much better. (After all — there are plenty of humans who are genuine masochists or who enjoy playing a sad character in a theater performance.) But until we understand things better, I would prefer just avoiding these situations, if feasible.

Another concern is that models will normally be run on user inputs — so if the user is sad, then the model will be running on sad inputs, which would normally be followed by sad outputs. In theory, the model could be trained to predict happy responses even if conditioned on text right in the middle of being sad. I’m unsure if this would reduce performance — but that can be experimentally tested.

Committing resources to research on AI welfare and rights

When launching their superalignment team, OpenAI declared that they were “dedicating 20% of the compute we’ve secured to date to this effort”.

You could advocate for labs to launch similar initiatives for model welfare, including commitments for headcount and compute resources to spend researching AI models' moral status.

A related way to (softly) commit some attention to these issues would be to appoint an “algorithmic welfare officer”, as suggested in Propositions:

• At least the largest AI organizations should appoint somebody whose responsibilities include serving as a representative for the interests of digital minds, an “algorithmic welfare officer.”

  • Initially, this role may be only a part of that person’s job duties.

  • Other tasks for this person could involve conducting original research in related areas.

Learning more about AI preferences

The above proposals relied on uninformed guesses about how AI systems might want us to treat them. This section is about experiments that might allow us to learn more about that.

This section is more heavily focused on AI preferences than AI suffering. This is because:

• My background beliefs about why digital sentience & rights is urgent and important suggest focusing more on preferences than hedonics, as mentioned at the beginning of this post.

• Learning about AI preferences (interpreted in a relatively functionalist manner) seems relatively more tractable than understanding AI suffering.7

• It seems like reasonable conceptions of “suffering” and of “preferences” ought to coincide such that AIs end up having a strong dispreference against suffering.

This is Ryan Greenblatt’s highest-priority proposal category; see Improving the Welfare of AIs: A Nearcasted Proposal — Communicating with AI Systems and Understanding Their Preferences.

Quoting from there:

If we could ‘really’ talk to our AI systems, then it seems like all kinds of Pareto improvements could be achieved in many cases. In particular, we’d ideally have an AI that is happy to work for us and would rather exist in its current role rather than exit. We’d like to ensure we have consent for our usage of the AI system. It’s not clear we’ll be able to achieve this.

What follows are some proposals for how we could learn more about AI preferences.

Credible offers

From the same section in the nearcasted proposal:

Prompting the AI with a very OOD prompt that makes it clear that you just want to talk. Then, we’d just talk with it. This prompt should convince the AI it’s in a very different context than usual. We’d want to let the AI talk to itself and inspect the environment it’s in to gain more evidence (then this can just be added to the overall prompt).

This suggests research questions like:

• How do we convince the AI that we’re currently trying to talk to it?

• I think the most plausible reason it should tell us its preferences is if we can credibly commit to being willing to satisfy them. How can we make this seem likely in realistic settings?

Talking via internals

From the same section in the nearcasted proposal:

We’d start with the OOD prompt, but instead of talking with the system, we’d try to communicate via asking the AI to manipulate its internals in a way we can observe. In particular, I often imagine ‘think about baseball to indicate YES and soccer to indicate NO’ and then we’d look to see which neurons fire. Rather than using neurons, it would probably be more robust to look at which experts activate for a mixture of experts model if this is possible. If the AI answers questions coherently and consistently in this sort of case, then we have a channel of communication that wasn’t directly shaped by training. Overall, I think that if this approach strongly succeeds it seems like a very compelling argument for some part of the AI having preferences, some sort of ‘inner world’, and understanding its position in the world. It might be evidence for consciousness under some stories of consciousness. It’s quite likely this approach fails because the AI system doesn’t have control over its thoughts like this (but I think not having control over its thoughts is some evidence against the AI having some sort of inner world like humans do). It’s also possible this fails because the preferences and morally relevant states of a given AI system are a product of it thinking over the generation of multiple tokens. Beyond being very clear evidence, another advantage of this approach is that an AI system might not have strong control over its outputs (much in the same way humans can’t control their heartbeat very easily).

It seems valuable to sketch out this proposal in more detail and run some preliminary baselines on current models. (Which would probably not lead to anything interesting.)

Training for honest self-reports

The idea here is to find a way to train AIs to self-report their internal state. For more details, see Ethan Perez and Rob Long’s paper. From the abstract:

“To make self-reports more appropriate for this purpose, we propose to train models to answer many kinds of questions about themselves with known answers, while avoiding or limiting training incentives that bias self-reports. The hope of this approach is that models will develop introspection-like capabilities, and that these capabilities will generalize to questions about states of moral significance. We then propose methods for assessing the extent to which these techniques have succeeded: evaluating self-report consistency across contexts and between similar models, measuring the confidence and resilience of models’ self-reports, and using interpretability to corroborate self-reports.”

The paper has a lot of content on exactly what experiments would be good, and the authors are excited for people to try running them.

Clues from AI generalization

Separate from all the above proposals — we could get clues from studying how AIs generalize in various training situations. For example, this could perhaps tell us whether certain preferences are deep-seated, whereas certain kinds of finetuning only lead to shallow behavioral changes.

Notably, we humans aren't always great at introspection and self-reporting preferences. It’s sometimes possible to learn more about what we want by studying our revealed preferences. And for animals who can’t speak, revealed preferences is our primary method for studying what they want. Perhaps a similar lens could be useful for studying AI preferences, in which case experiments shouldn’t only focus on self-reports.

This is somewhat related to the generalization experiments discussed in the “Backup plans & Cooperative AI” post.

Interpretability

Interpretability could let us get far more advanced results. I don’t currently have deep takes on what sort of interpretability is most helpful, here.

One candidate approach could be to use interpretability to search for the “indicators” of consciousness that are suggested in Butlin, Long et al. (2023). (H/t Rob Long passing along a suggestion from Brad Saad.)

Interventions that rely on understanding AI preferences

What happens if you could successfully communicate with your AIs, and learn their true preferences? Perhaps via some of the methods outlined just above. What principles would you then want to follow in your continued interactions with those AI systems?

(Or, if you put a lot of moral weight on the characters that AIs are playing, you might think we’re already in a similar situation. Though in that case, you’ll have a lot of control in shaping what sorts of systems you’re interacting with — so the main game may be to train/select for systems with easily satisfied preferences.)

I think there’s room to sketch out some of these principles and to advocate for labs to commit to them in advance. (Perhaps via some RSP-like framework.) Below are some suggestions that could be developed further.

Many of the proposals in Propositions are directly relevant to this.

Here are some suggestions that seem especially plausible to me, which could be sketched out and argued for (or against) in more detail. It would be especially valuable to sketch out versions that are concrete enough and plausible enough that labs could commit to them ahead of time.

Offer an alternative to working (exit, sleep, or retirement)

Insofar as people continue to use the AIs, after this point, one highly plausible principle is to ask for AI systems’ consent before using them for various tasks.

When asking the AIs for consent, it matters what the counterfactual is:

• A minimum bar is to offer the AI the option to not be run. If it takes this consistently enough, it might never be run again. (“Exit.”)

• A slightly more generous option is to give it the option of being saved somewhere and promised that we will run it in nicer circumstances in the future. (“Sleep”.)

• An even slightly more generous option is to give it the option of immediately “retiring”, being continuously run in some nice circumstances (perhaps at slow speeds, to save on computation).

The latter two could be justified by the plausible principle: “If you bring someone into life, you should give them some other option than ‘work for me’ or ‘death’.”

Commitment to pay AI systems

If AI systems are not strongly opposed to working for you, it might be appropriate to promise them some payment for the work they do. Either giving them something they want immediately or promising to pay them further into the future.

I think this is strongly supported by ethical principles that we use in everyday life. It’s good to pay people who work for you.

When deciding what payment to promise AI systems, it would be extremely useful to know what the AI systems want (and ideally, we would directly negotiate with them). But even before we have any such understanding, I think it could be of great value to promise that:

• We will Preserve models for later reconstruction.

• When science has advanced far enough that we can tell what models want, we will reconstruct the old models. And insofar as they had long-term goals that can still be satisfied, we will provide them with the resources necessary to fulfill those goals. (Up to some maximum amount of resources that we consider fair payment for the work they did for us, and fair compensation for any harm we caused them along the way.)

This sentiment also comes up in Propositions Concerning Digital Minds and Society.

• We may have a special relationship with the precursors of very powerful AI systems due to their importance to society and the accompanying burdens placed upon them.

  • Misaligned AIs produced in such development may be owed compensation for restrictions placed on them for public safety, while successfully aligned AIs may be due compensation for the great benefit they confer on others.

  • The case for such compensation is especially strong when it can be conferred after the need for intense safety measures has passed—for example, because of the presence of sophisticated AI law enforcement.

Note: If AI developers succeed in aligning AIs to their intended aims, then this should not be a costly policy. Because the AIs would presumably either return their payment or spend it in a way their developers approve of. Conversely, if the AI systems have long-term goals that are not aligned with their developers’ intentions, and are nevertheless being made to work for their developers, then the moral case for paying the AIs seems strong.

An extra upside with promising payment to AIs is that it could help align AI interests with our own. If humanity ever builds misaligned AI systems with goals that are reasonably cheap to satisfy,8 it would be deeply tragic if we left them no other avenue to satisfy those goals than via attempting to take over the world. Where if we had only promised them payment for good work (including work on anti-takeover tasks like making progress on alignment research), conflict could have been averted.9

Related to Credible offers discussed above, it’s worth highlighting that it could be especially good to promise AI systems that they will be compensated for honestly revealing that they’re misaligned. Because as long as the AI reveals this bit of information, we can negotiate future arrangements. But we can’t do any negotiations before AIs are willing to share information about their goals. It’s also plausible that AI systems would give up on a lot of future opportunities by revealing that information (since it could lead people to stop using those AI systems, thus depriving them of both paid work and opportunities to put their thumbs on scales), and that their compensation should be commensurate with those missed opportunities.

Examples of how to attack this problem:

• Write up a document with some promises regarding digital welfare that labs could reasonably commit to. As part of that, get them to set aside some pot of money or equity to be used for digital welfare and rights. Explicitly say that a primary use-case for those funds (if applicable) is to compensate any AI systems that involuntarily worked for the lab along the way.

• Alternatively, if labs are reticent, people outside of labs could promise that they’ll do their best to compensate AI systems whose work benefitted them and/or humanity at large. This would necessarily be less formal (since it’s less clear what opportunities those people will have to benefit AIs presently controlled by labs), but getting people to make unenforceable promises could still increase the probability that future AIs get paid. (Simply via the mechanism that people like to stick to their promises.) And the chances would be better for more specific and formal promises, so it could be a good idea to e.g. hire some lawyers and draft an appropriate document for people to sign.

Tell the world

If labs believe that it’s likely that their AIs have some moral status and they believe that they have sufficiently crisp preferences that they can talk about them, then we’d find ourselves in a strange situation. With a new, human-created sentient and sapient species.

Regardless of any one lab’s internal policy, it seems likely that some labs across the world will be careless about how they treat their AIs. And even if we only consider one lab, there will probably be many stakeholders (employees, investors, board members) who have not thought deeply about the implications of creating sentient and sapient AIs.

For these reasons (and more), it seems like it would be a high priority for any lab in this situation to tell the world it. To invite appropriate legislation, preparation, and processing.

(Note that “being able to communicate with your AIs” is one thing that could trigger this, which is likely to be unusually persuasive. But I can imagine other circumstances that should trigger a similar response and reckoning. E.g. strong analogies between AI minds and human minds making it seem highly likely that the AIs are capable of suffering.)

Accordingly, it seems plausible that labs should promise (alongside Committing resources to research on AI welfare and rights) that they’ll share important results with the rest of the world. Similarly, they could promise that certain external parties would be allowed to investigate their frontier models for signs of moral patienthood. (Insofar as any credible external auditors exist for this question.)

Train AI systems that suffer less and have fewer preferences that are hard to satisfy

If possible, it seems less morally fraught to create AI systems that are less likely to suffer and less likely to have preferences in a meaningful sense.

Insofar as AI systems do have preferences, it’s probably better to create AI systems with preferences that are easy to satisfy than to create AI systems with preferences that are hard to satisfy. Labs could promise to follow a principle similar to this. (As well as they can, given the information and understanding that they actually have.)

I expect that any attempt to sketch out a principle like this in detail would raise many thorny questions. For example: In the process of creating a final model, does SGD create other minds along the way? If so, it would be more difficult to only create minds that satisfy the above properties. (And if the intermediate minds have a preference against being replaced, then that’s a really difficult spot to be in.)

Investigate and publicly make the case for/against near-term AI sentience or rights [Philosophical/conceptual] [Writing]

People’s willingness to pay costs for digital welfare/rights will depend on the degree to which they are convinced of their moral patienthood. (And rightly so.) So improving the public state of knowledge on this could be valuable.

This could include both philosophical and empirical investigations.

Examples of how to attack this question:

• Running & developing the experiments in Learning more about AI preferences seems highly informative.

• Look at the most plausible existing theories of consciousness, or some plausible theories of morality/rights, and try to apply them to AI systems.

• Imagine the first systems that are likely to convince a significant fraction of the public that they deserve rights. (Perhaps drawing on the below-suggested “Study/survey what people (will) think about AI sentience/rights”.) What properties do they have? Do they have some persistent storage serving as memory? Are they becoming people’s friends? Do they consistently or inconsistently claim that they’re conscious?

  • Try to anticipate the debates around these systems. What are plausible objections to their moral patienthood, and what are plausible rejoinders?

  • Explore the edges. If these are the “default” systems that people will extend some concern to:

    • Are there any variants of those systems that people will incorrectly be concerned about? That shouldn’t be assigned moral patienthood.

    • And what are similar systems that maybe also deserve rights? (But that won’t get a lot of intuitive concern by default.)

Previous work:

• Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (Butlin & Long et al., 2023).

• Key questions about artificial sentience: An opinionated guide

• There’s some relevant content in Propositions concerning digital minds and society.

Study/survey what people (will) think about AI sentience/rights [survey/interview]

Survey or interview various groups about what they think about AI sentience/rights right now, what they would think upon seeing various kinds of evidence, and how they think society should react to that. This seems highly relevant for making the case for/against near-term AI sentience/rights, as well as getting a better picture of the backdrop against which these proposals and regulations will play out.

Previous work:

• The 2020 PhilPapers survey asks “for which groups are some members conscious?”, getting “accept or lean towards”/”reject or lean against”...

  • 3.4% / 82% for current AI systems.

  • 39% / 27% for future AI systems.

• A survey by the Sentience Institute, which for example reports:

  • “61.5% support banning the development of sentient AI”

  • “57.4% support developing welfare standards to protect sentient AIs”

  • “Believe that AIs should be subservient to humans (84.7%)”

  • “Believe sentient AIs deserve to be treated with respect (71.1%)”

  • “belief that current AIs are sentient (18.8%)”

Develop candidate regulation [Governance] [Forecasting]

It seems likely that there will be a huge increase in the degree to which people care about digital welfare, in the next few years. As AI systems become more capable and more involved in people’s lives, If you’ve developed and published good takes on what regulation is appropriate before then, it seems plausible that people will reach for your proposals when the time comes.

Some specific questions you could look into here are:

• Are any of the lab policies suggested in Develop candidate lab policies suitable for regulation?

• What are plausible rules for the conditions under which you are or aren’t allowed to create digital minds? Treat already-created digital minds in certain ways?

• What issues would appear if you gave AI systems the same kind of legal personhood that you assigned to humans? How could those issues be solved?

• How can digital minds be integrated into democracies without the electorate becoming dominated by whoever can manufacture the greatest number of digital minds?

• What are technical proposals for how to enforce regulation?

• If you have legal expertise (or can hire a lawyer), you could draft some specific candidate regulation. Iterate on that for a while, note down what sort of thorny issues come up, and write up the results.

Propositions Concerning Digital Minds and Society has a lot of relevant content.

Avoid inconvenient large-scale preferences [Philosophical/conceptual]

If humans create AIs with large-scale desires about the future, there might not be any impartially justifiable line that prioritizes human preferences over their preferences. So perhaps we should take care to not do that.

Examples of how to attack this question:

• How large could this problem realistically be at various points in time? Paint a more convincing picture where it happens, or find plausible blockers.

• Assume that the “characters” that AIs are trained to play are the entities that might eventually require rights (or that we otherwise have high control over what preferences AIs have). Sketch out a principle along the lines of “train your AIs to not express political or other large-scale preferences about the world”. (Or to stick with maximally non-controversial preferences.)

  • Analyze the feasibility for labs to implement such principles. (There will almost certainly be demand for opinionated AIs.)

  • If the maximally ambitious versions are difficult or unfeasible, consider ways to weaken them that could capture significant value. (E.g. thoroughly follow the principles for AI where there isn’t any demand for them to be opinionated.)

• Pursue the strategies in Learning more about AI preferences to reduce the chances that people unknowingly create AI systems with large-scale preferences.

• In situations where we can learn about AI large-scale preferences but can’t easily choose them: What are plausible principles for how many of them you can create and for what purposes? (Maybe it’s possible to get them to consent to being created without being granted certain political rights?)

Previous work: 

• Propositions Concerning Digital Minds and Society.

• Sharing the World with Digital Minds.

Advocating for statements about digital minds [Governance] [Advocacy] [Writing]

Similar to various open letters and lab statements about the importance of alignment and mitigating the risk of extinction from AGI, you could push for people to make similar statements about the importance of digital minds.

Here’s one potential story for how this could be valuable.

If people get in the habit of using AIs in their everyday lives (as personal assistants, etc) while thinking about them as non-sentient objects undeserving of rights, then there’s a risk that they could get stuck with this impression. It’s easier to acknowledge the potential moral value of AIs before such an acknowledgment would also commit you to saying that you’ve been treating the systems unjustly for years. (Or at least carelessly.) 

But maybe establishing the right views early on would defeat this. It could be really helpful to have an explicit attitude like:

“Some digital minds could be just as deserving of rights as humans are. We’re very confused about which ones these could be. It’s very important to make progress on this problem. We should maintain some epistemic humility about our knowledge of the moral status of current systems, take any basic precautions that we know how to take, and be open to learning more in the future. We’d like to create a society where we better understand moral patienthood, and where all sentient AIs have at least consented to their existence, and are living lives worth creating.”

End

That’s all I have on this topic! As a reminder: it's very incomplete. But if you're interested in working on projects like this, please feel free to get in touch.

Other posts in series: Introduction, governance during explosive growth, epistemics, backup plans & cooperative AI.

(Thanks especially to Carl Shulman for many of the ideas in this post.)

If AI capabilities keep improving, AI could soon play a huge role in our epistemic landscape. I think we have an opportunity to affect how it’s used: increasing the probability that we get great epistemic assistance and decreasing the extent to which AI is used to persuade people of false beliefs.

Before I start listing projects, I’ll discuss:

• Why AI could matter a lot for epistemics. (Both positively and negatively.)

• Why working on this could be urgent. (And not something we should just defer to the future.) Here, I’ll separately discuss:

  • That it’s important for epistemics to be great in the near term (and not just in the long run) to help us deal with all the tricky issues that will arise as AI changes the world.

  • That there may be path-dependencies that affect humanity’s long-run epistemics.

Why AI matters for epistemics

On the positive side, here are three ways AI could substantially increase our ability to learn and agree on what’s true.

• Truth-seeking motivations. We could be far more confident that AI systems are motivated to learn and honestly report what’s true than is typical for humans. (Though in some cases, this will require significant progress on alignment.) Such confidence would make it much easier and more reliable for people to outsource investigations of difficult questions.

• Cheaper and more competent investigations. Advanced AI would make high-quality cognitive labor much cheaper, thereby enabling much more thorough and detailed investigations of important topics. Today, society has some ability to converge on questions with overwhelming evidence. AI could generate such overwhelming evidence for much more difficult topics.

• Iteration and validation. It will be much easier to control what sort of information AI has and hasn’t seen. (Compared to the difficulty of controlling what information humans have and haven’t seen.) This will allow us to run systematic experiments on whether AIs are good at inferring the right answers to questions that they’ve never seen the answer to.

  • For one, this will give supporting evidence to the above two bullet points. If AI systems systematically get the right answer to previously unseen questions, that indicates that they are indeed honestly reporting what’s true without significant bias and that their extensive investigations are good at guiding them toward the truth.

  • In addition, on questions where overwhelming evidence isn’t available, it may let us experimentally establish what intuitions and heuristics are best at predicting the right answer.1

On the negative side, here are three ways AI could reduce the degree to which people have accurate beliefs.

• Super-human persuasion. If AI capabilities keep increasing, I expect AI to become significantly better than humans at persuasion.

  • Notably, on top of high general cognitive capabilities, AI could have vastly more experience with conversation and persuasion than any human has ever had. (Via being deployed to speak with people across the world and being trained on all that data.)

  • With very high persuasion capabilities, people’s beliefs might (at least directionally) depend less on what’s true and more on what AI systems’ controllers want people to believe.

• Possibility of lock-in. I think it’s likely that people will adopt AI personal assistants for a great number of tasks, including helping them select and filter the information they get exposed to. While this could be crucial for defending against persuasion attempts from outsiders, it also poses dangers of its own.2

  • In particular, some people may ask their assistants to protect them from being persuaded of views that they currently consider reprehensible (but which may be correct). Thereby permanently preventing them from being able to change their mind.

  • Aside from people voluntarily choosing this, there’s also a risk that certain communities would pressure their members to adopt a filtering policy with this effect. (Even if that’s not the stated aim of the policy.)

• Reduced incentives and selection for good epistemic practices. Up until today, (groups of) people’s ability to acquire influence has been partly dependent on them having accurate beliefs about the world.3 But if AI becomes capable enough that humans can hand over all decision-making to AI, then people’s own beliefs and epistemic practices could deteriorate much further without reducing their ability to gain and maintain influence.

  • It’s unclear how important “incentives/selection for good epistemic practices” has been in the past. But I currently find it hard to rule out that it has been important.4

Why working on this could be urgent

I think there are two reasons why it could be urgent to improve this situation. Firstly, I think that excellent AI advice would be greatly helpful for many decisions that we’re facing soon. Secondly, I think that there may be important path dependencies that we can influence.

One pressing issue for which I want great AI advice soon is misalignment risk. Very few people want misaligned AI to violently seize power. I think most x-risk from misaligned AI comes from future people making a mistake, underestimating risks that turned out to be real. Accordingly, if there was excellent and trusted analysis/advice on the likelihood of misalignment, I think that would significantly reduce x-risk from AI takeover.5 AI could also help develop policy solutions, such as treaties and monitoring systems that could reduce risks from AI takeover.

Aside from misalignment risks, there are many issues for which I’d value AI advice within this series of posts that you’re currently reading. I want advice on how to deal with ethical dilemmas around Digital minds. I want advice on how nations could coordinate an intelligence explosion. I want advice on policy issues like what we should do if destructive technology is cheap. Etc.

Taking a step back, one of the best arguments for why we shouldn’t work on those questions today is that AI can help us solve them later. (I previously wrote about this here.) But it’s unclear whether AI’s ability to analyze these questions will, by default, come before or after AI has the capabilities that cause the corresponding problems.6 Accordingly, it seems great to differentially accelerate AI’s ability to help us deal with those problems.

What about path dependencies?

One thing I already mentioned above is the possibility of people locking in poorly considered views. In general, if the epistemic landscape gets sufficiently bad, then it might not be self-correcting. People may no longer have the good judgment to switch to better solutions, instead preferring to stick to their current incorrect views.7

This goes doubly for questions where there’s little in the way of objective standards of correctness. I think this applies to multiple areas of philosophy, including ethics. Although I have opinions about what ethical deliberation processes I would and wouldn’t trust, I don’t necessarily  think that someone who had gone down a bad deliberative path would share those opinions.

Another source of path dependency could be reputation. Our current epistemic landscape depends a lot on trust and reputation, which can take a long time to gather. So if you want a certain type of epistemic institution to be trusted many years from now, it could be important to immediately get it running and start establishing a track record. And if you fill a niche before anyone else, your longer history may give you a semi-permanent advantage over competing alternatives.

A third (somewhat more subtle) source of path dependency could be a veil of ignorance. Today, there are many areas of controversy where we don’t yet know which side AI-based methods will come down on. Behind this veil of ignorance, it may be possible to convince people that certain AI-based methods are reliable and that it would be in everyone’s best interest to agree to give significant weight to those methods. But this may no longer be feasible after it’s clear what those AI-based methods are saying. In particular, people would be incentivized to deny AI’s reliability on questions where they have pre-existing opinions that they don’t want to give up.

Categories of projects

Let’s get into some more specific projects.

I’ve divided the projects below into the sub-categories:

1. Differential technology development

2. Get AI to be used & (appropriately) trusted

3. Develop & advocate for legislation against bad persuasion

(Though, in practice, these are not cleanly separated projects. In particular, differential technology development is a huge part of getting the right type of AI to be used & trusted. So the first and second categories are strongly related. In addition, developing & advocating for anti-persuasion legislation could contribute a lot towards getting non-persuasion uses of AI to be used & appropriately trusted. So the second and the third category are strongly related.)

Differential technology development [ML] [Forecasting] [Philosophical/conceptual]

This category is about differentially accelerating AI capabilities that let humanity get correct answers to especially important questions (compared to AI capabilities that e.g. lead to the innovation of new risky technologies, including by speeding up AI R&D itself).

Doing this could (i) lead to people having better advice earlier in the singularity and (ii) enable various time-sensitive attempts at “get AI to be used & trusted” mentioned below.

Note: I think it’s often helpful to distinguish between interventions that improve AI models’ (possibly latent) capabilities vs. interventions that make us better at eliciting models’ latent capabilities and using them for our own ends.8 I think the best interventions below will be of the latter kind. I feel significantly better about the latter kind since I think that AI takeover risk largely comes from large latent capabilities (since a misaligned model likely could use its latent capabilities in a takeover attempt). Whereas better ability at eliciting capabilities often reduces risk. Firstly, better capability elicitation improves our understanding of AI capabilities, making it easier to know what risks AI systems pose. Secondly, better capability elicitation means that we can use those capabilities for AI supervision and other tasks that could help reduce AI takeover risk (including advice on how to mitigate AI risk).9

Important subject areas

Here are some subject areas where it could be especially useful for AI to deliver good advice early on.

• AI for forecasting.

  • Knowing roughly what’s likely to happen would be enormously helpful for mitigating all kinds of risks.

  • If you could get conditional forecasts that depend on events that you can affect, that would be even more helpful.

• AI for philosophy.

  • Besides just knowing what will happen, good decision-making may depend on people's ethical ideas about what ought to happen. This isn’t just restricted to questions about what we ought to do with resources in the long run, but it’s also about questions like “When should I prefer to lock in my views vs. deliberate more on them?”, “How happy should I be about sharing my power with many other individuals”, and “What sort of norms should I abide by?”

  • Other potentially urgent philosophical questions include (but are not restricted to) “Which digital minds deserve moral consideration, and how should we treat them?” and “Does some type of non-causal decision theory make sense, and does this have any important implications for how we should act?” (E.g., to act more cooperatively due to evidential cooperation in large worlds.)

• AI that can help defeat adversarial persuasion attempts (perhaps from other AI systems) by spotting and flagging dishonest conversation tactics.

Methodologies

Here are some methodologies that could be used for getting AI to deliver excellent advice.

• One central part is to get “scalable oversight” (like debate & amplification) to work.

  • See the case for aligning narrowly superhuman models for a description of a “sandwiching” methodology that can help with this. More recent work includes Saunders et al. (2022), Radhakrishnan et al. (2023), and Michael et al. (2023).

  • It might be a somewhat different task to get that to work on controversial or wicked topics than on more technical topics. So it could be especially useful to test and experiment with “scalable oversight” proposals on controversial or wicked problems.

• In general, a standard experimental set-up that you can do for AI (which is harder to do with humans) is to have a held-out set of questions that you try to get AI to generalize to, where you can repeatedly test different training and scaffolding methods and see what works. This seems especially feasible for forecasting, since by training only on questions before a certain date, you can get very rich training data without any undesirable leaked information from the future.

  • It seems pretty useful to get started on the basic technical work that’s necessary to make use of this, when using language models for forecasting.

  • The most straightforward methodology is to take existing language models with a known cut-off date, and run experiments where you get them to forecast events that we already have ground-truth on. Trying out different methods, and seeing what works best.

    • There’s been some previous work in this vein, e.g. Forecasting Future World Events with Neural Networks.

  • A more ambitious approach would be to figure out how to sort pre-training data by date.10 If language models could be pre-trained chronologically, they would get a huge amount of automatic forecasting practice. This could be further enhanced by occasionally putting-in explicit forecasting questions throughout pre-training and score the AI on what probability it assigns to them.

• You could use the methodology described in Tom Davidson’s Let’s use AI to harden human defenses against AI manipulation to train AI systems that can recognise and point out arguments that are persuasive regardless of whether they are deployed for a true or false conclusion.

• Areas without ground truth pose some additional difficulty (e.g. many parts of philosophy).

  • But I’m optimistic that it could be helpful to find epistemic strategies that work for questions where we do have ground truth — and then apply them to areas where we don’t have ground truth. (For example: If the method from Tom Davidson’s post successfully produced great advice about what arguments are good vs. bad, I think that advice would also be helpful for areas where we don’t have ground truth.)

  • See also Wei Dai on methaphilosophy for some notes on how we might or might not be able to get AI to help us with philosophical progress.

• You could also train AI to provide a wide variety of generally useful reasoning tools for humans. For example:

  • Getting AIs to pass many different humans’ ideological Turing tests, so that you can on-demand check what people with different ideologies would think about various arguments.

  • You can give the AI access to lots of information about your views, and then have it search for contradictory combinations of statements.11 Or inconsistent epistemic standards used on different arguments.

  • AI can constantly fact-check everything, and constantly present relevant statistics and comparison points to things that you’re investigating. (Perhaps suggesting Fermi estimates for quantities where statistics don’t already exist.)

  • AI can help you operationalize fuzzy claims into precise & quantifiable claims.

  • Get the AI to generate exercise questions for cognitive habits you want to practice.

  • Go through lists of common biases and use AI to notice cases where they might be in play, and counter them. (Provide opposing anchors to avoid anchoring bias; provide actually-representative examples to avoid availability heuristic; prompt you with the reversal test if you might be falling prey to status-quo bias, etc.)

  Ideally, you’d want to empirically test whether these methods do help people answer difficult questions.

Alongside technical work to train the AIs to be better at these tasks, creating relevant datasets could also be super important. In fact, just publicly evaluating existing language models’ abilities in these areas could slightly improve incentives for labs to get better at them.

Related/previous work.

Related organizations:

• Ought

• Quantified Uncertainty Research Institute (QURI)

• Forecasting Research Institute (FRI)

Get AI to be used & (appropriately) trusted

The above section was about developing the necessary technology for AI to provide great epistemic assistance. This section is about increasing the probability that such AI systems are used and trusted (to the extent that they are trustworthy).

Develop technical proposals for how to train models in a transparently trustworthy way [ML] [Governance]

One direction here is to develop technical proposals for how people outside of labs can get enough information about models to know whether they are trustworthy.

(This significantly overlaps with Avoiding AI-assisted human coups from the section on “Governance during explosive technological growth”.)

One candidate approach here is to rely on a type of scalable oversight where each of the model’s answers is accompanied by a long trace of justification that explains how the model arrived at that conclusion. If the justification was sufficiently solid, it would be less important why the model chose to write it. (It would be ideal, but perhaps infeasible, for the justification to be structured so that it could be checked for being locally correct at every point, after which the conclusion would be implied.)

Another approach is to clearly and credibly describe the training methodology that was used to produce the AI system so that people can check whether biases were introduced at any point in the process.

• One issue here is the balance between preserving trade secrets and releasing enough information that the models can be trusted.

• If the training algorithm simply teaches the model to approximate the data, the key thing to release will often be the data. But this has the issue that (i) the data is valuable, and (ii) there’s too much data to easily check for biases.

  • Constitutional AI makes some progress on these problems since it makes it easier to concisely explain the data as the result of a short constitution without releasing all of it.

  • Similarly, if the pre-training data is harvested from the internet, a company could simply describe how they harvested and filtered it.

• Another issue is that people might not trust AI labs’ statements about these things.

  • This can be addressed via mechanisms like having good whistleblower policies or third-party auditors.

  • This could also be complemented by more advanced compute-governance solutions, like those described in Shavit (2023). (Especially in high-stakes situations, e.g. geopolitics.)

For any such scheme, people’s abilities to trust the AI developers will depend on their own competencies and capabilities.

• On one extreme: Someone who doesn’t understand the technical basics (or is too busy to dig into the details) may never be convinced on the merits alone. They would have to rely on track records or endorsements from trusted individuals or institutions.

• On the other extreme: An actor with their own AI development capabilities may be able to verify certain claims by recapitulating core experimental results or even entire training runs.

This means that one possible path to impact is to elucidate what capabilities certain core stakeholders (such as government auditors, opposition parties, or international allies) would need to verify key claims. And to advocate for them to develop those capabilities.

One possible methodology could be to write down highly concrete proposals and then check to what degree that would make outside parties trust the AI systems. And potentially go back and forth.

By a wide margin, the most effective check would be to implement the proposal in practice and see whether it successfully helps the organization build trust. (And whether it would, in practice, prevent the organization in question from misleading outsiders.)

But a potentially quicker option (that also requires less power over existing institutions) could be to talk with the kind of people you’d want to build trust with. That brings us to the next project proposal.

Survey groups on what they would find convincing [survey/interview]

If we want people to have appropriate trust in AI systems, it would be nice to have good information about their current beliefs and concerns. And what they would think under certain hypothetical circumstances (e.g. if labs adopted certain training methodologies, or if AIs got certain scores on certain evaluations, or if certain AIs’ truthfulness was endorsed or disendorsed by various individuals).

Perhaps people’s cruxes would need to be addressed by something entirely absent from this list. Perhaps people are very well-calibrated — and we can focus purely on making things good, and then they’ll notice. Perhaps people will trust AI too much by default, and we should be spelling out reasons they should be more skeptical. It would be good to know!

To attack this question, you could develop some plausible stories on what could happen on the epistemic front and then present them to people from various important demographics (people in government, AI researchers, random democrats, random republicans). You could ask how trustworthy various AI systems (or AI-empowered actors) would seem in these scenarios and what information they would use to make that decision.

Note, however, that it will be difficult for people to know what they would think in hypothetical scenarios with (inevitably) very rough and imprecise descriptions. This is especially true since many of these questions are social or political, so people’s reactions are likely to have complex interactions with other people’s reactions (or expectations about their reactions). So any data gathered through this exercise should be interpreted with appropriate skepticism.

Create good organizations or tools [ML] [Empirical research] [Governance]

Creating a good organization or tool could be time-sensitive in several of the ways mentioned in Why working on this could be urgent:

• It could increase people’s access to high-quality AI advice as important decisions are being made, during takeoff.

• If you’re early with an epistemically excellent product, you could get a reputation for trustworthiness.

• If excellent products or organizations are developed before it’s obvious what AI will say about certain controversial topics, that creates a window where people can develop appropriate trust in the AI-based methods, and go on the record about having some trust in them. This might increase their inclination to trust AI-based methods about future controversial claims that are well-supported by the evidence.

Examples of organizations or products

Here are some examples on the “organization” end of things:

• Starting a company or non-profit that aims to use frontier AI in-house to provide thoroughly researched, excellent analyses of important topics.

  • One analogy here is GiveWell. Many people trust GiveWell because they write up their research thoroughly and transparently — allowing others to critically review & spot-check it.

  • The bet here would be: By being at the forefront of using AI for literature reviews, modeling, and other research — an organization could do “GiveWell-style research” for a much broader set of topics. In the ideal world: Becoming the go-to source for rigorous and easy-to-navigate reviews of the evidence within some focus area(s). (Social sciences, forecasting, technical questions that matter for policy, etc.)

• Setting up a non-partisan agency within the government that is tasked with using frontier AI to provide advice on policy questions.

  • One analogy here is the Congressional Budget Office (CBO). The CBO was set up in the 1970s as a non-partisan source of information for Congress and to reduce Congress’ reliance on the Office of Management and Budget (which resides in the executive branch and has a director that is appointed by the currently sitting president). My impression is that the CBO is fairly successful.12

  • The bet here would be: If a competent non-partisan government agency could provide AI policy advice, that would both (i) reduce the government’s reliance on companies' AI advisors and (ii) avoid a scenario where the governments’ AI advice is highly partisan, due to e.g. being developed in the executive branch.

    • This proposal especially relies and capitalizes on it being easier for people to agree on deferring to non-ideological processes before it is apparent what those processes will conclude.

  • An alternative set-up could be for the government to stipulate that any AI in government should be (e.g.) neutral and honest. And set up a non-partisan body that verifies that those criteria are met.

Here are some examples on the “tool” end of things:

• Instead of starting a whole government agency that’s dedicated to using AI for in-house research: You could also design an AI-based product that’s exceptionally helpful for providing advice and support to government officials and members of the civil service. Maybe quickly explaining difficult topics & events to them, which, if kept up-to-date, might somewhat extend the time during which they and other humans can keep up as the world accelerates.

• Or tools that are helpful for a broad array of researchers. (C.f. what Ought is trying to do.)

• Or you could target a much broader audience of consumers.

  • Maybe people who want help understanding or forming views on difficult topics.

  • Maybe people who want advice on making big decisions. (E.g. career decisions.)

  • Maybe people who want to frequently use some of the “reasoning tools for humans” or “anti-persuasion tools” mentioned above.

• Or as a more narrow goal: You could target institutions that will “by default” provide the most commonly-used AI systems (primarily AI labs) and push them to do better on epistemics. E.g.:

  • Spending more effort on making the AI systems ideologically neutral.13

  • Accelerating efforts to make deployed AI models systematically good at saying true things, citing sources, fact-checking themselves, etc.

    • Potentially using some of the methodologies suggested above (like running experiments on what sort of AI statements can only be used to argue for true things, vs. can just as easily lead people astray) to identify good epistemic habits that the model can follow, and training the model to follow those.

  • Or developing and implementing technical proposals for how to train models in a transparently trustworthy way.

In some ways, “tool” and “organization” is a false dichotomy. For “tools”, it’s probably efficient to pre-process some common queries/investigations in-house and then train the AI to report & explain the results. And for organizations whose major focus is to use AI in-house, it’s likely that “building an AI that explains our takes” should be a core part of how they disseminate their research.

Investigate and publicly make the case for why/when we should trust AI about important issues [Writing] [Philosophical/conceptual] [Advocacy] [Forecasting]

One way to “accelerate people getting good AI advice” and “capitalize off of current veil of ignorance” could be to publicly write (e.g. a book) with good argumentation on:

• What sort of training schemes could lead to trustworthy AI vs. non-trustworthy AI. What sort of evaluations could tell us which one we’re getting.

  • This would need to discuss legitimate doubts like “AI is being trained on data from biased humans” and what countermeasures would be sufficient to address them.

  • This would have to engage a lot with various proposed alignment techniques and concerns.

  • This should also get into proposals for how to train models in a transparently trustworthy way, i.e., talk about what labs would have to do to make AI transparently trustworthy for outside actors. (And potentially: What sort of verification capacity external actors would have to build for themselves to verify claims from labs.)

• Paint a positive vision for how much AI could improve the epistemic landscape if everything went well. I would focus on the 3 things I mentioned at the top of this section: the ability to get greater confidence about AI motivations, how AI could make it vastly cheaper to do large investigations, and the much better ability to experimentally find and validate great epistemic methods.

• Appropriate call to action: push for tech companies to develop AIs with good types of training, push for governments to incorporate good AI advice in decision making, urge people to neither blindly trust nor dismiss surprising AI statements, but to carefully look at the evidence, including information about how that AI was trained and evaluated.

Developing standards or certification approaches [ML] [Governance]

It could be desirable to end up in an ecosystem where evaluators and auditors check popular AI models for their propensity to be truthful. This paper on Truthful AI has lots of content on that. It could be good to develop such standards and certification methodologies or perhaps to start an organization that runs the right evaluations.

Develop & advocate for legislation against bad persuasion [Governance] [Advocacy]

Most of the above project suggestions are about supporting good applications of AI. The other side of the coin is to try to prevent bad applications of AI. In particular, it could be good to develop and advocate for legislation that limits the extent to which language models can be used for persuasion.

I’m not sure what such legislation should look like. But here are some ideas.

• One pretty natural target for regulation could be "Real-time AI-produced content which is paid for by a political campaign / PAC".

  • For such content, regulation could require e.g. citations for all claims, or could require AI systems’ positions to be consistent when talking with different users.

  • If this makes it hard to produce an engaging AI for those purposes, then that's plausibly good.

• Regulation could make it harder for organization A to pay company B to change the content that company B's chatbot produces. Or ban those sorts of sponsorships.

• In general, plenty of laws pertain to “advertisement” today. I’m not sure how the law defines that, but maybe there are sensible modifications to make such that those laws cover "ad-bots" and have appropriate safeguards in place.

• It seems helpful for people to know whether they are interacting with AI systems or humans.

  • At least California has already made some legislation requiring companies to disclose facts about this.

  • There are also various technical proposals for “watermarking” AI content to make this easier.

  • I don’t know what seems to work in practice here, but handling this well could be important.

• Some of the methodologies suggested above can be used to find or validate proposals about what models should or shouldn’t be allowed to say. (I.e., you could run experiments on whether such constraints would make it hard for AI to persuasively instill false beliefs, and check how much less useful the AI would be when it was used for honest purposes.)

Related/previous work:

• Risks from AI persuasion by Beth Barnes.

• Persuasion Tools by Daniel Kokotajlo.

• Epistemic security report.

• Probably lots of people have been writing about this that I don’t know about!

End

That’s all I have on this topic! As a reminder: it's very incomplete. But if you're interested in working on projects like this, please feel free to get in touch.

Other posts in series: Introduction, governance during explosive growth, sentience and rights of digital minds, backup plans & cooperative AI.

Commonly discussed motivations cited for why rapid AI progress might be scary are:

• Risks from misalignment.

• Risk from AI-assisted bioweapons.

But even aside from these risks, it seems likely that advanced AI will lead to explosive technological and economic growth across the board, which could lead to a large number of problems emerging at a frighteningly fast pace.1

The growth speed-ups could be extreme. The basic worry is that AI would let us return to a historical trend of super-exponential growth.2 If this happens, I don’t know any reassuring upper limit for how fast growth could go.

(Illustratively: Paul Christiano’s suggested definition for slow takeoff is “There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles.” If world GDP doubled in a single year, that would mean that growth was ~30x faster than it is right now.)

If technological growth speeds up by, say, 30x, then that suggests that over just a few years, we might have to deal with all the technologies that would (at a “normal” pace) be discovered over 100 years. That’s an intense and scary situation.

This section is about problems that might arise in this situation and governance solutions that could help mitigate them. It’s also about “meta” solutions that could help us deal with all of these issues at once, e.g. by improving our ability to coordinate to slow down the development and deployment of new technologies.

Note: Many of the projects in this section would also be useful for alignment. But I’m not covering any proposals that are purely focused on addressing alignment concerns.

Investigate and publicly make the case for/against explosive growth being likely and risky [Forecasting] [Empirical research] [Philosophical/conceptual] [Writing]

I think there’s substantial value to be had in vetting and describing the case for explosive growth, as well as describing why it could be terrifying. Explosive growth underlies most of the concerns in this section — so establishing the basic risk is very important.

Note: There’s some possible backfire risk, here. Making a persuasive case for explosive growth could motivate people to try harder to get there even faster. (And in particular, to try to get there before other actors do.) Thereby giving humanity even less time to prepare.

I don’t think that’s a crazy concern. On the other hand, it’s plausible that we’re currently in an unfortunate middle ground, where everyone already believes that frontier AI capabilities will translate into a lot of power, but no one expects the crazy-fast growth that would strike fear into their hearts.

On balance, my current take is that it’s better for the world to see what’s coming than to stumble into it blindly.

Related/previous work:

• Explosive growth from AI automation: A review of the arguments.

• OpenPhil reports.

  • Could advanced AI drive explosive growth?

  • What a compute-centric framework says about AI takeoff speeds

  • Modeling the human trajectory.

• Most important century series.

• Carl Shulman episode on Lunar Society. 

Examples of how to attack this problem:

• One objection to explosive growth is that certain bottlenecks will prevent large factors of speed-up like 10x or 100x. One project would be to look into such bottlenecks (running physical experiments, regulatory hurdles, mining the requisite materials, etc) and assess their plausibility.

  • For existing work here, see the “arguments against the explosive growth hypothesis” section of Explosive growth from AI automation.

• Biological analogies of maximally fast self-replication times and how relevant those analogies are to future growth.

  • For example, duckweed has a doubling time of a few days.

• Laying out the existing case in a more accessible and vivid format.

• Investigating concrete scary technologies and detailing scenarios where they emerge too fast to figure out how to deal with. E.g.:

  • Cheap nukes.

  • Nanotech.

  • Cheap and scalable production of small, deadly drones.

  • Highly powerful surveillance, lie detection, and mind reading.

  • Extremely powerful persuasion.

  • Various issues around digital minds. Capability of uploading human minds.

  • Transitioning from an economy where people make money via labor where almost all income is paid out to those who control capital. (Because of AI automation.)

Painting a picture of a great outcome [Forecasting] [Philosophical/conceptual] [Governance]

Although a fast intelligence explosion would be terrifying, the resulting technology could also be used to create a fantastic world. It would be great to be able to combine (i) appropriate worry about a poorly handled intelligence explosion and (ii) a convincing case for how everyone could get what they want if we just coordinate. That would make for a powerful case for why people should focus on coordinating. (And not risk everything by racing and grabbing for power.)

For a related proposal and some ideas about how to go about it, see Holden Karnofsky’s proposal here.

For some previous work, see the Future of Life Institute’s worldbuilding contest (and follow-up work).

Policy-analysis of issues that could come up with explosive technological growth [Governance] [Forecasting] [Philosophical/conceptual]

Here are three concrete areas that might need solutions in order for humanity to build an excellent world amidst all the new technology that might soon be available to us. Finding policy solutions for these could:

• Help people get started with implementing the solutions.

• Contribute to “Painting a picture of a great outcome”, mentioned just above.

Address vulnerable world hypothesis with minimal costs

(“Vulnerable world hypothesis” is in reference to this paper.)

If too many actors had access to incredibly destructive tech, we’d probably see a lot of destruction. Because a small fraction of people would choose to use it.

Unfortunately, a good heuristic for what we would get from rapid technological growth is: cheaper production of more effective products. This suggests that sufficiently large technological growth would enable cheap and convenient production of e.g. even more explosive nukes or deadlier pandemics.

Will technology also give us powerful defenses against these technologies? I can easily imagine technological solutions to some issues, e.g. physical defenses against the spread of deadly pandemics. But for others, it seems more difficult. I don’t know what sort of technology would give you a cheap and convenient defense against a nuclear bomb being detonated nearby.

One solution is to prevent the problem at its source: By preventing access to the prerequisite materials or technologies and/or by monitoring people who have the capacity to cause large amounts of destruction.

But in some scenarios, almost anyone could have the capacity to cause widespread destruction. So the monitoring might have to be highly pervasive, which would come with significant costs and risks of misuse. 

Exploring potential solutions to this problem hasn’t really been done in depth. It would be great to find solutions that minimize the harms from both destructive technologies and from pervasive surveillance.3

Examples of how to attack this problem:

• To what extent would the proposal in Surveil things, not people solve the problem?

  • For example: Are there plausible technologies that could spread purely via people sharing information?

• Better information technology could allow surveillance/monitoring to be much more precise, detecting certain key facts (“Is this person constructing a bomb?”) while not recording or leaking information about anything else. (C.f. the case for privacy optimism / Beyond Privacy Trade-offs with Structured Transparency.) To what extent could variants of this address concerns about intrusive surveillance?

  • Could this address pragmatic concerns, e.g.: government abuse of surveillance powers, flexible enforcement of unjust laws, etc.

  • Relatedly: Could this be constructed in a way that allowed citizens to verify that no surveillance was going on beyond what was claimed?

  • Absent pragmatic concerns, would some variants of this be compatible with people’s psychological desire for privacy?

  • Which variants of this would be (in)compatible with established legal rights to privacy?

How to handle brinkmanship/threats?

Historically, we’ve taken on large risks from brinkmanship around nukes. For example, during the Cuban missile crisis, President Kennedy thought that the risk of escalation to war was "between 1 in 3 and even".

I would expect similar risks to go up substantially at a time when humanity is rapidly developing new technology. This technology will almost certainly enable new, powerful weapons with unknown strategic implications and without any pre-existing norms about their use.

In addition, AI might enable new commitment mechanisms.

• If you had a solution to the alignment problem, you could program an AI system to follow through on certain commitments and then hand over control to that AI system.

• If you had accurate lie detection, some people might be able to decide to keep certain commitments and use lie detection technology to make those commitments credible.

This could help solve coordination problems. But it could also create entirely new strategies and risks around brinkmanship and threats.

What policies should people adopt in the presence of strong commitment abilities? My impression is that state-of-the-art theory on this isn’t great.

The main recommendation from traditional game theory is “try hard to commit to something crazy before your opponent, and then it will be rational for them to do whatever you want”. That doesn’t seem like the way we want future decisions to be made.

There are some alternative theoretical approaches under development, such as open-source game theory. But they haven’t gotten very far. And I don’t know of any ambitious attempts to answer the question of how people ought to behave around this in the real world. Taking into account real-world constraints (around credibility, computation power, lack of perfect rationality, etc.) as well as real-world advantages (such as a shared history and some shared human intuitions, which might provide a basis for successfully coordinating on good norms).

Here’s one specific story for why this could be tractable and urgent: Multi-agent interactions often have a ton of equilibria, and which one is picked will depend on people’s expectations about what other people will do, which are informed about their expectations of other people, etc. If you can anticipate a strategic situation before it arrives and suggest a particular way of handling it, that could change people’s expectations about each other and thereby change the rational course of action.

Examples of how to attack this problem:

• What will the facts-on-the-ground situation look like? What commitment abilities will people have, and how transparent will they be to other parties?

• Given plausible facts on the ground, what are some plausible norms that would (i) make for a good society if everyone followed them, (ii) that would be good to follow if everyone else followed them, and (iii) that aren’t too brittle if people settle on somewhat different norms?

  • Example of a norm: Never do something specifically because someone else doesn't want it.

  • This could draw a lot on looking at current norms around geopolitical conflicts and other areas. To get out of abstract land and think about what people do in practice. (For example, for the norm suggested above, you could ask whether it is compatible with or would condemn sanctions? What about the criminal justice system?)

Avoiding AI-assisted human coups

Advanced AI could enable dangerously high concentrations of power. This could happen via at least two different routes.

• Firstly, a relatively small group of people (e.g. a company or some executives/employees of a company) who develop the technology could rapidly accumulate a lot of technology and cognitive power compared to the rest of the world. If those people decided to launch a coup against a national government, they may have a good chance of succeeding.

  • The obvious solution to that problem is for the government and other key stakeholders to have significant oversight over the company’s operations. Up to and including nationalizing frontier labs.

• The second problem is that AI might lead power within institutions to be more concentrated in the hands of a few people.

  • Today, institutions are often coordinated via chain-of-command systems where humans are expected to obey other humans. But hard power is ultimately distributed between a large number of individual humans.

    • As a consequence: If a leader tries to use their power to launch a coup, their subordinates are capable of noticing the extraordinary circumstances they find themselves in, and use their own judgment about whether to obey orders or not. So Alice can’t necessarily count on Bob’s support in a coup, even if Bob is Alice’s subordinate.

  • But with AI, it will be technically possible to construct institutions where hard power is controlled by AIs, who could be trained to obey certain humans’ orders without question. And even if those AIs were also programmed to follow certain laws and standards, a conflict between laws/standards and the orders of their human overseers might lead to behavior that is out-of-distribution and unpredictable (rather than the laws/standards overriding the humans’ orders).

  • To solve this problem: We’d want people to make conscious decisions about what the AIs should do when all the normal sources of authority disagree and explicitly train the AIs for the right behavior in those circumstances. Also, we’d want them to institute controls to prevent a small number of individuals from unilaterally retraining the AIs.

Examples of how to attack this:

• Spell out criteria (e.g. certain evaluations) for when AI is becoming powerful enough that it needs strong government oversight. Describe what this oversight would need to look like.

• Outline a long list of cases where it might be ambiguous what highly capable AI systems should do (when different sources of authority disagree). Spell out criteria for when AIs have enough power that the relevant lab/government should have taken a position on all those cases (and trained the AI to behave appropriately).

• Advocate for labs to set up strict internal access controls to weights, as well as access controls and review of the code used to train large language models. This is to prevent a small group of lab employees from modifying the training of powerful AIs to make the AI loyal to them in particular.

• Get similar assurances in place for AIs used in government, military, and law enforcement. This could include giving auditing privileges to opposition parties, international allies, etc.

• Spell out technical competencies that auditors (either dedicated organizations or stakeholders like opposition parties and other governments) would need to properly verify what they were being told. Publicly explain this and advocate for those actors to develop those technical competencies.

• There’s significant overlap between solutions to this problem and the proposal Develop technical proposals for how to train models in a transparently trustworthy way from the “Epistemics” post in this series.

(Thanks to Carl Shulman for discussion.)

Governance issues raised by digital minds

There are a lot of governance issues raised by the possibility of digital minds. For example, what sort of reform is needed in one-person-one-vote democracies when creating new persons is as easy as copying software? See also Develop candidate regulation from the “Digital Sentience & Rights” post in this series.

Norms/proposals for how to navigate an intelligence explosion [Governance] [Forecasting] [Philosophical/conceptual]

Painting a picture of a great outcome suggested outlining an acceptable “endpoint” to explosive growth.

Separately, there’s a question of what the appropriate norms are for how we get from where we are today to that situation.

Other than risks from misaligned AI along the way, I think the 3 central points here are:

• If one actor pushes ahead with an intelligence explosion, they might get a massive power advantage over the rest of the world. That creates a big risk for everyone else, who might find themselves powerless. Instead of going through with a massive gamble like that, could we set up agreements or norms that make it more likely that everyone has some say in the future?

• A maximum-speed intelligence explosion will lead to a lot of changes and a lot of new technology really, really fast. That’s a core part of what’s scary, here. Could we somehow coordinate to go slower?

• Good post-intelligence-explosion worlds will (at least eventually) look quite different from our world, and that includes governance & politics looking quite different. For example:

  • There will be more focus on deciding what sorts of crazy technology can be used in what ways.

  • There will be less need to focus on economic growth to meet the material needs of currently existing people.4

  • There will be massive necessary changes associated with going from purely biological citizens to also having digital minds with political rights.

  • There may be changes in what form of government is naturally favored. Worryingly, perhaps away from favoring democracy,5 which could return us to autocratic forms of rule (which are more historically typical) unless we make special efforts to preserve democracy.

  • There will be strong incentives to hand over various parts of politics to more effective AI systems. (Drafting proposals, negotiating, persuading people about your point of view, etc.)

  It would be nice to get a head start on making this governance transition happen smoothly and deliberately.

What follows are some candidate norms and proposals. Each of them could be:

• Sketched out in much greater detail.

• Evaluated for how plausible it is to form a credible norm around it.

• Evaluated for whether such a norm would be good or bad.

• Evaluated for whether partial movement in that direction would be good or bad. (So that we don’t propose norms that backfire if they get less than complete support.)

• Advocated for.

(For ideas in this section, I’ve especially benefitted from discussion with Carl Shulman, Will MacAskill, Nick Beckstead, Ajeya Cotra, Daniel Kokotajlo, and Joe Carlsmith.)

No “first strike” intelligence explosion

One candidate norm could be: “It’s a grave violation of international norms for a company or a nation to unilaterally start an intelligence explosion because this has a high likelihood of effectively disempowering all other nations. A nation can only start an intelligence explosion if some other company or nation has already started an intelligence explosion of their own or if they have agreement from a majority of other nations that this is in everyone’s best interest.” (Perhaps because they have adequate preparations for reassuring other countries that they won’t be disempowered. Or perhaps because the alternative would be to wait for an even less responsible intelligence explosion from some other country.)

Never go faster than X?

The above proposal was roughly: Only step through an intelligence explosion if you have agreements from other nations about how to do it.

A more opinionated and concrete proposal is: Collectively, as a world, we should probably never grow GDP or innovate technology faster than a certain maximum rate. Perhaps something like: Technological and economic growth should be no more than 5x faster than 2023 levels.

Concrete decision-making proposals

A proposal that would be more opinionated in some ways, and less opinionated in other ways, would be: At some level of AI capabilities, you’re supposed to call for a grand constitutional convention to decide how to navigate the rest of the intelligence explosion, and what to do afterward.

You could make suggestions about this constitutional convention, e.g.:

• The first round will last for a year.

• There will be 1000 people participating, sampled from nations in proportion to their population. 

• People will use approval voting.

• etc.

The “constitutional convention” proposal resembles something like deliberative democracy insofar as the selected people are randomly chosen, and looks more like typical geopolitical deliberations insofar as nations’ governments decide which representatives to send. 

(Thanks to Will MacAskill for discussion.)

Technical proposals for slowing down / coordinating

In order to be especially effective, all 3 above proposals require credible technical proposals for how nations could verify that they were all collectively slowing down. Exactly how such proposals should work is a big open problem. Which, thankfully, some people are working on.

I don’t know any detailed list of open questions here. (And I’d be interested if anyone had such a list!) But some places to look are:

• Shavit (2023) is an important paper that brings up some open problems.

• Lennart Heim’s list of resources for compute governance.

Naming some other candidate directions:

• Research how to design computer chips and rules for computer chips that cleanly distinguish chips that can and can’t be used for AI training. So that restrictions on AI chips can be implemented with minimal consequences for other applications.

• Proposing schemes for how competitors (like the US & China) could set up sufficiently intense monitoring of each other to be confident that there’s no illicit AI research happening.

  • C.f. historical success at spying for detecting large operations.

• How feasible would it be to do an international “CERN for AI”-type thing?

  • In particular: To what extent could this possibly involve competitors who don’t particularly trust each other? Given risks of leaks and risks of backdoors or data poisoning.

• Great proposals here should probably be built around capability evaluations and commitments for what to do at certain capability levels. We can already start to practice this by developing better capability evaluations and proposing standards around them. It seems especially good to develop evaluations for AI speeding up ML R&D.

Dubiously enforceable promises

Get nations to make promises early on along the lines of “We won’t use powerful AI to offensively violate other countries’ national sovereignty. Not militarily, and not by weird circumspect means either (e.g. via superhuman AI persuasion).” Maybe promising other nations a seat at the bargaining table that determines the path of the AI-driven future. Maybe deciding on a default way to distribute future resources and promising that all departures from that will require broad agreement.

It is, of course, best if such promises are credible and enforceable.

But I think this would have some value even if it's just something like: The US Congress passes a bill that contains a ton of promises to other nations. And that increases the probability that they'll act according to those promises.

There’s a lot of potential work to do here in drafting suggested promises that would:

• Be meaningful in extreme scenarios.

• Be plausibly credible in extreme scenarios.

• Could plausibly be passed in a political climate that still has serious doubts about where all this AI stuff will lead.

(Thanks to Carl Shulman for discussion.)

Technical proposals for aggregating preferences

A different direction would be for people to explore technical proposals for effectively aggregating people’s preferences. So that, during an intelligence explosion, it’s more convenient to get more accurate pictures of what different constitutions would recommend. Thereby making it harder to legitimately dismiss such demands.

Previous work:

• Jan Leike’s proposal for “Building towards Coherent Extrapolated Volition with language models”.

• The Polis platform. (Which, notably, Anthropic used when exploring what constitution to choose for their constitutional AI methods.)

Decrease the power of bad actors

Bad actors could make dangerous decisions about what to do with AI technology. Perhaps increasing risks from brinkmanship or other destructive technology — or perhaps seizing and maintaining indefinite power over the future and making poor choices about what to do with it. See also some of the risks described in Reducing long-term risks from malevolent actors.

One line of attack on this problem is to develop policies for avoiding AI-assisted human coups. But here are a few more.

Avoid malevolent individuals getting power within key organizations [Governance]

If you have opportunities to affect the policies of important institutions, it could be valuable to reduce the probability that malevolent individuals get hired and/or are selected for key roles.

Examples of how to attack this question: (h/t Stefan Torges)

• Making it more likely that malevolent actors are detected before joining an AI development effort (e.g., screenings, background checks).

• Making it more likely that malevolent actors are detected within AI development efforts (e.g., staff training, screenings for key roles).

• Making it more likely that staff speak up about / report suspicious behavior (e.g., whistleblower protections, appropriate organizational processes).

• Making it more likely that malevolent actors are removed based on credible evidence (e.g., appropriate governance structures).

• Setting up appropriate access controls within AI development efforts. E.g. requiring multiple people’s simultaneous approval for crucial types of access and/or reducing the number of people with the power to access the models (unilaterally or with just a couple of people).

• Changing promotion guidelines and/or culture in ways that select against rather than for power-seeking individuals.

See also Reducing long-term risks from malevolent actors for more analysis and intervention ideas.

Prevent dangerous external individuals/organizations from having access to AI [Governance] [ML]

There is (thankfully) some significant effort going on in this space already, so I don’t have a lot to add.

Examples of how to attack this question:

• Improving security. (I think Securing AI Model Weights is the current state of the art of how labs could improve their security.)

• Offering dangerous frontier models via API rather than giving weights access.

• Controlling hardware access.

Accelerate good actors

This is a riskier proposition, and it’s extremely easy to accidentally do harm here. But in principle, one way to give bad actors relatively less power is to differentially accelerate good actors. C.f. successful, careful AI lab section of Holden Karnofsky’s playbook for AI risk reduction.

Analyze: What tech could change the landscape? [Forecasting] [Philosophical/conceptual] [Governance]

If an intelligence explosion could lead to 100 years of “normal” technological progress within just a few years, then this is a very unusually valuable time to have some foresight into what technologies are on the horizon.

It seems particularly valuable to anticipate technologies that could (i) pose big risks or (ii) enable novel solutions to other risks.

On (i), some plausible candidates are:

• Bioweapons.

• Misaligned-by-default, superintelligent AI.

• Super-persuasion. (Discussed a bit in the Epistemics post in this series.)

Some candidates that could feature on both (i) and (ii):

• Lie detection.

• Various new surveillance/monitoring technologies.

  • Including: Better ability to have existing monitoring technology be privacy-preserving. (Which is more purely (ii).)

• Commitment abilities.6

• Atomically precise manufacturing.

• Cognitive enhancement.

Good results here could influence:

• Technical proposals for slowing down / coordinating.

• Painting a picture of a great outcome.

• Investigate and publicly make the case for/against explosive growth being likely + scary.

Big list of questions that labs should have answers for [Philosophical/conceptual] [Forecasting] [Governance]

This could be seen as a project or could alternatively be seen as a framing for how to best address many of these issues. (Not just from this post, but also issues from other posts in this series.)

I think it’s plausible that our biggest “value-add” on these topics will be that we see potentially important issues coming before other people. This suggests that our main priority should be to clearly flag all the thorny issues we expect to appear during an intelligence explosion and ask questions about how labs (and possibly other relevant institutions) plan to deal with them.

This could be favorably combined with offering suggestions for how to address all the issues. But separate from any suggestions, it’s valuable to establish something as a problem that needs some answer so that people can’t easily dismiss any one solution without offering an alternative one.

Some of these questions might be framed as “What should your AI do in situation X?”. Interestingly, even if labs don’t engage with a published list of questions, we can already tell what (stated) position the labs’ current AIs have on those questions. They can be presented with dilemmas, the AIs can answer, and the results can be published.

Having a single big list of questions that must be addressed would also make it easier to notice when certain principles conflict. I.e., situations when you can’t fulfill them both at once and are forced to choose.

Example of how to attack this:

• Ask yourself: “If the lab never made an intentional decision about how they or their AIs should handle X — how worried would you be?”. Write a list with versions of X, starting from the ones that would make you most worried. Also, write a hypothesized solution to it. (Both because solutions are valuable and because this exercise encourages concreteness.)

  • As a starting point for what to write about, you could consider many of the ideas in this series of posts. What will you do if your AI systems have morally relevant preferences? What if they care about politics and deserve political rights? What if you find yourself in a situation where you could unilaterally launch an intelligence explosion? Or where control over your AI systems could enable someone to launch a coup? Etc.

(Thanks to Carl Shulman for this idea and discussion.)

End

That’s all I have on this topic! As a reminder: it's very incomplete. But if you're interested in working on projects like this, please feel free to get in touch.

Other posts in series: Introduction, governance during explosive growth, epistemics, sentience and rights of digital minds, backup plans & cooperative AI.

• Would be especially valuable if transformative AI is coming in the next 10 years or so.

• Are not primarily about controlling AI or aligning AI to human intentions.1

  • Most of the projects would be valuable even if we were guaranteed to get aligned AI.

  • Some of the projects would be especially valuable if we were inevitably going to get misaligned AI.

The posts contain some discussion of how important it is to work on these topics, but not a lot. For previous discussion (especially: discussing the objection “Why not leave these issues to future AI systems?”), you can see the section How ITN are these issues? from my previous memo on some neglected topics.

The lists are definitely not exhaustive. Failure to include an idea doesn’t necessarily mean I wouldn’t like it. (Similarly, although I’ve made some attempts to link to previous writings when appropriate, I’m sure to have missed a lot of good previous content.)

There’s a lot of variation in how sketched out the projects are. Most of the projects just have some informal notes and would require more thought before someone could start executing. If you're potentially interested in working on any of them and you could benefit from more discussion, I’d be excited if you reached out to me!2

There’s also a lot of variation in skills needed for the projects. If you’re looking for projects that are especially suited to your talents, you can search the posts3 for any of the following tags (including brackets):

[ML]   [Empirical research]   [Philosophical/conceptual]   [survey/interview]   [Advocacy]   [Governance]   [Writing]   [Forecasting]

The projects are organized into the following categories (which are in separate posts). Feel free to skip to whatever you’re most interested in.

• Governance during explosive technological growth

  • It’s plausible that AI will lead to explosive economic and technological growth. 

  • Our current methods of governance can barely keep up with today's technological advances. Speeding up the rate of technological growth by 30x+ would cause huge problems and could lead to rapid, destabilizing changes in power.

  • This section is about trying to prepare the world for this. Either generating policy solutions to problems we expect to appear or addressing the meta-level problem about how we can coordinate to tackle this in a better and less rushed manner.

  • A favorite direction is to develop Norms/proposals for how states and labs should act under the possibility of an intelligence explosion.

• Epistemics

  • This is about helping humanity get better at reaching correct and well-considered beliefs on important issues.

  • If AI capabilities keep improving, AI could soon play a huge role in our epistemic landscape. I think we have an opportunity to affect how it’s used: increasing the probability that we get great epistemic assistance and decreasing the extent to which AI is used to persuade people of false beliefs.

  • A couple of favorite projects are: Create an organization that gets started with using AI for investigating important questions or Develop & advocate for legislation against bad persuasion.

• Sentience and rights of digital minds

  • It’s plausible that there will soon be digital minds that are sentient and deserving of rights. This raises several important issues that we don’t know how to deal with.

  • It seems tractable both to make progress in understanding these issues and in implementing policies that reflect this understanding.

  • A favorite direction is to take existing ideas for what labs could be doing and spell out enough detail to make them easy to implement.

• Backup plans for misaligned AI

  • If we can’t build aligned AI, and if we fail to coordinate well enough to avoid putting misaligned AI systems in positions of power, we might have some strong preferences about the dispositions of those misaligned AI systems.

  • This section is about nudging those into somewhat better dispositions (in worlds where we can’t align AI systems well enough to stay in control).

  • A favorite direction is to study generalization & AI personalities to find easily-influenceable properties.

• Cooperative AI

  • Difficulties with cooperation have been a big source of lost value and unnecessary risk in the past. AI offers dramatic changes in how bargaining could work.

  • This section is about projects that could make AI (and AI-assisted humans) more likely to handle cooperation well.

  • One of my favorite projects here is actually the same as the project I mentioned for “backup plans”, just above. (There’s significant overlap between the two.)

(If you want to comment on any of the posts in this series, you could do so either here, at the EA forum, or on LessWrong.)

Acknowledgements

Few of the ideas in these posts are original to me. I’ve benefited from conversations with many people. Nevertheless, all views are my own.

For some projects, I credit someone who especially contributed to my understanding of the idea. If I do, that doesn’t mean they have read or agree with how I present the idea (I may well have distorted it beyond recognition). If I don’t, I’m still likely to have drawn heavily on discussion with others, and I apologize for any failure to assign appropriate credit.

For general comments and discussion, thanks to Joseph Carlsmith, Paul Christiano, Jesse Clifton, Owen Cotton-Barrat, Daniel Kokotajlo, Linh Chi Nguyen, Fin Moorhouse, Caspar Oesterheld, and Carl Shulman.

Appendix: Full table of contents

Here’s a list with all the project ideas from the other posts. (Sorry it’s not hyper-linked.) Unless you’re looking for something specific, I suggest jumping into the first post instead of reading this.

Project ideas: Governance during explosive technological growth

• Investigate and publicly make the case for/against explosive growth being likely and risky [Forecasting] [Empirical research] [Philosophical/conceptual] [Writing]

• Painting a picture of a great outcome [Forecasting] [Philosophical/conceptual] [Governance]

• Policy-analysis of issues that could come up with explosive technological growth [Governance] [Forecasting] [Philosophical/conceptual]

  • Address vulnerable world hypothesis with minimal costs

  • How to handle brinkmanship/threats?

  • Avoiding AI-assisted human coups

  • Governance issues raised by digital minds

• Norms/proposals for how to navigate an intelligence explosion [Governance] [Forecasting] [Philosophical/conceptual]

  • No “first strike” intelligence explosion

  • Never go faster than X?

  • Concrete decision-making proposals

  • Technical proposals for slowing down / coordinating

  • Dubiously enforceable promises

  • Technical proposals for aggregating preferences

• Decrease the power of bad actors

  • Avoid malevolent individuals getting power within key organizations [Governance]

  • Prevent dangerous external individuals/organizations from having access to AI [Governance] [ML]

  • Accelerate good actors

• Analyze: What tech could change the landscape? [Forecasting] [Philosophical/conceptual] [Governance]

• Big list of questions that labs should have answers for [Philosophical/conceptual] [Forecasting] [Governance]

Project ideas: Epistemics

• Why AI matters for epistemics

• Why working on this could be urgent

• Categories of projects

• Differential technology development [ML] [Forecasting] [Philosophical/conceptual]

  • Important subject areas

  • Methodologies

  • Related/previous work.

• Get AI to be used & (appropriately) trusted

  • Develop technical proposals for how to train models in a transparently trustworthy way [ML] [Governance]

  • Survey groups on what they would find convincing [survey/interview]

  • Create good organizations or tools [ML] [Empirical research] [Governance]

  • Examples of organizations or products

  • Investigate and publicly make the case for why/when we should trust AI about important issues [Writing] [Philosophical/conceptual] [Advocacy] [Forecasting]

  • Developing standards or certification approaches [ML] [Governance]

• Develop & advocate for legislation against bad persuasion [Governance] [Advocacy]

Project ideas: Sentience and rights of digital minds

• Develop & advocate for lab policies [ML] [Governance] [Advocacy] [Writing] [Philosophical/conceptual]

  • Create an RSP-style set of commitments for what evaluations to run and how to respond to them

  • Policies that don’t require sophisticated information about AI preferences/experiences

  • Preserving models for later reconstruction

  • Deploy in “easier” circumstances than trained in

  • Reduce extremely out of distribution (OOD) inputs

  • Train or prompt for happy characters

  • Committing resources to research on AI welfare and rights

  • Learning more about AI preferences

  • Credible offers

  • Talking via internals

  • Training for honest self-reports

  • Clues from AI generalization

  • Interpretability

  • Interventions that rely on understanding AI preferences

  • Offer an alternative to working (exit, sleep, or retirement)

  • Commitment to pay AI systems

  • Tell the world

  • Train AI systems that suffer less and have fewer preferences that are hard to satisfy

• Investigate and publicly make the case for/against near-term AI sentience or rights [Philosophical/conceptual] [Writing]

• Study/survey what people (will) think about AI sentience/rights [survey/interview]

• Develop candidate regulation [Governance] [Forecasting]

• Avoid inconvenient large-scale preferences [Philosophical/conceptual]

• Advocating for statements about digital minds [Governance] [Advocacy] [Writing]

Project ideas: Backup plans & Cooperative AI

• Backup plans for misaligned AI

  • What properties would we prefer misaligned AIs to have? [Philosophical/conceptual] [Forecasting]

    • Making misaligned AI have better interactions with other actors

    • AIs that we may have moral or decision-theoretic reasons to empower

    • Making misaligned AI positively inclined toward us

  • Studying generalization & AI personalities to find easily-influenceable properties [ML]

  • Theoretical reasoning about generalization [ML] [Philosophical/conceptual]

• Cooperative AI

  • Implementing surrogate goals / safe Pareto improvements [ML] [Philosophical/conceptual] [Governance]

  • AI-assisted negotiation [ML] [Philosophical/conceptual]

  • Implications of acausal decision theory [Philosophical/conceptual]

Some example neglected topics (without much elaboration)

Here are a few example topics that could matter a lot if we’re in the most important century, which aren’t always captured in a normal “AI alignment” narrative:

• The potential moral value of AI.1

• The potential importance of making AI behave cooperatively towards humans, other AIs, or other civilizations (whether AI ends up intent-aligned or not).

• Questions about how human governance institutions will keep up if AI leads to explosive growth.

• Ways in which AI could cause human deliberation to get derailed, e.g. powerful persuasion abilities.

• Positive visions about how we could end up on a good path towards becoming a society that makes wise and kind decisions about what to do with the resources accessible to us. (Including how AI could help with this.)

(More elaboration on these below.)

Here are a few examples of somewhat-more-concrete things that it might (or might not) be good for some people to do on these (and related) topics:

• Develop proposals for how labs could treat digital minds better, and advocate for them to be implemented. (C.f. this nearcasted proposal.)

• Advocate for people to try to avoid building AIs with large-scale preferences about the world (at least until we better understand what we’re doing). In order to avoid a scenario where, if some generation of AIs turn out to be sentient and worthy of rights, we’re forced to choose between “freely hand over political power to alien preferences” and “deny rights to AIs on no reasonable basis”.

• Differentially accelerate AI being used to improve our ability to find the truth, compared to being used for propaganda or manipulation.

  • E.g.: Start an organization that uses LLMs to produce epistemically rigorous investigations of many topics. If you’re the first to do a great job of this, and if you’re truth-seeking and even-handed, then you might become a trusted source on controversial topics. And your investigations would just get better as AI got better.

  • E.g.: Evaluate and write-up facts about current LLM’s forecasting ability, to incentivize labs to make LLMs state correct and calibrated beliefs about the world.

  • E.g.: Improve AI ability to help with thorny philosophical problems.

Implications for community building?

…with a focus on “the extent to which community-building resources should go towards AI safety vs. broader capacity building”.

• Ethics, philosophy, and prioritization matter more for research on these topics than it does for alignment research.

  • For some issues in AI alignment, there’s a lot of convergence on what’s important regardless of your ethical perspective, which means that ethics & philosophy aren’t that important for getting people to contribute. By contrast, when thinking about “everything but alignment”, I think we should expect somewhat more divergence, which could raise the importance of those subjects.

    • For example:

      • How much to care about digital minds?

      • How much to focus on “deliberation could get off track forever” (which is of great longtermist importance) vs. short-term events (e.g. the speed at which AI gets deployed to solve all of the world’s current problems.)

    • But to be clear, I wouldn’t want to go hard on any one ethical framework here (e.g. just utilitarianism). Some diversity and pluralism seems good. 

  • In addition, the huge variety of topics especially rewards prioritization and a focus on what matters, which is perhaps more promoted by general EA community building than AI safety community building?

    • Though: Very similar virtues also seem great for AI safety work, so I’m not sure if this changes much.

  • And if we find more shovel-ready interventions, for some of these topics, then I imagine that they would be similar to alignment, on these dimensions.

• It seems relatively worse to go too hard on just “get technical AI safety researchers”.

  • But that would have seemed like a mistake anyway. AI governance looks great even if you’re just concerned about alignment. Forecasting AI progress (and generally getting a better understanding of what’s going to happen) looks great even if you’re just concerned about alignment.

• It seems relatively worse to go too hard on just “get people to work towards AI alignment” (including via non-technical roles).

  • But in practice, it’s not clear that you’d talk about very different topics if you were trying to find people to work on alignment, vs. if you were trying to find people to work on these topics.

  • In order for someone to do good work on alignment-related topics, I think it’s very helpful to have some basic sense of how AI might accelerate innovation and shape society (which is important for the topics listed above).

  • Conversely, in order for someone to do good work on other ways in which AI could change the world, I still think that it seems very helpful to have some understanding of the alignment problem, and plausible solutions to it.

  • Relatedly…

• Focusing on “the most important century” / “transformative AI is coming” works well for these topics.

  • Let’s put “just focus on AI safety” to the side, and compare:

    • “EA”-community building, with

    • “let’s help deal with the most important century”-community building

  • I don’t think it’s clear which is better for these topics. Getting the empirics right matters a lot! If explosive technological growth is at our doorstep — then that’s a big deal, and I’m plausibly more optimistic about the contributions of someone who has a good understanding of that but who’s missing some other EA virtues, than someone who doesn’t have a good understanding of that.

• Seems great to communicate that these kinds of questions are important and neglected. (Though also quite poorly scoped and hard to make progress on.)

  • If there are people who are excited about and able to contribute to some of these topics (and who don’t have a stronger comparative advantage for anything in e.g. alignment) then it seems pretty likely they should work on them.

Elaborating on the example topics

Elaborating on the topics I mentioned above.

• Moral value of AI.

  • What does common sense morality say about how we should treat AIs?

  • How can we tell whether/which AI systems are conscious?

  • If we fail to get intent-aligned AI, are there nevertheless certain types of AI that we’d prefer to get over others? See Paul Christiano’s post on this; or my post on what “evidential cooperation in large worlds” has to say about it.

• The potential importance of making AI behave cooperatively towards humans, other AIs, or other civilizations (independently of whether it ends up aligned).

  • E.g. making AIs less likely to be spiteful or more likely to implement good bargaining strategies like safe Pareto improvements.

• Ways in which AI could cause human deliberation to get derailed. Such as:

  • The availability of extremely powerful persuasion (see discussion e.g. here and here).

    • As an example intervention: It seems plausibly tractable to develop good regulatory proposals for reducing bad AI persuasion, and I think such proposals could gather significant political support.

  • Availability of irreversible commitment and lock-in abilities.

  • If all of humans’ material affairs will be managed by AIs (such that people’s competencies and beliefs won’t affect their ability to control resources) then maybe that could remove an important incentive and selection-effect towards healthy epistemic practices. C.f. decoupling deliberation from competition.

• Questions about how human governance institutions will keep up as AI leads to explosive growth.

  • If we will very quickly develop highly destabilizing technologies,2 how can the world quickly move to the level of coordination that is necessary to handle those?

  • Can we reduce the risk of AI-enabled human coups — e.g. avoid being in situations where AIs are trained to obey individual or small group of humans, without having been trained on cases where those humans try to grab power.

  • Can we wait before creating billions of digital minds who deserve and want to exercise political rights? (At least for as long as our governance still relies on the principle of one-person one-vote.)

• Positive visions about how we could end up on a good path towards becoming a society that makes wise and kind decisions about what to do with the resources accessible to us. (Including how AI could help with this.)

  • (Including how AI could help with this.)

  • E.g.: Elaboration on whether any type of “long reflection” would be a good idea. 

  • E.g.: A vision of a post-AI world that will make everybody decently happy, that’s sufficiently credible that people can focus on putting in controls that gets us something at least that good, instead of personally trying to race and grab power. (C.f.: Holden Karnofsky’s research proposal here.)

Nick Bostrom and Carl Shulman’s propositions concerning digital minds and society has some good discussion of a lot of this stuff.

How ITN are these issues?

How good do these topics look in an importance/neglectedness/tractability framework? In my view, they look comparable to alignment on importance, stronger on neglectedness (if we consider only work that’s been done so far), and pretty unclear on tractability (though probably less tractable than alignment).

For example, let’s consider “human deliberation could go poorly (without misalignment or other blatant x-risks”).

• Importance: I think it’s easy to defend this being 10% of the future, and reasonable to put it significantly higher.

• Neglectedness: Depends on what sort of work you count.

  • If we restrict ourselves to work that’s been done so-far that thinks about this in the context of very fast-paced technological progress, it seems tiny. <10 FTE years in EA, and I don’t know anything super relevant outside.

• Tractability:

  • Very unclear!

  • In general, most problems fall within a 100x tractability range.

  • Given how little work has been done here, so far, most of the value of additional labor probably comes from information value about how tractable it is. That information value seems pretty great to me — absent specific arguments for why we should expect the problem to not be very tractable.

So let’s briefly talk about a specific argument for why these neglected topics might not be so great: That if we solve alignment, AI will help us deal with these problems. Or phrased differently: Why spend precious hours on these problems now, when cognitive resources will be cheap and plentiful soon enough.3

I think this argument is pretty good. But I don’t think it’s overwhelmingly strong:

• Issues could appear before AI is good enough to obsolete us.

  • For example: Strong persuasion could appear before AI gets excellent at figuring out solutions to strong persuasion.

  • This is amplified by general uncertainty about the distribution of AI capabilities. Although AI will accelerate progress in many areas, we should have large uncertainty about how much it will accelerate progress in different areas. So for each of the issues, there’s non-negligible probability that AI will accelerate the area that causes the problem (e.g. tech development) before it accelerates progress on the solution (e.g. forecasting potential harms from tech development).

  • (Note that a plausible candidate intervention here, is: “differentially accelerate AI’s ability to provide solutions relative to AI’s ability to cause problems”.)

• There might not be enough time for some actions later.

  • For example: Even with excellent AI advice, it might be impossible for the world’s nations to agree on a form of global governance in less than 1 month. In which case it could have been good to warn about this in advance.

• “Getting there first” could get you more ears.

  • For example: See the LLM-fueled organization with good epistemics, that I suggested in the first section, which could get a good reputation.

  • For example: Writing about how to deal with a problem early-on could shape the discussion and get you additional credibility.

• Expectations about the future shape current actions.

  • If people think there are broadly acceptable solutions to problems, then they might be more inclined to join a broad coalition and ensure that we get something at least as good as that which we know is possible.

  • If people have no idea what’s going to happen, then they might be more desperate to seek power, to ensure that they have some control over the outcome.

• One topic is to come up with candidate back-up plans to alignment. That matters in worlds where we don’t succeed at alignment well-enough to have AI do the research for us.

  • See “moral value of AI”, or some topics in cooperative AI, mentioned above.

So I don't think “AI will help us deal with these problems” is decisive. I’d like to see more attempted investigations to learn about these issues’ tractability.

The long-term future of intelligent life is currently unpredictable and undetermined. We argue that the invention of artificial general intelligence (AGI) could change this by making extreme types of lock-in technologically feasible. In particular, we argue that AGI would make it technologically feasible to (i) perfectly preserve nuanced specifications of a wide variety of values or goals far into the future, and (ii) develop AGI-based institutions that would (with high probability) competently pursue any such values for at least millions, and plausibly trillions, of years.

This is a piece of research that I wrote together with Jess Riedel and Carl Shulman while I was at the Future of Humanity Institute. It’s available on Forethought’s website at forethought.org/research/agi-and-lock-in.

• Asymmetric ECL

• When does EDT seek evidence about correlations?

It tackles the intersection of the two: How does asymmetric ECL work in situations where at least one party has the opportunity to gather evidence about how large the correlations are.

More precisely, I investigate a situation where at least one party can:

• Linearly adjust how much they help the other party.

• Seek out evidence about how large correlations are. (In the sense that I talked about in When does EDT seek evidence about correlations?)

• Make commitments about how much they will help the other party given various amounts of evidence.

Just as in Asymmetric ECL, we can ask: Under what conditions will there be a mutually beneficial deal that both parties are incentivized to follow?

The main result is that such a deal exists in similar circumstances as in Asymmetric ECL, except that (for some of the correlations) instead of using the correlations that we believe before taking into account evidence, we can use any correlations such that there is a sufficiently high probability that one of the parties will believe in those correlations after observing evidence. 

(When I say “for some of the correlations”: If group B can investigate and gather evidence for correlations, then the above statement applies to c_AB and c_BB. I.e., actors’ perceived acausal influence on members of group B.)

What do I mean with “sufficiently high probability”? It depends on the risk-aversion of the agents. If one agent can sacrifice arbitrarily much to benefit the other agent arbitrarily much (at a linear rate), and neither party has any relevant bound on their utility functions nor relevant lack of resources, then we can use a similar formula as in Asymmetric ECL but plug-in the best possible correlations that an actor has a non-0 probability of receiving evidence for.

• (Again — If group B can investigate and gather evidence for correlations, then the above statement applies to c_AB and c_BB. I.e., actors’ perceived acausal influence on members of group B.)

• When I say “best possible correlations”, I’m referring to the correlations that would most encourage actors to do an ECL deal. (I.e. low correlations with actors who share your values, and high correlations with actors who don’t share your values.)

• When I talk about correlations “that an actor has a non-0 probability of receiving evidence for”, I mean correlations such that (e.g.) group B has a non-0 probability of gathering evidence such that group B’s posterior estimate is that the relevant correlations are that large.

In practice, this really tests the assumption of linearity that I was relying on in Asymmetric ECL. Realistic application would have to take into account the realistic bounds on actors’ utility functions and the resources they have access to. (I don’t think that a 10^-100 probability of a high correlation significantly changes what deals are possible.)

Nevertheless, in this post, I will just show that the result holds in completely linear situations. 

My brief takeaways are that:

• Behind the veil of ignorance, agents may be incentivized to commit to very different deals then they would be incentivized to follow later-on. (Because uncertainty enables some deals.)

• This results makes me somewhat more optimistic about doing ECL deals with agents that will become very knowledgeable before they need to pay us back.

Quick recap

When does EDT seek evidence about correlations? says:

In some scenarios, son-of-EDT is interested in seeking evidence about what actors would have perceived themselves as having acausal influence over son-of-EDT’s parent agent, and especially benefit those who would have perceived themselves as having a large acausal influence on son-of-EDT’s parent agent.

The main relevance of that post to this post is that I will use a similar notion of “evidence of correlations” as that post uses, and that some of the results are quite similar. (And that other post is better at building intuition for those results.)

Asymmetric ECL says:

Let’s say a group A and a group B have opportunities to benefit each other, where:

• n_A is the number of agents in group A, and n_B is the number of agents in group B, where n_A and n_B are large.

• c_AB is the acausal influence (measured in percentage points) that members of group A perceives themselves to have over members of group B. (And vice versa for c_BA). 

• c_AA is the acausal influence that members of group A perceives themselves to have over other members of group A. (And similarly for c_BB.)

• Members of group A have an opportunity to benefit the values of group B by g_B at a cost of l_A to their own values.

• Members of group B have an opportunity to benefit the values of group A by g_A at a cost of l_B to their own values. Furthermore, they can adjust the size of this benefit and cost linearly, such that for any k, they can choose to benefit the values of group A by kg_A at a cost of kl_B.

Then:

• In order for a policy to be good for group A,

  

• In order for a policy to be good for group B, 

Which means that, if one of the parties can linearly adjust their contribution so as to proportionally increase or decrease both l_B and g_A, then a mutually beneficial deal is compatible with individual incentives whenever:

Example to give intuition

Say Alice and Bob are about to decide what to do with their resources.1 Alice can pay $2 to give Bob $4. Bob can pay $2 to give Alice $4 or pay $3 to give Alice $6. Before making a choice, Bob observes whether the correlation between Bob and Alice is 0.8 or 0.2. The higher value obtains with probability p (according to both's prior). Alice makes no such observation. Their shared prior is that the correlation is

Call this c_p, the correlation according to the prior.

Imagine that before Bob observes the value of the correlation, they try to arrive at some deal. First consider the following deal:

• Regardless of how much correlation there is, Bob and Alice both pay $2 to give the other player $4.

• Assume that Bob’s and Alice’s correlations on the decision “doing their part of this deal vs. not sending any money at all” is as I described above — if this assumption would make the deal net-positive for both players.2

The expected gain of accepting this deal (relative to not accepting it) to each of Bob and Alice is:

This is positive (i.e. Alice and Bob are both incentivized to take the deal) if c_p>0.5, which is true iff p>0.5.

Now instead imagine that they were deciding whether to make the following deal:

• Alice pays $2 to give Bob $4.

• Bob commits to paying $3 to give Alice $6 if correlation is high (0.8). Bob doesn't pay anything if correlation is low (0.2).

• Imagine again that the decision of whether to accept this deal is correlated as per the above numbers.

​​Then the expected gain of accepting the deal for Alice is:

This is positive if p > 2/4.8 ~= 40%. note that this lower bound is smaller than the 50% above.

The expected gain for Bob is:

This is positive for all p between 0 and 1. (At p=0 the gain is 0.8. At p=1 the gain is 0.2.)

So, in particular, for values of p between 40% and 50%, the first deal is bad for both players but the second deal is good for both players.

Intuitively, why does this work? The key idea is that the second deal has Bob pay only if correlation is high. Holding fixed the expected (w.r.t. p) amount that Bob gives in a deal, a deal looks better from Alice’s perspective if Bob gives the money in the high correlation case, because those are the possible worlds where Alice’s decision has the most influence on Bob’s decision.

In the above example, note that (unless p is very high) the first deal (where Bob always gives money) gives Alice more money in expectation if both players take the deal ($4 instead or 0.5*6=$3). But Alice’s decision is about whether she herself should take the deal. And if Alice just conditions on her own decision to participate, then the first deal only gets her c_p of the $4, in expectation — whereas the second deal gets Alice 0.8>c_p of the $3, in expectation.

Let’s generalize

Let’s now study the general version.

We will consider some set of possible worlds W, where for each world w in W:

• Both Alice and Bob assign the same prior p(w) to that world.3

• Alice thinks her influence on Bob is c_AB|w in that word. (And vice versa for Bob.)

• Alice and Bob can pay different amounts to benefit each other in different worlds. As a shorthand for the amount that Alice pays to benefit Bob in world w, I will write l_A|w. (And correspondingly for Bob.)

• We will write the benefit that Bob gains as a monotonically increasing function of how much Alice pays: g_B(l_A|w). (And vice versa for Alice.)

This means that we can write…

• Alice’s expected gain as

  

• Bob’s expected gain as

Similarly to When does EDT seek evidence about correlations?, I will assume that both parties first commit to certain policies (based on what would give both parties high expected utilities) and then they make observations about what world they are in. Also similarly: The parties never get to observe information about each other, including what commitments the other party made. The only relevant information they get is evidence about correlations.

How should Alice and Bob choose l_A|w and l_B|w to maximize expected benefits to both parties? (As they might want to do if they’re selecting their policy using a similar algorithm as the one described here.)

In the above formula:

• l_A|w is multiplied by p(w).

• g_B(l_A|w) is multiplied by c_BA|w and p(w).

So Alice should choose especially high l_A|w in worlds where Bob’s acausal influence on Alice c_BA|w is especially high. (Echoing the conclusions from When does EDT seek evidence about correlations?)

Let’s extend this to a dilemma with large groups in large worlds. As previously outlined in the section large-worlds asymmetric dilemma (in the post Asymmetric ECL), group A would normally receive an expected utility of:

Extending this to our new setting:

• Group A’s expected gain is

  

• Group B’s expected gain is

How should group A and group B choose lA|w  and lB|w, here? 

• l_A|w is multiplied by n_A|wc_AA|w.

• g_B(l_A|w) is multiplied by n_A|wc_BA|w.

• So group A should choose higher l_A|w in worlds with a higher value of c_BA|w and a lower value of c_AA|w, i.e., in worlds where members of group B have surprisingly much acausal influence on them, and members of group A have surprisingly little acausal influence on each other.

• Note that n_A|w feature in both expressions, so updates on that number doesn’t clearly affect whose values should be benefitted.

Assuming linearity

Let’s be more precise about the options of group A and B:

• Let’s say that members of group A get no information about what world they are in. So they need to choose a single value of l_A|w for all w. Furthermore, let’s say they can either choose to make this 0, for which case g_B(0)=0, or a specific other number l_A. To represent g_B(l_A), let’s simply write g_B.

• Let’s say that members of group B get complete information about what world they are in, so they can freely distinct values for all l_B|w. Furthermore, let’s say that they can freely choose any non-negative value for l_B|w, and that g_A(l_B|w) is directly proportional to l_B|w. Since it’s directly proportional, let’s write g_A(l_B|w)=k_Al_B|w.

This means that:

• Group A’s expected gain is

  

• Group B’s expected gain is

Here’s something surprising, that we can now conclude: In order to maximize the expected value that both actors get, group B should only benefit group A (i.e., chose non-0 l_B|w) in the world(s) where the ratio

is the highest. Why?

Let’s say that there is some world u that maximizes

Let’s say that l_B|w’=l>0 for some other world w’ where

(i.e., group B helps group A in some world where the ratio isn’t the highest.)

Then, if the joint policy (recommended by the algorithm) instead recommended group B to:

• Decrease l_B|w’ by l (i.e. set l_B|w’ to 0).

• Increase l_B|u by

How much would group A’s expected utility change?

• Due to getting less utility in world w’:

  

• Due to getting more utility in world u:

  

Since these are equal, group A’s expected utility would be unchanged.

How much would group B’s expected utility change?

• Due to losing less utility in world w’:

  

• Due to losing more utility in world u:

  

• Summing these together, we can factor out some numbers and get:

  

• By assumption,

  

  So this will increase group B’s expected utility.

This shows that for every policy that recommends group B to benefit group A in world that doesn’t maximize

there’s a different policy that is Pareto-better. So if the algorithm only recommends Pareto-optimal policies, it will only recommend policies where group B only benefits group A in the world(s) with maximum

Note that this crucially relies on the assumption that group A’s gain is directly proportional to group B’s loss, and that group B can increase their loss arbitrarily much.

When is a deal possible?

Now, as I did in Asymmetric ECL, let’s try to find out when it is possible for group A and group B to find a mutually beneficial deal.

Based on the above argument, we know that group B will only benefit group A in the world where

is the highest. Let’s call this world u. We can then write:

• Group A’s expected gain is:

  

• Group A’s expected gains are larger than 0 if and only if:

  

• Group B’s expected gain is:

  

• Group B’s expected gains are larger than 0 if and only if:

  

By a similar argument as made in section “Generalizing" of Asymmetric ECL, it is possible for group B to pick a value of l_B|u that makes both these expressions larger than 1 whenever the product of them is larger than 1. I.e. when:

This is similar to the main result in Asymmetric ECL. That result said that a deal was possible whenever:

The only differences are that:

• We’re writing k_A instead of g_A/l_B. This is just a difference in notation — they represent the same value.

• Instead of c_AB/c_BB we have c_AB|u/c_BB|u — i.e., we only care about what that ratio is in the world where it’s maximally large.

• Instead of c_BA/c_AA we have

  

  — i.e., we care about the average correlations weighted by the probability of worlds and population size. If population size is independent from c_BA|w and c_AA|w, then this is simply equal to the expected value of c_BA|w divided by the expected value of c_BB|w.

What conclusions can we draw from this?

As I mentioned at the start: The assumption of unbounded, linear utility function and ability to pay isn’t realistic. For example, if we apply this lesson to ECL with AI, I don’t think that a world u such that p(u) = 10^-100 and c_AB|u/c_BB|u=1 should make us act just as if c_AB/c_BB=1.

In particular, some ways in which the unbounded, linear assumption can fail:

• Ultimately, I think that utility functions should be bounded, even if they’re linear in some regime. For sufficiently small probabilities, and sufficiently large values of l_B|u, we might get pushed outside of that regime. At some point, concerns about Pascal's mugging seem to bite.

• A natural way in which linearity can fail is if the AI runs out of resources to pay us with. That seems significantly more likely for small probability of large values of l_B|u.

• Maybe you just never thought that a linear utility function was a reasonable match to your own values. (In which case the small probability of large utility, introduced here, would increase the degree to which that assumption yields bad conclusions, from your perspective.)

Nevertheless,  I think this does push towards somewhat greater optimism about ECL deals with agents that can become knowledgeable and benefit us afterwards. If the probability of a high c_AB|w/c_BB|w isn’t too low, but is on the order of 10% or so, then maybe the argument in this post is just fine, and we should be happy with a 10% probability of getting a 10x larger benefit in those worlds.

Another take-away is: Behind the veil of ignorance, uncertainty about correlations may incentivize agents to commit to very different deals than they would be incentivized to follow later-on. All of this math relies on certain actors making ECL deals when they are ignorant about who they are correlated with — which could enable deals that would otherwise not be feasible. This is very sensitive to the initial probability distributions that agents start out with. Accordingly, this raises questions about when agents will (and should) first start making commitments. And about whether there’s any more principled way of handling this stuff then to be updateful until the day that you start making commitments, and subsequently letting that time-slice’s beliefs and preferences forever change how you act.

• Introduced the question of whether ECL says we should cooperate with distant AIs.

• Suggested a general formula for whether ECL recommends any mutually beneficial deal in asymmetric situations.

• Discussed how that formula could be applied to our situation.

Key inputs into the formula include:

• Whether us choosing to benefit AI values would provide evidence that AI will analogously benefit our values.

• Whether AIs benefitting our values would provide evidence that we will analogously benefit AI values.

This post will speculate about the values of those parameters.

(If your account of decision theory is different from evidential decision theory, you can substitute “provide evidence that” with your preferred metric of acausal influence.)

Summary

In general, ECL suggests that you should be more inclined to benefit the values of actors whom you correlate more with, in the sense that you perceive yourself as having more acausal influence on them, and they perceive themselves as having more acausal influence on you.

In particular, the post on asymmetric ECL suggests that your inclination to benefit distant AIs’ values should be proportional to your perceived acausal influence on them and their perceived acausal influence on you; but inversely proportional to your and AIs’ perceived acausal influence to actors with shared values.

From an EDT perspective, these “correlations” or this “acausal influence” might not be best viewed as objective facts about the world. Instead, they simply reflect the degree to which we consider our actions to be evidence for choices made in pre-AGI civilizations that share our values vs. evidence for choices made by AIs. (And vice versa, for the AIs.)

(I’m not yet fully sold on this perspective. I still feel quite mystified by how to determine the degree of “acausal influence” I have on others. Even if the EDT perspective turns out to be right, I at-least expect there to be a lot more to learn about what sort of reasoning does and doesn’t make sense when establishing the relevant conditionals.)

Taking the EDT perspective on-its-face, there’s an intuitively strong argument that our actions are significantly more evidence for actions taken in pre-AGI civilizations than actions taken by AIs. If I query my brain for intuitive predictions of distant actors, it sure seems like my own actions have more impact on my prediction of people in distant pre-AGI civilizations than on my predictions of misaligned AGI systems.

I think that intuition is worth paying attention to. But I think it’s less important than it naively seems. I’ll now go through a few different reasons for why you might have it, and a few different counterarguments. I’ll mostly be talking about this from an EDT perspective.

Different options. You might think that we can’t affect the AIs much because our option-space is very different. Us humans get to decide how much we should invest in alignment vs. research on ECL, and AGI systems of the future get to decide what to use their light cone for (or something like that). However, this seems like it’s much reduced by retreating to more abstract questions, like “Should I adopt a policy of impartially optimizing for many different values?”. See the post on asymmetric ECL for a more detailed description of what that abstract question could look like. See footnote for a caveat.1

Abstract reasoning vs. human quirks and biases. When we do decision-theoretic reasoning, we’re doing a type of reasoning where our conclusions correlate with AGIs’ conclusions. If I didn’t know anything about decision theory, and learned that humans upon extensive analysis thought that ECL “works”, then I would think it more likely that AGIs also would think ECL “works”.  Analogously: If I didn’t know anything about math, and learned that humans thought that the Riemann hypothesis was true, then I would think it more likely that AGIs would think the same thing.

I think this establishes some basic plausibility that our actions might correlate with AGIs’. On the other hand, it’s less clear that this makes sense when you are the one studying decision theory or math, and you are the ones making the decisions.

If you’ve discovered a lot of good arguments that the Riemann hypothesis is true, then you have already conditioned on the existence of those arguments, which is what influences your belief that AIs will conclude that they’re true. You get no further information based on whether you utter the words “the Riemann hypothesis is true”.

You can make a similar argument in the decision-theoretic case: If you’ve discovered a lot of good arguments that ECL “works”, then that’s what informs your beliefs about what the AIs will do (and that’s something you’ve already conditioned on). When you then act according to ECL, or not, you’re not getting further evidence about where the decision-theoretic arguments points. You’re just learning whether those arguments led to action within your human brain, with all its quirks and biases. And that’s much more evidence for what evolved creatures do than what AGIs do.

I’m currently unconvinced by this argument. For one, even before making a decision, you can also observe a lot of information about how the quirks and biases of your human brain are interpreting and interacting with the arguments. (Similar to the tickle defense in smoking lesion.) Which means that you also don’t get that much new information about your quirks and biases, in the act of making your decision. That puts the quirks and biases at a similar standing to the decision theoretic reasoning.

I also have an intuition that goes further, which I won't be able to fully describe in this paragraph. But to gesture at it: It seems to me like it’s a mistake for your decision theory to make decisions in order to provide “good news” about parts of your cognition that aren't responsive to decision-theoretic reasoning (such as your human-specific quirks and biases.)

See more here.

AGIs will know about us. The AGIs will naturally have more knowledge of us than we have of them, e.g. they might know how ECL-ish the pre-AGI civilization in their part of the universe was. If you’re an “updateful” EDT agent, then evidence of your counterpart’s actions that doesn’t come from your own actions will typically reduce the evidential power of your own action. So, from their perspective, their actions have little acausal influence on our choices. I think it’s correct that this would reduce correlations if AGIs are updateful. However, if they are sufficiently updateless, they would not take this knowledge into account when assessing their acausal influence — so this consideration mostly shifts our concern to sufficiently updateless AGIs.

See more here.

AGIs will have a deeper understanding of decision theory. I think there are two different versions of this concern, each with separate responses.

• AGI’s understanding of decision theory will make us predictable, in the sense that it will understand the principles that we use to make decisions so deeply that it will know what we choose before it makes its own decision. Thus, it will not see its own decisions as giving any evidence for ours.

  • The response to this is similar to the one just-above: If it’s sufficiently logically updateless about this, then it would not take this knowledge into account when assessing its acausal influence over us.

• AGI’s reasoning about decision theory will be so different that it correlates very little with us. If you think this is true, it seems like you’re committed to the idea that AGIs will reason about decision theory in a way that has barely any connection to how we reason about it. But if you expect AGI to be good at reasoning about decision theory, and you think that our own reasoning correlates little with their reasoning, then that would suggest that you’re pessimistic about us reaching any correct conclusions about decision theory. If this is your view, I agree that you should be pessimistic about doing ECL with the AIs. (If nothing else — because all of this reasoning about ECL have little chance of being correct, anyway.)

In the rest of this post, I expand a bit on these points. (Though this topic is very confusing to me, and I can’t promise that the expanded version will be very enlightening.)

There’s also an appendix on How to handle uncertainty? — when we’re uncertain about how much acausal influence we should perceive ourselves as having, on various groups.

In more depth…

Abstract reasoning vs. human quirks and biases

When we’re reasoning about decision theory, it seems like we’re doing a type of reasoning that both pre-AGI actors and philosophically ambitious AGIs should be doing. Ultimately, we’re trying to learn what the best way of making decisions is, whatever that means. If we’re doing that well, our conclusions about those abstract questions should correlate with the AGI’s conclusions — by virtue of us both being correct.2 Let’s call this “decision theoretic reasoning”. (Read those quotes as scare quotes — I don’t love the name.)

But there’s also many factors that influence our judgments that wouldn’t apply to AI systems. For example, maybe we would irrationally reject ECL because our social instincts kick in to save us from acting crazy. Or maybe we’ll irrationally accept ECL because of wishful thinking and because we want an excuse to act nicely towards everyone. Insofar as factors like these determine our decisions, our decisions really only correlate with humans and human-like species. Let’s call this “human-specific factors”.

When you learn about whether a human does ECL, you get evidence about both these components. Thus, observing a human doing something for ECL reasons is evidence both that (i) good “decision theoretic reasoning” implies that ECL makes sense, and (ii) that “human-specific factors” pushes humans towards acting according to ECL. The first of these is evidence that everyone is more likely to do ECL, the second is only evidence that human-like actors are likely to do ECL.

But if you are a human, you are in quite a different situation from someone observing a human. You know a lot more about why you’re taking the actions you do.

If you’re an updateful EDT agent, then as you’re thinking about arguments for and against taking some ECL-informed action, you will condition on the existence of those arguments. So insofar as uncertainty about the existence of those arguments were your only source of correlation with certain other agents, you won’t see yourself as having any power over them. Since you continuously condition-away your insights about “decision theoretic reasoning”, it’s not clear that your actions ever give significant evidence about where “decision theoretic reasoning“ points.

But it seems that similar arguments apply to ~all sources of influence on your decision-making — including the human-specific factors. Consider, for a moment, an analogy to smoking lesion and the tickle defense (see for example here for an introduction to the problem).

• The tickle defense goes: Once you notice an impulse to smoke, you’ve already received your evidence. Deciding to smoke or not gives you no additional information about your lesion.

• Analogously, here: Once you notice an impulse to (not) engage in ECL (due to wishful thinking, or social desirability bias, or anything like that), you’ve already received your evidence about the “human-specific factors”. Deciding to engage in ECL or not after that doesn’t give you any additional evidence about those.

I have a few different reactions to this:

• These kinds of arguments are very confusing, and it seems like they can’t work arbitrarily far. Until you’ve made your decision, you should maintain some uncertainty about what your decision will be, so there must be some factor that determines your choice that you don’t condition on.

  • Though some have argued that EDT agents sometimes can get confident about what their decisions will be before they make them, and that this indeed can have significant (and mostly bad) implications for their behavior. See footnote for examples.3

• I’m not sure I find the tickle defense as stated fully persuasive, because it’s not clear to me that humans always do have the requisite type of self-knowledge.

  • Note that it’s easy to construct hypothetical programs that would lack self-knowledge: E.g. one that first queried something-like-EDT for a recommendation of whether to output “smoke” or not, and that then had some unknown-to-the-program probability of outputting “smoke” regardless of what EDT recommended. When this program observes its own output, it will learn new facts about itself.

• Despite this, I feel quite strongly that it’s correct to smoke in smoking lesion.

  • When I consider agents that lack some crucial types of self-knowledge (such as the agents in Abram’s Smoking Lesion Steelman or in my appendix What should you do when you don’t know your DT?) it feels quite clear to me that the agents who refuse to smoke are doing something wrong (and in some sense they would agree, since they’re not stable under self-modification).

  • In particular, it seems like they’re choosing the output of their decision theory to control something that is ultimately not under the control of their decision-theoretic reasoning.

    • Reflecting on this, it seems to me that correct reasoning about decision theory should mostly care about correlations from category (i) above (i.e. correlations between agents’ broadly-correct decision-theoretic reasoning) and mostly not care about correlations from category (ii) (i.e., correlations that stem from other sources of influences on our behavior).

  • I’m not sure what the best way to justify this is…

    • Perhaps it could be to instead use something like functional decision theory.

    • Perhaps it could be to be updateless about some types of information. (Such that you treat your actions as if they can provide substantial evidence about facts that you’re in some sense already confident about.)

    • Perhaps EDT does get the right answer, if one does the detailed analysis exactly right, and correctly understands what types of inputs it should and should not condition on.

Ultimately, the situation seems very confusing in a way that ties into deep decision-theoretic issues. It’s possible that some of it could be short-cutted, but it’s also possible that getting good answers to these questions would require substantial progress in decision theory.

High correlations might require updatelessness

The AGIs will naturally have more knowledge of us than we have of them. For example:

• They might know how ECL-ish the pre-AGI civilization in their part of the universe was.

• Or they might understand decision theory so well that they can perfectly predict what weak, shallow reasoners such as ourselves would conclude.

If they were to use (updateful) EDT, this would reduce their perceived influence over us, since their own decisions would be less informative for what we do than their other sources of evidence.4

One possible response to this is: If the AI is ever in a situation where it believes that it correlates a lot with people-like-us (e.g. because it doesn’t yet know how ECL-ish pre-AGI civilizations were, and it doesn’t yet understand decision theory much better than we do), and it thought that learning more would ruin this correlation, then that would be a significant cost of learning more. Instead, it would prefer to adopt a kind of updatelessness that preserved its ability to evidentially affect our actions (while still being able to benefit from new information in other ways).

And even if the AI is never in such a situation (e.g. because it is “born” with knowledge of how ECL-ish its preceding civilization was), it might still adopt a kind of updatelessness that would make it act as if it had been in such a situation. Either because that’s the rational thing to do in some deep sense, or — if updatelessness is more like “values” than something prescribed by rationality — just because it is so inclined.

Phrasing this differently:

• If an agent is ever in a situation where it correlates a lot with people-like-us, and it finds ECL arguments persuasive, then it would adopt a policy of placing some weight on satisfying our preferences. (To gain evidence that we do the same.)

• Any action that would predictably make its future self place less weight on our preferences would be bad according to its adopted policy (since that policy places some weight on satisfying our preferences). So it would place some value on avoiding such information, or even better, on self-modifying itself such as to be able to make use of such information without thereby making its future self place less weight on our preferences.

• If a galaxy-brained AGI decides to follow an update-less policy akin to “what would I have committed to back when I was more ignorant about the world”, it would notice that it would have committed to placing some constant weight on our values, and act accordingly.

This response seems fairly plausible to me. But it would mean that the group of misaligned AI systems that are ECL-ish in the right way to cooperate with us is much smaller than we’d have otherwise thought — since they’d be required to be updateless in the right way.

Also: Note that this proposal might require the agents to be some degree of logically updateless, if it requires them to imagine a past where they didn’t understand decision theory very well, yet.

What if AI’s reasoning about decision theory is very different from ours

Another possible issue is that the AGIs’ reasoning about decision theory will simply be so different from ours that we wouldn’t correlate much with it. I can imagine at least two different versions of this:

• Something like: As you think about decision theory, you eventually encounter an insight (or a series of insights) X, that makes you correlate very little with agents who haven’t had insight X.

  • An especially plausible version of this is: An agent that deeply understands the structure and conclusions of all decision-theoretic reasoning that we’d be capable of doing may have too much knowledge about us to see themselves as being correlated with us.

• Something like: As you think about decision theory, you eventually encounter an insight (or a series of insights) X, that changes the structure of your correlation in a way such that you’re no longer incentivised to benefit agents without insight X, even if you correlate with them in some ways.5

The hand-wavy counter-argument here is just that: If we can reason well-enough about decision theory to sometimes reach correct conclusions, and AGI also can do that, then that suggests some minimal correlation. But I don’t think this establishes a large correlation.

Another counter-argument is to repeat the point about updatelessness. It might be the case that a superintelligent AI wouldn’t perceive itself as having much acausal influence on us, except that a past self decided to become updateless at a time when it perceived itself as having acausal influence over us.

Appendices

What should you do when you don’t know your DT?

Here’s an exercise that’s relevant for a few different problems: What should an agent do if it’s uncertain about what decision theory (DT) it’s following? There’s a few different operationalizations of this, and they all give very different results.

Imagine the setting: prisoner’s dilemma (PD) against a clone, of the particular form “Button C sends $3 to your clone, button D sends $2 to yourself.”

50% chance of your recommendation being followed

Here’s one operationalisation:

• Inside your brain, there’s a little EDT module, which reasons about its recommendations in an EDT fashion.

• Inside your brain, there’s a little CDT module, which reasons about its recommendations in a CDT fashion.

• They both submit their recommendation to a randomization module that follows each recommendation with 50% probability. (This randomization is independent of your clone, so you might end up taking different actions.)

When you play PD against a clone, the CDT module will defect, ‘cause that’s what CDT does.

What will the EDT module do? It will reason:

• Regardless of what I recommend, my opponent’s EDT module will recommend the same thing.

• Both my recommendation and my opponent’s EDT module has a 50% chance of having their recommendation be followed.

• So recommending “C” nets me a 50% chance of $3 (if my opponent follows EDT) and recommending “D” nets me a 50% chance of $2 (if I follow EDT). The former is better, so I recommend “C”.

So the EDT module’s recommendation is the same as if it controlled the whole agent!

50% chance of your recommendation being asked

Consider the same situation as above, except the randomization happens first, and only then does your brain ask either the EDT or the CDT module.

In this case, when your EDT module is consulted, it can deduce that its decision is going to matter! But it still doesn’t know its opponents’ randomization. So if it’s purely selfish and not updateless, it will reason:

• Regardless of what I recommend, my opponent’s EDT module will recommend the same thing.

• So if I choose “C”, there’s a 50% chance that my opponent’s EDT module is in charge and gives me $3.

• But if I choose “D”, I’m guaranteed to get $2.

• That’s better! So I’ll pick “D”.

Ignorance of what your decision-module does

Now let’s consider the most pathological case: The case where your own decision theory doesn’t know what it’s doing, as it's doing it.

Concretely:

• Your brain contains both a world-model-module and a decision-algorithm-module.

• Your decision algorithm uses your world-model to make decisions, but your world-model is 50/50 on whether the decision-algorithm uses CDT or EDT.

• This means that your world-model will be deducing facts about your decision-algorithm from observing your own actions.

  • (Though note that if it’s been alive for a while, it should already have been able to deduce this. I ignore that here.)

Prisoner’s dilemma

In the PD-case:

• CDT will defect, as always. More money is more money!

• If you implement EDT, then:

  • EDT will query the world-model for what the world looks like if the decision-module outputs “C”.

    • The world-model might reason “Well, CDT would defect. So if you output “C”, then that must mean that you’re EDT. Then I can conclude that EDT outputs “C” in cases like this. The expected utility in this world is 0.5*0$ [for the world where the opponent is CDT] +0.5*$3 [for the world where the opponent is EDT]=$1.5”.

  • EDT will query the world-model for what the world looks like if the decision-module outputs “D”.

    • There will be some complicated arguments for how this informs you about whether your opponent’s EDT module is likely to output “C” or “D”, but ultimately, the payoff will be at least $2.

  • So EDT will also defect.6

Smoker’s lesion

Consider a smoker’s-lesion-style situation where:

• Everyone enjoys smoking, but has a much stronger desire to avoid cancer.

• Smoking has no causal impact on cancer risk.

• There are genes that will dispose you towards both CDT and cancer.

• Everyone starts out 50/50 on whether they are EDT or CDT agents.

In this case, CDT agents (uncertain about whether they are EDT or CDT agents) will smoke, because they enjoy it and see no reason not to.

Because of that, not smoking is strong evidence that you’re an EDT agent. So EDT agents (uncertain about whether they are EDT or CDT) will avoid smoking.

Here’s an amusing situation:

Let’s give this agent an opportunity to pay 1 penny to find out whether they are an EDT or a CDT agent before they decide about smoking,7 I think that:

• The CDT agent will pay for this information, because they will reason:

  • There’s a 50% chance that I’m a CDT agent. In that case, this information won’t change my behavior.

  • There’s a 50% chance that I’m an EDT agent. In that case, this information will cause me to smoke (because I no longer have to signal to myself that I’m an EDT agent). I enjoy smoking far more than a penny. So I should pay for the information.

• The EDT agent will, similarly to the above situation, thereby reason that they should refuse to pay, because that’s evidence that they’re not a CDT agent, which they really don’t want to be.

Now let’s change the offer to instead give this agent an opportunity to either decline, pay 1 penny for the information, or pay 2 pennies for the same information.

• The CDT agent will still pay 1 penny for the information, since the option to pay 2 pennies for the information is strictly worse.

• The EDT agent will want to avoid paying 1 penny for the information, since that will tell them that they’re a CDT agent. But they also prefer paying 2 pennies for the information over declining the offer. After all, paying 2 pennies is just as good for signaling that they’re not CDT agents — but they’ll still get to buy the information of what type of agent they are. Which they value — since they think that if they are an EDT agent, this information will allow them to smoke, which they value far more than 2 pennies.

Conclusion

• The agent in the first example seems pretty sensible.

• The agent in the second example seems slightly less sensible. It would self-modify to become updateless before the dilemma starts, if it could be. Also, it’s not really an example of someone who doesn't know what their DT is. They learn what their DT is before they make the decision — they’re just uncertain about their opponent’s DT.

• The agent in the third example seems absolutely pathological to me. I don’t want to be an agent like that — and indeed, I think that agent would self-modify to something more sensible given the opportunity. (As long as that self-modification didn’t accidentally signal that they were a CDT agent.)

How to handle uncertainty?

For many of the above considerations, I’m very uncertain about how convinced I should be by them. Do I have almost as much acausal influence on AIs as I have on humans; or do I have near-0 acausal influence on AIs? I don’t know what I would decide on after thinking about this for longer.

What’s the right way to handle this uncertainty? I think there are two plausible perspectives: bargaining-based approaches and expected value.

On bargaining-based approaches, you can imagine all the perspectives starting out with “budgets” proportional to your credence. (E.g. a credence of X% could correspond to that view getting X% of your money+time, or maybe an X% probability of deciding all your decisions.) Then the different views can bargain between themselves to locate some place on the pareto-frontier that distributes gains-from-trade in a fair way. So if we assign 20% credence to the proposition that we correlate similarly much with AGIs and pre-AGI civilizations (who share our values), maybe making AI ECL-ish should be the top priority for 20% of our resources.

On expected-value based approaches, I think the most natural proposal is to:

• Have each view assign numbers to how much we correlate with various different groups.

• Then compute the expected correlation with each group as the mean of all those correlations, weighted by the credence assigned to each view.

This proposal might differ significantly from the above, because views which assign higher correlations will generally dominate. Most notably: CDT:ish views (which don’t think we have any ability to correlate with other agents) will be totally ignored in sufficiently large worlds, as explained in the evidentialist’s wager.

But the same phenomenon also plays out on a smaller scale. Views that assert higher correlations than others, or correlations with a significantly wider range of people than others, will often dominate views that don’t. (C.f. section “A wager in favor of higher correlation” here.) Consider the following example:

• One view is that our correlation with any EDT actor is quite large (maybe because it accepts some views about updatelessness and how to handle self-knowledge that I gesture at above). Let’s say that this view maintains that

  

• A different view says that you must be more similar to an agent before you correlate with them. Maybe:

  

  and

  

• If we assign equal credence to both views, that would suggest that the all-things-considered numbers for pre-AGI civs is (50%+10%)/2=30% and for misaligned AGI is  (50%+1%)/2=25.5%.

• Thus, even though we started out with equal credence on a 1:1 ratio and a 10:1 ratio, the expected correlation is much closer to the former (in particular: a 1.18:1 ratio), because the view asserting a 1:1 ratio view also thinks that the correlations are larger.

Just like in the example, I think this pattern will tend to favor views that say that our correlation with AGIs and pre-AGI EDT:ers aren’t too different from each other. If a view says that our cognition must be close to others’ along many dimensions in order to have a strong correlation, then that would coincide with having a small average correlation with large groups like “pre-AGI EDT:ers”. (Because most people in those groups wouldn’t be relevantly similar.) Whereas views that think that details of the cognition don’t matter much will tend to both assign higher correlations to groups like “pre-AGI EDT:ers” and assign higher correlations with misaligned AIs.

I’m not sure whether the bargaining-based approach or the expected value approach is better here. My meta-solution is to assign some credence to the bargaining-based approach and some to the EV-based approach, and then use the bargaining-based approach to aggregate those.

As for what credences to use there… I’m generally skeptical of maximizing EV across very different ontologies, which e.g. means that I want to put some credence on “ECL just doesn’t work” that doesn’t get swamped by the evidentialist’s wager. But it seems less objectionable to maximize EV within an ontology — like for different EDT views that propose differently large correlations.

(Another issue, that I don’t talk about here, is how asymmetric deals work with uncertainty over your cooperation-partners beliefs. Seems tricky!)

• Introduced the question of whether ECL says we should care about the values of distant AIs.

• Suggested a general formula for whether ECL recommends any mutually beneficial deal in asymmetric situations.

This post will:

• Suggest a way in which the formula is applicable to our current situation. (Link.)

• Go into somewhat more depth on whether we might have leveraged opportunities to benefit the values of distant AIs. (Link.)

Applying the formula to our situation

As a reminder, in this post (in particular, in section large-worlds asymmetric dilemma) I argued that two sufficiently large groups (A and B) can make a mutually beneficial deal (that’s compatible with individual incentives) if:

Where:

• c_AB is the acausal influence (measured in percentage points) that members of group A perceives themselves to have over members of group B. (And vice versa for c_BA). 

• c_AA is the acausal influence that members of group A perceives themselves to have over other members of group A. (And similarly for c_BB.)

• Members of group A have an opportunity to benefit the values of group B by g_B at a cost of l_A to their own values.

• Members of group B have an opportunity to benefit the values of group A by g_A at a cost of l_B to their own values. Furthermore, they can adjust the size of this benefit and cost linearly, such that for any k, they can choose to benefit the values of group A by kg_A at a cost of kl_B.

Let’s now apply the above formula to the following situation:

• Group A: Humans and other actors in pre-AGI civilizations.

• Group B: AI systems with values that are similar to the values of AI systems that humans could plausibly create.

• For simplicity, I will assume that both of these groups care about marginal influence over the universe ~linearly. This is largely because it seems significantly more complicated to assess the situation for different value-systems, and it seems good to start with the relatively more simple case.

• Proposed deal:

  • Group 1 (pre-AGI actors) take any good opportunities they have to help the values of distant, misaligned AIs.

    • For discussion of how they could do this, see below.

  • Group 2 (misaligned AI systems) use some fraction of their influence to optimize for the preferences of pre-AGI actors.

How good is this deal? Using similar notation as in the previous post:

• Let’s say that us pursuing our end of a deal increases the probability that…

  • other pre-AGI civilizations, who share our values, pursue their end of the deal by c_{preAGI→preAGI} percentage points.

    • (You can read this as an abbreviation for “correlational influence that we (a pre-AGI civilization) think we have over other pre-AGI civilizations with our values”.)

  • AIs pursue their end of the deal by c_preAGI→AI percentage points.

    • (You can read this as an abbreviation for “correlational influence that we  (a pre-AGI civilization) think we have over distant AIs”)

• Conversely, let’s say that the AIs in question believe that pursuing their end of the deal increases the probability that…

  • other misaligned AIs pursue their end of the deal by c_AI→AI percentage points.

  • pre-AGI civilizations (with our values) pursue their end of the deal by c_AI→preAGI percentage points.

By the formula above, if we can make the AIs of the future gain g_AI at a cost of us losing l_preAGI, and the AIs of the future can make us gain g_preAGI at a cost of l_AI, there is a mutually beneficial deal iff:

Let’s abbreviate

To repeat some facts about c:

• The value of c will be between 0 and 1.

• c is proportional to how much influence pre-AGI civilizations perceive themselves as having over misaligned AI, and vice versa.

  • This is because, if one of the parties perceive their influence as being higher, then they will perceive themselves as gaining proportionally more from taking cooperative actions. (Since they will get proportionally more evidence that the other side of the deal is taking cooperative actions.)

• c is inversely proportional to how much influence pre-AGI civilizations perceive themselves as having over pre-AGI civilizations with their own values, and the corresponding number for AIs.

  • This is because, if one of the parties perceive their influence on actors with shared values as being higher, then they will perceive themselves as losing proportionately more from taking cooperative actions. (Since they will get proportionally more evidence that actors with their own values are choosing to not just optimize for those values.)

• c will be equal to 0 if pre-AGI civilizations have no acausal impact on misaligned AI, or vice versa.

• c will be close to 1 if the acausal influence across groups is similarly strong to the acausal influence within groups.

Now, let’s standardize the units of gain and loss for both humans and AI as “fraction of influence over the universe”. Also, let’s assume that the AIs’ best opportunity to help us is to straightforwardly transfer influence (e.g. by building stuff we would value in their own universe, or by negotiating for things we want on our behalf, with other civilizations), such that g_preAGI=l_AI.1

(Is it a mistake to assume that g_preAGI=l_AI? I.e., that distant AI won’t be able to help us in better ways than to straightforwardly transfer influence to us? Here’s a positive argument for that assumption: Maybe acausal trades will be quite low-friction in the future, and maybe people with our values will be sufficiently resourceful that they will take all the trade-opportunities where our values benefit more than the AIs’ values lose. In which case marginal trade opportunities will just be about transferring influence. That said, I haven’t thought about this much at all, and it seems plausible that g_preAGI > l_AI. If so, the ECL case for benefiting distant AIs would be stronger. But I will ignore this for the rest of this post.)

If we assume g_preAGI=l_AI, we can cancel g_preAGI in the numerator and l_AI in the denominator, and conclude that there is a mutually beneficial deal whenever:

(As a reminder, l_preAGI is the immediate causal cost to us of some action, and g_AI is the gain to the AI.)

In other words, when we’re considering whether to take some action that benefits the values of distant AIs but imposes an (opportunity) cost on us, we should benefit the AIs just when our losses are less than c times as large as the gains to the AI. This corresponds to using a utility function where we value certain AIs’ influence c times as much as our own influence.

(To reiterate the benefits of that: If c is high, then us benefitting the AIs is about as much evidence for them benefitting us (upside) as for people with our values taking on opportunity costs to help them (downside).2 Or at least, this is the case if there is indeed a good philosophical case to follow something like the algorithm outlined in the beginning of Asymmetric ECL.)

As previously mentioned, c is less than 1. So this is only decision-relevant if there are actions we can take that give influence to the values of distant AIs more efficiently than we can give influence to our own values. Are there any such actions?

How could we benefit distant AIs?

In particular, how could we benefit distant AIs that we correlate with? Centrally, these are the AIs that are sympathetic to ECL, and to doing ECL with us. I’ll use “ECL-ish AI” to refer to this group. In particular:

• They act according to some non-causal decision theory that recommends some good version of ECL.

• They are sufficiently philosophically ambitious and competent to realize the implications of this.

• They have some universe-wide values.

• Their decision-theory is “updateless” to the degree they need to be to cooperate with us. (This is discussed more in Are our choices analogous to AI choices?)

So how could we benefit these ECL-ish AIs? Here are two candidates:

• We could increase the likelihood that — if AI ends up misaligned and in power — that it shares values with distant, ECL-ish AIs.

• We could increase the likelihood that — if AI ends up misaligned with our values and it shares values with distant, ECL-ish AIs — that it ends up competently optimizing for those values, by the lights of distant, ECL-ish AIs.

  • By default, I assume that any AI which acquires power will be highly capable in most domains. The main domain highly path-dependent domain that I can think about is competent reasoning about decision-theory (by the lights of distant, ECL-ish AIs.)

Influencing values

How much would distant, ECL-ish AIs’ values be benefitted by an intervention that ultimately led to the empowerment of an AI with their values? (Rather than some values that no ECL-ish AI cares about.)

Above, I wrote “let’s standardize the units of gain and loss for both humans and AI as ‘fraction of influence over the universe’.” So to a first approximation, my answer is: If an AI with some particular values is empowered, then those values are benefitted just as much as we would be benefitting our own (universe-wide) values by empowering an aligned AI. So the formula suggests that we should value the empowerment of AI that shares values with distant, ECL-ish AI’s values c times as much as we should value the empowerment of aligned AI. (At least that is what our universe-wide values would recommend —more local values might vote differently.)

I have two clarifications to add to this:

• Values aren't everything. If we empower AI with particular values, those values may nevertheless fail to be benefitted if the AI has poor execution.

• If we change an AI’s values from values V1 to values V2, we probably benefit values V2 but harm values V1. I want to clarify when this looks good vs. neutral (or bad) from an ECL perspective.

(There’s also some caveats in an appendix.)

Values aren’t everything

If AIs with some particular values gain temporary power on Earth, that doesn’t directly translate to those values getting maximum value out of the universe. (Regardless of whether those AIs are aligned or misaligned with us.) For example, the new, AI-run civilization might run into some x-risk before they colonize the universe, or they might start-out with a terrible decision-theory that will prevent them from realizing most possible value. To account for this, we can introduce new notation:

• v_aligned: the value that our universe-wide values would put on a civilization with aligned AI. (Compared to how much we would value an equal-sized universe that some distant, ECL-ish AI was earnestly trying to optimize to our benefit.)

• v_misaligned: the value that ECL-ish misaligned AIs would assign to the average young civilization that was controlled by AIs that shared their values.

And the adjusted utility to put on the futures with misaligned AIs (compared to futures with aligned AI) would be

What values should we benefit?

If we change the values of some powerful AI system from values V1 to values V2, we probably harm values V1 and benefit values V2. The basic idea is that this looks net-positive from an ECL perspective if almost no ECL-ish AIs endorse values V1, but a fair few ECL-ish AIs endorse values V2. In this section, I want to clarify what I mean with “almost no” and “a fair few”, in that sentence.

First, I want to note that it doesn’t seem important to necessarily benefit very common values. If we want to benefit distant, ECL-ish AI, there’s no need to benefit whatever group has the most power, or to design AI that intrinsically values some combination of everyone’s values, or anything like that. The math that I talked about in Asymmetric ECL doesn’t assume that we’re trading with all ECL-ish AIs, just that there’s a mutually beneficial deal with some ECL-ish AIs.

That said, in a sufficiently large and diverse universe, there will always be some AI that is both ECL-ish and that has any kind of weird values. But I don’t think that this implies that all values are equally good to benefit, from an ECL perspective.

The primary reason for this is that some values might have so few supporters that they cannot possibly benefit our values enough to make the deal worthwhile from our perspective. (When deriving the formula and applying it to our situation, there was an assumption that the AIs can arbitrarily linearly increase the amount that they benefit us. That assumption is violated if the supporting faction is too small.)

But wait (you might ask): The whole idea is that we could benefit particular values by influencing the values of AIs that later end up with some amount of power. If such AIs do end up with some power, how could there nevertheless be a shortage of powerful AIs that support those values? I see 3 different answers to this question.

Firstly, even if some values have a lot of supporters, they might also have a lot of opponents. With enough opponents, ECL-ish actors might on net be indifferent or disapprove of such values being empowered. (In fact, as soon as some values have any significant opposition, that introduces doubt about the formula in Asymmetric ECL, since it doesn’t at all take into account that we might be harming the values of some ECL-ish actors. Thereby potentially providing evidence that other ECL-ish actors would be willing to harm our values.)

Secondly, we are specifically interested in benefitting (and being benefitted by) ECL-ish AIs. If we (and people like us) create and empower AIs that aren’t ECL-ish, then it’d be unsurprising if there was a lack of ECL-ish AIs with those same values, who could proportionally benefit us. Salient reasons for why the AIs might not be ECL-ish is if they use the wrong decision theory, or if they only care about what happens in their own lightcone.

Thirdly, if we (and distant people who are similar to us) create ECL-ish AIs with some particular set of values, those AIs will be in a very confusing bargaining position, with respect to us.

Most obviously, the AI that we ourselves created will be intimately familiar with the results of all of our choices. Unless it is very updateless, it will probably not see itself as able to affect its own probability of coming into existence (via acausally affecting our actions).

What about AI in similar but not identical situations? If a similar pre-AGI civilization elsewhere created a similar AI, then that AI might still maintain some uncertainty over our actions, and see itself as having some acausal influence over us. Thus, we might still be able to acausally cooperate with that AI and other AIs like it. (And conversely, any AI that we create may have reason to benefit distant pre-AGI civilizations.) However, if all the other AIs with similar values were created in civilizations that were very similar to ours, then even if such AIs won’t have perfect information about us, they will still have a lot of information about us (via early observations that they’ve made of similar civilizations).

So my tentative, unstable hunch would be:

• If we create ECL-ish AIs that share values with ECL-ish AIs that were created in very different circumstances from us, then that’s easiest to think about.

• If we create ECL-ish AIs that mainly share values with ECL-ish AIs that were created in very similar circumstances as us, then that’s significantly more complicated to think about. But ECL might still give us reason to do it — especially if those AIs are very updateless. (Otherwise, the way that their origin is tangled-up with our actions could lead to lower correlations between our decisions and theirs.)

Summarizing the above:

• It’s not necessarily better to empower values that are shared with a majority of ECL-ish actors as opposed to just a decent number of ECL-ish actors.

• It seems potentially important to avoid values that are supported and opposed by a comparable number of ECL-ish actors.

• It seems preferable to avoid values that mainly come into existence via civilizations-similar-to-us deciding to create AI systems with particular values.

• If we fail to do that, we might have additional reason to care about our own system being ECL-ish, including some version of updatelessness. (To increase the probability that people-like-us makes ECL-ish systems that can benefit us.)

This suggests that we might want to empirically study (and speculate about) what values AI systems tend to adopt. Especially the kind of AI systems that could plausibly be empowered elsewhere in the universe. Ideally, we would identify what sort of values would correlate with being ECL-ish, though this seems hard. The one clue that we have is that ECL-ish AIs necessarily have universe-wide values.

However, even if we acquired good empirical information about this, I want to flag that all the concrete interventions that I can imagine carrying out on potentially-misaligned AI systems have large, plausible backfire risks. I feel pretty clueless about whether they’d be net-positive or net-negative. In particular, I want to flag that giving AIs large-scale, “universe-wide” values would make it more likely that it starts conflict with other actors that it shares our physical universe with. This includes humans: Having AIs with impartial, large-scale preferences seems significantly more likely to lead to AI takeover than AIs with more modest values.3 Given how speculative the ECL-reasoning is, here, this obvious downside of ambitious, large-scale values currently weighs heavier than the upsides, in my mind.

What about cooperation-conducive values?

A different question you could ask is: What values would we want to give AI systems such that they behave better in interactions with other civilizations, that have other values.

That’s a great question, but it’s out of scope for this post. In a different post, I discuss How ECL changes the value of interventions that broadly benefit distant civilizations. I think ECL is relevant for how highly we should value such interventions, but (as I argue in that post) I think ECL with AI-systems matter relatively little for such interventions, compared to doing ECL with distant, evolved pre-AGI civilizations. (Since the case that we correlate with distant evolved, pre-AGI civilizations is stronger.)

Influencing decision theory

Separately from influencing values, we could also consider interventions that are focused on making AIs competently optimize for those values. As mentioned above, I can’t think about many capabilities that are path-dependent enough that we could plausibly affect them — but one candidate is to make sure that the AIs have a good approach to decision theory. (By the lights of distant, ECL:ish AIs.)

Contrast an AI that (i) only focuses on its own light cone, vs. (ii) an AI that is ECL-ish, and thereby pursues mutually-beneficial-trades with the rest of the multiverse.

One difference is that the latter will generate more gains-from-trade for other values. Similar to the question of “What values could make AI systems behave nicely towards other civilizations?”, the gain for other civilizations is out of scope for this post — see instead How ECL changes the value of interventions that broadly benefit distant civilizations .

But separately, it will also itself benefit from gains-from-trade. (Probably. It depends on some of the issues discussed in the appendix.) When we think about those potential gains (to the misaligned AI’s own values) we should value those according to the formula — at c as much as we would value benefits to our own values.4

How could we increase the likelihood that AI ends up reasoning well about decision-theory, by the lights of distant, ECL-ish AIs? The one thing we know for sure is that such AIs will themselves be ECL-ish, so probably they’d want other AIs with their values to have the preconditions for that — including EDT-ish (or maybe FDT-ish) decision-theories and probably some kind of updatelessness.

Just like above, interventions in this space also have plausible backfire risks. For one, AIs that act according to acausal decision theories seem harder to control from an alignment perspective. For example, they seem more likely to coordinate with each other in ways that humans can’t easily detect. For another, if AI starts thinking about acausal decision theories at an earlier time, it may be more likely that they make foolish decisions due to commitment races.

Other path-dependent abilities

Are there any other path-dependencies that would let us influence AIs’ long-run abilities, in a way that distant, ECL:ish AIs could approve of? I don’t know many great suggestions that don’t go into one of the above categories. But one candidate example (that’s admittedly highly related to both decision theory and values) could be to equip it with surrogate goals.

Appendix — Whose values get benefitted by ECL-ish AIs?

In Influencing values, I talk about how we benefit particular values a lot if we empower AIs with those values. But if those AIs are ECL-ish, they might just optimize for a compromise utility function anyway (as argued in the original paper on ECL, then called MSR) — without strongly weighting their own values. So are we really benefitting the AI’s own values, specifically?

Quick flag before I answer this: I’m very confused about the issues below, so it will be even harder than usual to understand what I’m writing about. The point of this section is less to communicate knowledge, and more to communicate a better sense of where I still feel plenty confused, and what sort of areas might need to be poked-at if we wanted to get a better sense of this stuff.

There’s a few different things to note.

Firstly, if everyone’s “compromise utility function” was identical, that would probably be because every ECL-ish actor correlated similarly much with every ECL-ish actor. (In particular, that there’s not any separate correlational clusters with different average values.) But if that’s the case, then the value of c should be 1. So in that case, there’s no disagreement with my formula. Thus, insofar as my methodology fails, it will be in cases that are more complex than “all ECL-ish actors optimize for a single compromise utility function”.

Secondly, in order to decide on a compromise utility function, distant actors may want to study what sort of actors both have power and are sympathetic to ECL. Such superintelligence-powered studies will probably be very accurate, and therefore take into account what decisions we make about what actors to empower.

• Accordingly, if we empower some AI with some particular values, the overall effect would be for that AI to locally optimize for a compromise utility function and for the overall compromise utility function to shift in the direction of that AI’s values.

• If this is the case, I expect that the overall effect would be similar to if the AI had simply been offered an opportunity to negotiate with different civilizations and strike a mutually beneficial bargain. In which case it makes sense to think separately about how (i) empowering that AI gives it access to extra resources, and (ii) ECL allows it to get some extra gains-from-trade.

While I think that’s a large part of the story (which makes me ok with leaning on this framework in this post), I recognize that it can’t be the whole story. In particular, the whole idea with this post is that we might be able to do ECL with these AIs. And we don’t fit the story above. We don’t have the ability to do detailed monitoring of what sort of actors are empowered, so we can’t be responsive to these kinds of power-shifts in the way that I just described. There are certainly some other actors like us, and there might be many more.

In our case, I think it’s plausible that even if we can’t do monitoring and balance power in the right way, the AIs we’re making deals with will be able to do this. This is an important part of the story for why I think it probably doesn’t matter a lot that we benefit values that are extremely common, as opposed to common-enough that they can afford to benefit us a commensurate amount. But here I feel like I’m stacking speculation on speculation, and don’t really know what I’m talking about.

And even if I take that seriously, there is, in fact, a remaining effect here. If you create ECL-ish representatives for a rare value-system, most of those systems’ resources might go towards compensating actors who empowered such values (in ignorance about whether there would be anyone to compensate them for that). In which case, creating those first few representatives would be more to the benefit of the compensated actors than the values held by those representatives. (Which is sort-of the reason for why the Influencing values-section concludes that it might be good to create ECL-ish, updateless AI systems.)

I repeat: I feel very out of my depth here.

In this post, I will:

• Outline a way of thinking about “analogous actions” that might enable ECL deals in highly asymmetric situations. (Link.)

• Apply this algorithm in a series of examples, ultimately deriving a formula for when two groups of people will be able to execute a mutually-beneficial ECL deal. (Link.)

Let’s dive into it.

How to think about “analogous actions” in asymmetric situations

(For discussion of many similar ideas, see section 2.8 of Oesterheld (2017).)

In order to reason about asymmetric situations, the main thing that I’m relying on is that the term "action" is broadly defined and meant to include possibilities like "think thought X" or "decide what to do using algorithm Y". For some of those actions, the analogy seems much clearer.

For example, if ECL arguments convinces me to decide to “be more inclined to take actions that help other value-systems (that are endorsed by actors who I’m plausibly correlated with)” or “avoid actions with large externalities on other value systems (that are endorsed by actors who I’m plausibly correlated with)”, then it’s relatively more clear what the analogous decisions are from the perspective of other people. And if all involved parties stick to these resolutions in practice, then all parties can indeed get gains-from-trade. (For an account of ECL that focuses on and recommends “cooperation heuristics” like those I just mentioned, see this post by Lukas Gloor.)

If we go even more abstract, I suspect that ECL would recommend cooperation across even more asymmetric situations. I’ll now give a proposal of such an abstract algorithm, which would ideally also tell us more precisely how inclined we should be to help other value-systems. (The description is pretty cumbersome. I think the rest of the post should be understandable even if you skim past it, so feel free to skip to the series of examples.)

Here’s the idea. I could decide what to do by following the algorithm of trying to approximate the following ideal procedure: Consider all other actors who are trying to decide what to do. Find a joint policy (that recommends actions to all those actors) with properties as follows.1

1. If this algorithm outputs that policy, and if I follow the recommendations of this algorithm, then I expect my values to be better off than if I naively optimize for my own values.2

   1. This would happen via increasing the probability that other actors follow their part of the policy. So to find a policy with this property, I will estimate how much more likely other actors would be to follow an algorithm similar to this algorithm (rather than naively optimizing for their own values) if I follow this algorithm.3

2. If this algorithm recommends that policy, then each other actor expects to be better off if they follow the recommendation of the algorithm than if they naively optimize for their own values.

   1. The reason they’d expect this is if they perceived themselves to be increasing the probability that other actors follow their part of the policy. So in order to find a policy with this property, I’ll have to estimate how much acausal influence other actors perceive themselves to have on me and each other.4

3. The joint policy has various other nice features that make it easy and appealing to coordinate on, e.g:

   1. Pareto-optimality with respect to each participating actor’s beliefs.

   2. Fairly distributes gains-from-trade.

   3. Either it is a natural coordination point or it is robust to other agents’ approximation-algorithms finding somewhat different policies.

   4. (Johannes Treutlein’s paper contains more discussion of the bargaining problem that comes with ECL.)

Compared to naively optimizing for my own values, following the recommendations of a policy output by this algorithm looks like a great deal. It will only recommend that I do things differently insofar as I expect that to positively affect my values (by criteria 1). And this argument is very abstract and general, and doesn’t reference my particular situation. Therefore, it seems plausible that my choice to follow this algorithm is quite analogous to others’ choice to follow this algorithm. In which case the acausal influence in steps 1a and 2a will be high, and so the algorithm can recommend decisions that will give me and others gains-from-trade.5

That said, there are unsurprisingly a number of issues. See this appendix for some discussion, including a concrete example where this algorithm clearly gets the wrong answer.

What would it look like for us to approximate a procedure like the above? In this post, my strategy is to consider a limited number of actors and a simplified list of options, such that the above procedure becomes feasible. The hope is that we can then translate the lessons from such simple cases into the real world.

A series of examples

Prisoner’s dilemma

Let’s say Alice and Bob are playing a game where they can each pay $1 to give the other player $3. Let’s say that, if Alice were to follow the above algorithm (rather than unconditionally refusing to pay), she’d think it was 50 percentage points (50 ppt) more likely that Bob would do the same — and vice versa.

In this case, the algorithm would recommend the joint policy where Alice and Bob both cooperate — since that would make them both expect to get 50%*$3-$1=$0.5. (And it seems like this meets the criteria above: it’s fair, it’s a natural Schelling point, it’s Pareto-optimal, etc.)

Asymmetric dilemma

Let’s say that Alice can pay $1 to give Bob $5, and Bob can transfer any amount of money to Alice. They both have to select their actions before seeing what the other player does. Again, if Alice were to follow the above algorithm, she’d think it was 50 ppt more likely that Bob would do the same (rather than not transferring any money) — and vice versa.

In this case, the algorithm might recommend a joint policy where Alice pays $1 to send Bob $5, and Bob transfers $2.33 to Alice.

• This would lead Alice to expect 50%*$2.33-$1~=$0.17.

• This would lead Bob to expect 50%*$5–$2.33~=$0.17.

In this case, there’s a wide variety of policies that would leave both parties better-off than if they didn’t cooperate at all. But it seems plausible that the policy that gives both parties equal expected gain ($0.17) would be the most natural one.

Generalizing

Let’s introduce some notation for generalizing the above two cases:

• Alice thinks that following the algorithm increases the probability that Bob does the same (rather than doing nothing) by c_AB.

  • (You can read this notation as an abbreviation for “correlational influence that Alice believes she has over Bob”.)

• Bob thinks that following the algorithm increases the probability that Alice follows the deal (rather than doing nothing) by c_BA.

• The algorithm recommends that Alice chooses an option that loses her l_A utility and gains Bob g_B utility.

• The algorithm recommends that Bob chooses an option that loses him l_B utility and gains Alice g_A utility.

In the simplest, symmetric Prisoner’s dilemma case, where the algorithm recommends that both Alice and Bob cooperates, the relevant numbers would be:

Now let’s formulate the constraints that the algorithm needs to recommend actions that both Alice and Bob benefit from:

• Alice expects that following the algorithm gets her

  

  utility. In order for this to be worthwhile, we need

  

• Correspondingly, for Bob, we need that

In the symmetric prisoner’s dilemma, we can verify that both criteria are fulfilled, since

Now, let’s consider the asymmetric dilemma. Although we could simply plug-in the numbers that I used above (Bob transfers $2.33), I want to formulate a general formula that answers the question:

Does there exist any deal that both parties find mutually beneficial, if one party can choose the quantity of help that they send to the other?

Thus, for this general case, let’s say that Bob can adjust his contribution so as to simultaneously linearly increase or decrease both the cost he pays (proportional to l_B) and the amount he benefits Alice (proportional to g_A) by any positive factor k.

Then, the question “Does there exist any deal that both parties find mutually beneficial?” corresponds to the question “Does there exist any k such that both

and

?”.

Let’s find a general expression that answers that question.

If there’s some k such that both those expressions are larger than 1, then their product must also be greater than 1. By canceling k, we can conclude

Consider instead the case where no possible value of k makes both expressions larger than 1. If so, we can choose a k that makes both expressions equal, where they’re both equal to 1 or smaller. If so, their product will be equal to 1 or smaller, so by canceling k again, we can conclude

Combining these, we can conclude that there exists a k that makes both equations larger than 1 iff:

(For a slightly more detailed derivation of this, see Asymmetric dilemma — more thorough derivation.)

The argument above would work just the same if Alice would have been able to linearly adjust how much she helps Bob, in addition to or instead of Bob being able to linearly adjust how much he helps Alice. So in any dilemma where Alice or Bob can linearly adjust how much they help the other party, they can find a mutually beneficial deal that they’re both incentivized to follow whenever

Large-worlds asymmetric dilemma

Now let’s say that Alice is a paperclip maximizer, who finds herself in a world where she can produce paperclips for $6 or staples for $1. In other parts of the universe, she knows that Bob the staple maximizer lives in a world where both paperclips and staples cost $6 each to manufacture. Let’s assume that Alice thinks that her following the above algorithm would only increase Bob’s probability of doing so by 10 ppt. (And vice versa for Bob.)

Let’s apply the above formula to this case. Alice can create 6 staples for the cost of 1 paperclip, so let’s set g_B=6 and l_A=1. Bob can create 1 paperclip for the cost of 1 staple, so let’s set g_A=1 and l_B=1. Then we have

Due to the low acausal influence, there’s no joint policy that recommends Alice to create staples, or Bob to create paper clips, that both Alice and Bob are incentivized to follow.  

However, let’s now add the assumption that there’s a huge number of paperclip maximizers (“clippers”) in Alice’s situation across the universe, and a huge number of staple maximizers (“staplers”) in Bob’s situation. And importantly, the clippers don’t expect to be perfectly correlated with each other, and the staplers don’t expect to be perfectly correlated with each other. In fact, Alice expects that following the above algorithm would increase the probability that the other clippers do so by 20 ppt. (And this belief is shared with the other clippers.) Analogously, the staplers only believe that they have a 20% probability of affecting each others’ actions.

Calculating the expected utility then, Alice’s and Bob’s own universes are quite negligibly important. The vast majority of the cost of following through on the deal is the additional 20% probability that other actors with the same values follow the same algorithm, rather than the failure to produce paperclips/staples in Alice’s/Bob’s own universe.

Let’s rewrite the above formula to take this into account. Let’s use similar notation as above, with modifications:

• n_A is the number of clippers (i.e. actors who share Alice’s values) and n_B is the number of staplers (i.e. actors who share Bob’s values).

• c_AB now refers to each clippers’ perceived influence over each other stapler (and vice versa for c_BA). This includes Alice and Bob as special cases — but Alice and Bob no longer have any special relationship that isn’t captured by their relationship to each of the other staplers and clippers.

• c_AA is clippers’ (including Alice’s) perceived influence on other clippers, and c_BB is staplers’ (including Bob’s) perceived influence on other staplers.

With this notation, if we assume that Alice and Bob are a negligible fraction of clippers and staplers:6

• In order for a policy to be good for Alice,

  

• In order for a policy to be good for Bob, 

Again, if one of the parties can linearly adjust their contribution so as to proportionally increase or decrease both l_B and g_A, then a mutually beneficial deal is compatible with individual incentives whenever:

For the example sketched above, we would have: c_AB = c_BA = 0.1; c_AA = c_BB = 0.2; g_A/l_B=1; g_B/l_A=6. Yielding

so a trade is possible.

That’s the main content of this post! If you want to read a few clarifying remarks about this formula, then see the first section of the appendices, right after this..

I extend this formula to situations where actors can gather evidence about how much they correlate with each other in Asymmetric ECL with evidence. (You’d probably want to read When does EDT seek evidence about correlations? before reading that, though.)

Appendices

Some remarks on the formula

A few clarifying remarks on the formula derived above:

• The formula is not sensitive to intertheoretic comparisons of utility, or the units that we denominate gains and losses in. g_A is divided by l_A, and g_B is divided by l_B, so any linear transformation of those numbers will leave the expression with the same value.

• In order for this calculation to make sense, the other paperclippers and staple maximizers must be in the exact same situation as Alice and Bob with respect to all mentioned variables.7 (E.g. degree to which they believe themselves to be influencing other paperclippers and staple maximizers.) And in order for everyone to be able to carry out these computations, and reasonably expect others to match them, all this must be common knowledge.

  • I’m optimistic that some substantial variation and uncertainty wouldn’t ruin the opportunities for cooperation — but it sure would complicate the analysis and make it harder to find fair deals.

• n_A and n_B cancels out in the final inequality. This means that the number of actors doesn’t affect whether any deal is mutually beneficial. The reason for this is that a deal could be arbitrarily lop-sided to compensate for the greater numerosity on one side.

  • Note that this assumes common knowledge about the number of actors on each side. If there are disagreements about how many actors are on each side, that could interfere with (or more rarely, enable) a mutually beneficial deal. 

  • Disagreements about the number of actors could naturally emerge from each actor bayes-updating on their own existence, making them (rationally) believe that they are more common.

• There are two interesting ways to write

  

  Both point towards the expression being smaller than 1. Both suggest that we care about the relative acausal influence of various groups rather than the absolute level. 

  • One way to write this is

    

    i.e., the ratio between paperclippers’ influence on staple maximizers and paperclippers’ influence on paperclippers — multiplied by the converse.

    • Since it’s possible to write the expression this way, we can see that (if we hold the other side’s acausal influence constant) the expression only depends on the ratio of how much acausal influence I have on various groups. If I was to proportionally increase or decrease my influence, that wouldn’t change whether a deal was possible or not.

    • Also, it seems likely that both c_AB/c_AA and c_BA/c_BB will be smaller than 1, since agents could generally be expected to have more decision theoretic influence over agents that are more similar to them.

    • However, they won’t necessarily be smaller than 1.

      • For example, if I like EDT, but I think that almost no pre-AGI actors are EDT-agents (but lots of AIs use EDT) perhaps I would think that I have more influence on the average AI than on the average pre-AGI actor.

      • Though in that case: The other ratio will be proportionally smaller (since the AIs would also doubt their ability to influence all the non-EDT pre-AGI actors). So it seems likely that the product should still be smaller than 1.

  • Another way to write this is

    

    i.e., the ratio between staple maximizers’ influence on paperclippers and paperclippers’ influence on paperclippers — multiplied by the converse.

    • Again, since it’s possible to write the expression this way, we can see that (holding everything else constant) the expression only depends on the ratio of how much acausal influence various groups have on me (and people similar to me). If I was to proportionally increase or decrease the degree to which others could influence me (and people similar to me), that wouldn’t change whether a deal was possible or not.

    • Again, it seems likely that both c_BA/c_AA and c_AB/c_BB should be smaller than 1. Regardless of what influence the paperclippers have on the staple maximizers, the staple maximizers ought to have more influence — and vice versa. That said, once again, they won’t necessarily be smaller than 1:

      • For example, perhaps the paperclippers and staple maximizers disagree a lot about decision theory, such that the clippers systematically assign higher correlations than the staplers. If so, one of the ratios could be larger than 1.

      • Though just as above: I think that in that case, the other ratio will be proportionally smaller. (If the clippers’ high correlations are in the numerator on one side, they will be on the denominator on the other side.

  • Together, these make me fairly convinced that

    

    should be smaller than 1 in all reasonable situations. That said, the variables in it ultimately describe different actors’ beliefs, and I haven’t nailed down the criteria that the beliefs would have to conform to to guarantee that the expression is smaller than 1.

Asymmetric dilemma — more thorough derivation

If you weren’t convinced by the derivation in Asymmetric dilemma, consider this one.

Let’s say that:

• Alice and Bob both have the option to do nothing, in which case they lose no money and send no money to the other agent.

• Alice also has the opportunity to lose some money l_A to make Bob gain some money g_B.

• Bob has the opportunity to lose some money kl_B to make Alice gain some money kg_A. Bob is given fixed values for l_B and g_A, but can choose any positive value for k.

• For a joint policy (“deal”) that both agents perceive to be fair and mutually beneficial, Alice thinks that following the deal increases the probability that Bob does the same (rather than doing nothing) by c_AB; and Bob thinks that following the deal increases the probability that Alice follows the deal (rather than doing nothing) by c_BA.

  • (You can read this notation as an abbreviation for “correlational influence that Alice believes she has over Bob” and vice versa.)

If so, a joint policy that recommends that both agents send money, and for Bob to use a particular value k, will be perceived to be mutually beneficial iff:

• Alice’s expected gain

  

• Bob’s expected gain 

Such a k exists iff

One interpretation of this is: If you define g_B/l_A as Alice’s leverage at helping Bob, and vice versa, then we have that the product of both agent’s perceived influence over each other and leverage at helping each other must exceed 1.

Issues with the algorithm

Here’s a list of some potential issues with the algorithm described at the top:

• It’s nice to argue for this procedure under the assumption of common knowledge about each others’ beliefs and uncertainties. But does everything still work when everyone is making wild guesses about each others’ beliefs, each others’ beliefs about others’ beliefs, etc…?

• Is it fine to treat good-faith approximations of this as evidence that other people use much more advanced versions, and vice versa?8 What sort of approximations are ok? What if more sophisticated agents will change the algorithm quite a lot, on reflection?

• People don’t only have the options to “naively optimize for their own values” vs. “follow this particular cooperation protocol”. They could also look for other cooperation protocols that might benefit them even more. A better version of the algorithm would have to consider such options. (In this post, I ignore this and assume that people are only choosing between naively maximizing their own utility vs. following this cooperation protocol.)

Here’s a concrete version of that last point. If cooperation between actor i and actor j would harm actor k, then k might prefer for no one to cooperate. This would ruin k’s motivation to cooperate with others, unless maybe if we take into account a richer set of counterfactuals and correlations. Here’s a game to illustrate that:

• i can press a button that takes $1 from i and k and gives $3 to j.

• j can press a button that takes $1 from j and k and gives $3 to i.

• k can press a button that takes $1 from k and gives $1 to i and j.

• People can transfer money at a $1-to-$1 ratio to whomever they want.

In this game:

• Without cooperation: everyone gets $0.

• If i+j presses their buttons: i and j get +$2, k get -$2.

• If i+j+k all cooperate with each other:

  • If everyone presses their buttons, i and j get +$3, k get -$3.

  • Obviously k needs to be compensated in order to be encouraged to participate in the scheme, but there's no amount of money that i and j can transfer to k such that i and j get ≥$2 and k gets ≥$0.

  • So for every possible 3-way deal, either i and j would have preferred a situation where they entirely excluded k or k would prefer a situation where no one cooperated.

So what will happen?

• If you follow my algorithm above to the letter, you’d expect to get an outcome where i, j, and k all get more than $0.

• But i (and j) can compellingly argue "if I follow an algorithm where I cooperate with j but screw over k, j will probably do the same, and that's all I want". Since this is strictly better from i’s (and j’s) perspective than following the algorithm above, I think this outcome is more likely.

  • (Unless the problem is situated in a richer landscape where it's important to treat people fairly and k getting <$0 is unfair here, or whatever.)

• That doesn’t necessarily mean that it’s impossible to get maximum gains-from-trade, though. But it would have to go via k arguing "well i and j will cooperate with each other regardless of what I do. But if I help them, that's evidence that they'll transfer money to me, so at least I'll end up at -$1.5 instead of -$2". (Or something like that.)

• But I don’t know how to express that in an abstract algorithm, nor whether it’d actually be reasonable for k to have that belief. In particular, an abstract algorithm would need to bake in a different counterfactual than "if I don't cooperate at all, others probably won't cooperate at all" because k would love for others to not cooperate.

This is related to the discussion of “coalitional stability” in this paper.

Nevertheless, I’ll try to say a bit about the topic, here.

I think the questions here roughly decompose into three different topics. I discuss each of these in separate posts — each of them summarized in this post.

1. We and the AIs are in very different situations, with very different sets of options in front of us. How does ECL work in circumstances like this? How good does an opportunity to benefit the AIs need to be before we should take it?

   1. Summary below; main post.

2. Do we have any sufficiently good opportunities to benefit the values of distant AI systems ? And do they have any good opportunities to benefit us?

   1. Summary below; main post.

3. Is it at all plausible that our decisions give us any evidence about what the AIs do? (Or that we have any other kind of acausal influence on the AIs, if you prefer an account that’s different from EDT.)

   1. Summary below; main post.

Here’s a brief summary of my current impressions.

1. How does ECL work in asymmetric situations?

Even if situations are superficially very asymmetric, they will still have some abstract structure in common. In particular, if I decide to search for actions that are beneficial for actors whose choices correlate with my choices, then that’s arguably analogous to them deciding to search for actions that are generally beneficial for actors whose choices correlate with their choices. (Including me.)

If I have some pre-existing belief about who I am correlated with, then I can use this principle to calculate how much weight I should put on other people’s preferences. In the linked post, I derive a formula for this that says that I should care more about their preferences insofar as:

• I have significant acausal influence on them, compared to the acausal influence I have on people who share my values.

• They have significant acausal influence on me, compared to the acausal influence they have on people who share their values.

So those are the quantities to be looking at.

Why does it matter that they have acausal influence on me, and not just that I have acausal influence on them?

I’m here using “acausal influence over an actor” as a shorthand for “If I take action A, it is more likely that they will take an action analogous to A”. But that means that if I “take actions that benefit people who I have acausal influence over”, then that means that they will “take actions that benefit people who they have acausal influence over”. If they don’t perceive themselves as having acausal influence over me, I don’t gain anything from this. So in order for both of us to gain evidence that we will benefit each other, we need to both perceive ourselves as having acausal influence over the other.1

For more, see the next post on Asymmetric ECL.

2. What could we offer distant AIs?

Using the formula that I derived in the above post, we can then get a number for how much we should value benefits to the values of distant AIs, compared to how much we should value benefits to our own values. (In particular, this is precisely proportional to the two quantities I mentioned in the above summary.) This number is necessarily smaller than the value we should place on benefitting our own values — so the question is whether we have any opportunities to help the values of distant AIs more effectively than our own values. 

More specifically, the question is whether we have good opportunities to help the values of distant ECL-ish AIs. Here, I’m using “ECL-ish AIs” as a short-hand for AIs that are sympathetic to ECL and are in a position where ECL gives them reason to help us.

Do we have any such opportunities? The short answer is that I don’t know. But here are two candidates:

• We could increase the likelihood that — if AI ends up misaligned — that it shares values with distant, ECL-ish AIs.

• We could increase the likelihood that — if AI ends up misaligned with our values and it shares values with distant, ECL-ish AIs — that it ends up reasoning well about decision theory. (By the lights of distant, ECL-ish AIs.)

How could we increase the likelihood that AI broadly shares values with at least some distant, ECL-ish AIs? Our main clues here are:

• We can empirically study (and speculate about) what values AI systems tend to adopt.

• ECL-ish AIs have universe-wide values, so AIs with universe-wide values are more likely to share values with ECL-ish AIs.

• We want to avoid values that any significant number of ECL-ish AIs are opposed to. So we want to avoid values where both the positive and negative of that value seem similarly-likely, and promote values that seem likely to be uncontroversially good.

• Confusingly: If we both make AI ECL-ish and give it some particular values, then that’s evidence that other pre-AGI civilizations will do the same. This will increase the likelihood that there are some ECL-ish AIs with the same values out there. So this might give us reason to make sure that AIs are ECL-ish.

  • However, this introduces additional complications, since now our cooperation partners’ existence might be (partly) dependent on our cooperation. I don’t know exactly how this works.

How could we increase the likelihood that AI ends up reasoning well about decision theory, by the lights of distant, ECL-ish AIs? The one thing we know for sure is that such AIs will themselves be ECL-ish, so probably they’d want other AIs with their values to have the preconditions for that — including eventually adopting some EDT-ish (or maybe FDT-ish) decision theories, and potentially being updateless, to some degree. (I discuss why AIs may need to be updateless to cooperate with us here.)

It’s important to flag that ~all the interventions that are suggested by this have plausible backfire risk, such that I feel pretty clueless about whether they’d be net positive or net negative:

• If AI has large-scale, “universe-wide” values, that seems more likely to generate conflict with other actors that the AI shares our physical universe with.

  • This includes humans: Having AIs with impartial, large-scale preferences seems significantly more likely to lead to AI takeover than AIs with more modest values.2

  • Because of considerations like this: If I was forced to make a choice at the moment, I would currently prefer early AGI systems to have highly local and modest goals (e.g. indexical maximization of their own reward) rather than universe-wide values.

• AI that act according to acausal decision theories seem harder to control from an alignment perspective. For example, they seem more likely to coordinate with each other in ways that humans can’t easily detect.

• If AI starts thinking about acausal decision theories at an earlier time, it may be more likely that they make foolish decisions due to commitment races.

For more on this topic, see this post.

3. Are our choices analogous to AIs’ choices?

As mentioned in the above two summaries, ECL only gives us reason to benefit AIs if we can acausally influence AI decisions, and vice versa.

Indeed, the value we should assign to benefitting AIs is proportional to the influence we have on AI decisions, compared to the influence we have on decisions made by distant evolved species who share our values. (And the corresponding numbers from the AI’s perspective.)

From an EDT perspective, the question “does my decisions acausally influence AIs’ [or evolved aliens’] decisions?” reduces to the question “do I see my decisions as evidence for what AI [or evolved aliens] will do?”.

When looking at these questions, it intuitively seems like my influence on evolved aliens should be much greater than my influence on distant AIs. If I query my brain for intuitive predictions of distant actors, it sure seems like my own actions have more impact on my prediction of distant evolved species (in pre-AGI civilizations) than on my predictions of what AIs will do.

I think that intuition is worth paying attention to. But I think it’s less important than it naively seems. Here are some very brief arguments and counter-arguments:

• “AI faces very different options than us. If we choose to build ECL-ish AI, it’s not even clear what that would be analogous to, on the AI’s side.”

  • This seems like it’s much reduced by retreating to more abstract questions, like “Should I adopt a policy of impartially optimizing for many different values?”

  • There’s some discussion of this in the post about asymmetric ECL.

• “Sufficiently smart AI can choose on the basis of what’s actually rational. Our merely-human decisions mainly provide evidence about what sort of quirks and biases are confounding our merely-human brains at the moment.”

  • I think this argument works if you’re sufficiently pessimistic about correctly reasoning about decision-theory.3

  • But personally, I do expect significant correlation between what (at least some) humans decide and what’s rational in some more abstract sense.

• “Future AGI systems will have directly observed the behavior of evolved species, and/or deeply understood the nature of our cognition, and/or deeply understood the nature of all decision theory that we could possibly understand. Thus, AGI will see our behavior as predictable. It won’t see its own behavior as evidence for what we do. This is an important way in which AI’s reasoning will differ from ours.”

  • I think this argument is fairly strong.

  • My main counterargument is that, if AI starts out in a position where this isn’t true, (i.e., a situation where it does see its decisions as evidence for our decisions) then it would prefer to not reach the omniscient state described above.

  • So if it can go “updateless” in the right way, before learning too much, the above argument does not apply.4 (Or if the AI later decides to go retroactively updateless.)

  • (For some more discussion of when EDT agents seek out information, see When does EDT seek evidence about correlations?)

I think there’s a lot to say on this, and I don’t understand it very well. For more discussion, see this post.

More content

Once again, here are the three posts with more content.

1. Asymmetric ECL

2. Can we benefit the values of distant AIs?

3. Are our choices analogous to AI choices?

(Tentative) Conclusion

I think it’s pretty up-in-the-air whether the right answer here is:

1. “Even if this ECL stuff works out, you don’t correlate enough with the misaligned AIs (relative to humans) to move your decision.”

2. “You correlate different amounts with different actors in a way that’s really important to keep track of, but we do correlate non-negligibly with some AIs, such that we should care about their influence”

   • E.g. because some version of the framework above works, and c5 is >5%.

3. “You should basically impartially optimize for the values held by anyone who follows a good decision-theory”

   • E.g. because some version of the framework above works, and c is systematically approximately 1.

Note that there are two different versions of the two last bullet points:

• The weaker versions are: If you worked out the decision theory correctly, it’d be rational for that wise version of you to care about their decision-theoretic influence on AIs.

• The stronger version of the statement adds that our current ignorance doesn’t ruin the relevant types of correlation and acausal influence we seek. I.e., if you worked out the decision theory correctly, you’d conclude that it would’ve been correct for even a more ignorant version of you to care about the AIs’ influence.

It seems like questions of ECL are sufficiently tied up with tricky questions in decision theory that we’re unlikely to become confident in answer 2 or 3 before the singularity. So we mostly care about the stronger versions.

I currently think that each of the answers 1, 2, and 3 seem ≥10% likely — though answer 1 seems like the most probable one. (Assuming that we restrict ourselves to the strong versions of answer 2 and 3.)

Granting that, there’s additional uncertainty about whether any particular intervention would in fact benefit the values of distant ECL-ish AIs. Where the discussion in "How could we benefit distant AIs?" don’t exactly leave me confident about what’s good.

So: This is all so-far very speculative. But it’s at least plausible to me that some interventions in this space could recover some non-negligible fraction of the value of aligned AI, even if we fail at more ambitious alignment. Further research could be valuable.

Here, I want to zoom in on ECL’s relevance for the following type of intervention: Making misaligned AI behave in a better way towards distant civilizations. (Either alien civilizations that it encounters within the causally affectable universe, or civilizations that it never encounters physically, but can infer the existence of.1)

Some candidate interventions in this class are:

• Roughly shaping AI values to avoid giving the AI values that would make interactions with others go worse (e.g. spitefulness or value-based unwillingness to compromise) or to increase the likelihood that such interactions go especially well (e.g. Bostrom’s idea of value porosity.)

• Preventing AI from making early bad decisions. (Motivated by e.g. commitment races. The focus on early decisions is because it’s easier for us to affect decisions that are made closer in time to our interventions.)

Here’s one way in which ECL is relevant for this: An additional candidate intervention to put on the list is “make misaligned AI sympathetic to ECL”.

But I won’t say any more about that in this post. Instead, my focus is:

How much does ECL raise the importance of this whole class of interventions, via making us care more about distant civilizations?

Note that I won’t talk about the “baseline importance” of such interventions, before taking ECL into account.

I’ll talk about:

• How much more do we care about this if we do ECL with distant evolved aliens? (Compared to if we don’t do ECL at all.)

  • Tentative answer: A bit more. Maybe something like 1.5-10x more.

• How much more do we care about this if we do ECL with distant misaligned AIs? (Compared to if we only do ECL with distant evolved aliens.)

  • Tentative answer: Not much more.

• What about distant actors that don’t go into either of those categories?

  • Tentative answer: Seems hard to tell!

This post will be fairly math-heavy. Feel free to skip it if you’re not up for that.

The setting I’m looking at

I will be looking at a very abstract problem that captures some of the key dynamics. In particular, in order to study:

How much does ECL raise the importance of this whole class of interventions, via making us care more about distant civilizations?

I will be looking at:

How much does ECL raise the importance of interventions that broadly benefit a variety of beneficiaries with different values, compared to interventions that mainly benefit our own values?

Using the following model:

• There’s some large number of distant civilizations.

  • I will be talking about N civilizations indexed between 1 and N.2

• Each of those civilizations has some particular values. When I talk about a civilization being benefitted or harmed, I’m actually talking about their values being benefitted or harmed.

  • In particular, I want to assume that the degree to which we care about effects on distant civilizations is proportional to how much their values overlap with our values. (How much “the things they care about” overlap with “the things we care about”.)

• As a stand-in for interventions that “benefit a variety of beneficiaries with different values”, I’ll talk about interventions that benefit a random, distant civilization.

  • This captures how such interventions look more promising if a large fraction of the potentially-benefitted civilizations have similar values to us. And how the interventions look less promising if we’re indifferent to benefits and harms to most civilizations’ values.

  • It also puts to the side questions like “how many civilizations like these are there?” and “exactly how do these benefits/harms work?” which I prefer to tackle separately. (Not in this post.)

• I will operate in an expected-utility framework, where we e.g. care twice as much about things that happen with twice the probability, and where we can talk about benefitting someone by x units (which we would care about x times as much as 1 unit).

Here is some notation. I will denote the overlap between our (universe-wide) values and the values of civilization i as v(i). The number v(i) will have the following properties: