#1 - Katja Grace - AI, The Future, and Risk

I'm pleased to introduce Katja Grace. She is the co-founder of AI Impacts, a staff researcher at the Machine Intelligence Research Institute, and writes on AI and human behavior at her blog Meteuphoric.

You walk into an elevator with someone else who starts talking about AI forecasting, and they mutter that it’s too unpredictable to be helpful. How would you respond?

Policy decisions are implicitly bets about the future, and on a topic like this one, the stakes might be our lives or livelihoods. So even if the topic is hard to predict well, isn't it better to predict it as well as we can than to play with our eyes closed?

What are you currently researching at AI Impacts right now, and why does it matter?

I just got all the responses back from a 2024 iteration of the big survey of AI researchers I've now run four times. So I'm winding up logistics of sending $50 for taking it to researchers in many corners of the world and answering emails about it, before setting out to analyze the results.  It's important because AI researchers are credible experts who are widely known to be informed about what is happening in AI, as well as having unique power in it.

So it is both helpful for the world to know what they expect from AI as evidence about what will happen by default, and as evidence about the AI researchers themselves and how they might behave (for instance, what policies they might support). I hope it is useful to AI researchers themselves, for updating more quickly about how opinion is moving in the field, so that their discussion on these topics can be of the highest quality.

The questions about risk have a particular value to the public discussion of AI risk, because one might normally expect that if a technology seems risky to laypeople, then those building it can substantially debunk the lay-worries. And some people even publicly say that AI researchers aren’t concerned about AI risk. But if you check with a survey, the median AI researcher puts about 5% chance on human extinction or something like it coming from advanced AI.

So that’s an important correction to the public discussion: AI researchers do not have special expert access to compelling reasons to not treat this as a major risk. (Incidentally, this is a place where the value of a forecasting exercise is particularly independent of whether AI researchers are good at forecasting: if there is an assumption that they are forecasting safety with some sort of scientific expertise, then hearing their real guesses tells us that they aren’t, setting our expectations back toward the more risky prior.)

Your best guess prediction for the top two disruptors AI will have in our future…and how do we plan for it?

1. Becoming the main decision-makers in the world: when AI systems are better at making decisions than any human, they will probably decide most things. This might be because we choose it, or it might be because our inferior decision making means we are no longer in control—when AI systems can steer us like a skilled manipulator can steer a child, then our efforts to stay in control will be easily stepped around, perhaps without us noticing. 

   
   

   Whether another system 'making all the decisions' means a tragic loss of control for humans seems to depend  on whether that system has its own goals that it makes decisions in order to forward, or is 'making decisions' in the sense that Google Maps makes the decision for me of how to get across town: outputting an answer to the question I put in, with no hidden motives. So to prepare:

   
   

A) Build AI systems that are the Google Maps equivalents—reflexive responders to our requests—not autonomous players where possible. (At least until we much better understand the situation). For reference, current systems are the former, but we appear to be working actively on the latter.

B) If you must make agents, be very sure they have exactly the goals you want, such that you are OK with giving up any control for the rest of the future (this seems like a dubious short term plan at a glance).

2. Obsoleting human labor. This is related to 1, but is a big deal so deserves its own consideration. My best guess about how to prepare as a society is to have a plan for massive redistribution of wealth, such as a universal basic income. As an individual, you should probably prepare by prioritizing having capital to invest, rather than prioritizing being a good laborer. (It is very unclear how soon this disruption will arise.)

What’s the most glaring mistake you see in AI development today?

It's hard for me to say what other people are doing right or wrong, but from the outside it looks like pouring full-time-labor-level effort into pushing for acceleration of the AI frontiers while apparently allocating sophomore-at-a-cocktail-party level effort into thinking through high level strategy e.g. whether such acceleration is likely to be advantageous.

How important is anthropic reasoning in relation to artificial intelligence?

I'm not sure. For me the biggest place it comes up is as evidence that humanity faces major existential risk (see my undergrad thesis or blog-post length argument), but where that risk comes from something other than sophisticated AI taking over the world. I thought about that a lot in 2010, but am not sure how seriously to take it.

AI might also cause anthropic reasoning to come up a lot more often: AI allows for many stranger scenarios of duplicate minds and fake worlds, which are the stuff of anthropic thought experiments!

Katja Grace is the co-founder of AI Impacts. You can read more about her here.

Jonathan Runyan is a senior cyber security engineer and former pastor writing on the intersection of spiritual and virtual reality. You can read more about him here.