*This essay was submitted to the* [*AI Progress Essay Contest*](https://www.metaculus.com/project/ai-fortified-essay-contest/)*, an initiative that focused on the timing and impact of transformative artificial intelligence. You can read the results of the contest and the winning essays* [*here*](https://www.metaculus.com/questions/11429/results-from-the-ai-progress-essay-contest/)*.*

  


Betting on the end of the world as you know it is a questionable enterprise.

You can't get paid if you're dead, for one. If you manage to secure the dubious luxury of living in a nuclear wasteland, any dollars you may receive would be worthless. And if the world ends *well*, and you find yourself in a post-scarcity technological utopia... Well, it's all nice and good, but your dollars would be worthless all the same.

Thus, forecasts related to global paradigm shifts have ever been the bane of prediction markets.

Questions that resolve in many years or decades offer similar problems. When you place a bet, it's not just a bet on the subject matter, it's also a bet that both you and the prediction platform would be alive and solvent when it's time for the pay-out. And as timelines grow longer, it becomes that much less certain.

Thus, foreseeing the development of AI over the next century is one of the most difficult tasks you could set a prediction market.

Reputation-based prediction aggregators avoid the problem somewhat. Your dollars might become worthless, but the social capital you'll get for having *called it* is more likely to survive an upending of the established order — as long as there's anyone at all left alive afterwards. It also helps that you don't have to bet with your livelihood, don't have to lock-in for decades the resources you may need for something else.

As such, the lack of a direct profit motive should make the forecasters' predictions *more* accurate, in this specific category of cases.

How well is Metaculus doing, then?

In this piece, I'll take the most broad forecasts related to AI progress, assume that they're literally true, do my best to build a consistent picture of the future based on them, then evaluate that picture's merits.

  


# Part 1: At Face Value

## 1.1: AI Development Timeline

To start off, two direct questions:

The difference between them, essentially, is that the first one asks for an AGI that can match the performance of an average college-educated human on mostly virtual tasks, while the second one asks for a polymath-equivalent that can act in the real world. It seems that, on average, they will be fourteen years apart: the median predictions for them are 2042 and 2056, respectively.

Some questions arise when we consider them jointly with the following forecasts:

Both count from the date of the creation of the college-level AGI. The difference between them is that the first one asks for a superintelligent artificial intelligence that can *at least match* human performance across all human activities, while the second one asks for "reliably superhuman" performance either in question-answering ability *or* in general performance. Let's dub them "weak SAI" and "strong SAI", respectively.

The difference between the SAI forecasts seems consistent. The longer it takes to build a weak SAI after a college-level AGI is made, the longer it takes to build a strong SAI after a weak one. If the weak SAI takes one month, the strong SAI takes less than one. If the weak SAI takes half-a-year, the strong SAI takes about a full year. Two years => five years, and so on.

However, this doesn't square with the timeline revealed by the difference in the college-level AGI vs. polymath AGI. For example, if we get a college-level AGI in 2042, and in six months after that a weak SAI (which the forecasters expect with 50% probability), then predicting a polymath AGI only by 2056 seems clearly inconsistent!

But let's look at the fine print. The AGI questions include this clause:


> We will thus define "an artificial general intelligence" as a single unified software system that can satisfy the following criteria [...]. By "unified" we mean that the system is integrated enough that it can, for example, explain its reasoning on a Q&A task, or verbally report its progress and identify objects during model assembly. (This is not really meant to be an additional capability of "introspection" so much as a provision that the system not simply be cobbled together as a set of sub-systems specialized to tasks like the above, but rather a single system applicable to many problems.)

The SAI questions, though, allow *the opposite*:


> "Superintelligent Artificial Intelligence" (SAI) is defined for the purposes of this question as an AI which can perform any task humans can perform in 2021, as well or superior to the best humans in their domain. The SAI may be able to perform these tasks themselves, or be capable of designing sub-agents with these capabilities (for instance the SAI may design [robots capable of beating professional football players](http://www.robocup.org/objective) which are not successful brain surgeons, and design top brain surgeons which are not football players).

So, in 2042 or thereafter, we create a college-level AGI. Shortly afterwards, likely in a matter of months, we create some ML model (probably based on the AGI) that can spin up task-specific AIs that can show reliably superhuman performance, but only in the narrow domains they were specifically built for. These sub-AIs would be meaningfully distinct from the initial AI (such that it can't explain how they think, for example), and we would only learn how to create a more advanced "unified" AGI much later, on the scale of years or decades.

Why would the AI not be able to understand its sub-agents? Well...

We, humans, are pretty good at spinning up Machine Learning models that do things well, but whose internals are nigh-incomprehensible to us. Perhaps we'll figure out how to train up a ML model that is *even better* at this, and understands its creations *just as badly*. If DL is still the dominant paradigm by then, that seems more likely than if it isn't. The college-level AGI might double as some sort of very advanced [meta-optimizer](https://arxiv.org/abs/1606.04474), for example.

That leaves the question of why it'd take so long to go from a (unified) college-level AGI to a (unified) polymath AGI, given that task-specific SAIs are so easy. Can't we just ask our college-level AGI to design a sub-agent SAI that'd know how to train up a unified polymath AGI, or something? Let's come back to this.

One addition:

So 2042 for a college-level AGI, and the decade-long gap before a unified polymath, might be the *conservative* estimates.

  


## 1.2: AI Development Background

In the years leading up the creation of the first AGI, the US government tries to spearhead the research. However...

They're not very likely to succeed. Neither are [nationalized organizations](https://www.metaculus.com/questions/8325/nationalized-organization-develops-first-agi/) (25%), [non-profits](https://www.metaculus.com/questions/8328/non-profit-organization-develops-first-agi/) (12%), or [universities](https://www.metaculus.com/questions/8326/university-group-develops-first-agi/) (20%). Who is the favorite to succeed? [A for-profit corporation](https://www.metaculus.com/questions/8324/corporation-develops-first-agi/), of course, at 70%.

  


## 1.3: The AI Catastrophe

That's looking pretty bleak. If we haven't solved the control/[alignment problem](https://en.wikipedia.org/wiki/AI_alignment) by the time AGI arrives, and AGI can quickly learn to spawn superhuman AIs, the world is [very likely to end](https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintelligence-faq#3__Why_might_a_fast_takeoff_be_dangerous_), in a *bad* way.

Consider how much money Satoshi Nakamoto has amassed by leveraging some mathematical insights; how much influence religious figures could attain by just addressing the people; how much change someone in control of a billion-dollar company could affect. Consider that all of that could be achieved by *slightly-smarter-than-usual* people, with nothing more than an internet connection, creativity, and some time.

Now imagine how much resources a superintelligent entity with capabilities for science and persuasion far surpassing any human's could amass, and the damage it could inflict.

However. The forecasters thinks there's only [a 30% chance of a global catastrophe by 2100](https://www.metaculus.com/questions/1493/ragnar%25C3%25B6k-question-series-by-2100-will-the-human-population-decrease-by-at-least-10-during-any-period-of-5-years/), plus only a 20% chance that it'd be caused by an AI if it *does* occur:

Which is to say, there's only a 6% chance that a rogue AI will wipe out >10% of humanity by the end of the century. Despite us likely (with 0.75*0.75*0.85 = ~48% probability) having unaligned superhuman AIs by 2073. How is that possible? I see two explanations:

1) Despite the fact that the alignment problem isn't solved by the time the first AGI is deployed, that AGI quickly solves it *for* us, and the superhuman tool AIs it deploys are aligned.

2) Despite the proven *capability* to create superintelligent AIs, humanity successfully coordinates to not do it before we can align them.

Spoiler: Given how the next section goes, it's the second one.

  


## 1.4: The Economic Effects

25% probability for "by 2050", 50% for "by 2074", and 75% for "by 2133".

The AGI and SAI forecasts, in comparison, say the following:

1) AGI by 2031 with 25% probability, by 2042 with 50% probability, by 2068 with 75% probability.

2) SAI by 2043 with 25% probability, by 2063 with 50% probability, by 2126 with 75% probability. (I got these numbers by multiplying the 50%, 70%, and 87% estimates of the college-level AGI and the strong SAI forecasts, respectively. I think it's a fair way to do it: both distributions reflect the difficulty of AI progress, so longer timelines in one should correspond to the same in the other.)

It seems AGI doesn't have a dramatic effect on the economy, only SAI does. This makes sense: we already have plenty of human-level intelligences involved in the economy, and human-level AI might be more expensive to run, at first, than simply employing humans.

Still, the economic indicator lags a bit behind even the SAI forecast. And it's not just noise, it gets much more pronounced:

Looking at the medians, there's a 52-year delay between strong SAI and 10%/year growth. Naively, you'd expect SAI to lead to even faster growth, and much sooner. What might be impeding it?

Well, remember how we don't want the SAIs to kill us? And that, by the time we create them, we don't yet know how to make them not *want* to kill us? And how we survive anyway? And how the US government finally took AI seriously back in the 2030s, but didn't manage to out-compete the private sector?

At some point before the first AGI is created, the alignment problem goes mainstream, perhaps the way global warming is "mainstream" now. The public and the governments notice it, and appreciate its risks. When the first SAI is made, that serves as a trigger. In a feat of unprecedented coordination, all dangerous AI research worldwide is heavily regulated or banned. The measures taken are so efficient as to hold up for many decades afterwards.

It explains why it takes so long to go from a college-level AGI to a (unified) polymath despite the ease of SAI: because of a sudden increase in scrutiny and regulation.

An additional factor behind the relatively slow economic growth might be the compute costs. Perhaps training up and running a SAI would be ruinously expensive, requiring hundreds and hundreds of billions of dollars. That would make SAIs less useful to the economy at first, and easier to regulate — easier to detect any supercomputers rogue agents or nations might be using.

First SAIs still have significant effects on the economy — perhaps we've figured some small part of the control problem out, allowing limited applications, to the tune of 6%/year growth. In the following decades, we acquire more compute and solve additional sub-problems, speeding up to 10%/year growth:

(Note that these timelines don't necessarily reflect the *real* difficulty of alignment. If all research is regulated by overcautious governments, extreme slow-downs are to be expected.)

At last, much later, probably around the 2200s, we solve the alignment problem completely:

We unbottle our SAIs, and at last have full automation. No-one has to work if they don't want to anymore, the Singularity is achieved, and everything is perfect forever.

  


## 1.5: A Different Timeline

There's this prediction that seems at odds with everything else:

It implies much faster growth much sooner. But!

This question depends on a different forecast about AGI, [*Human-machine intelligence parity by 2040?*](https://www.metaculus.com/questions/384/human-machine-intelligence-parity-by-2040/) It asks for a weaker version of the polymath AGI to be built before the 2040s, and *if* that is done (which is estimated at 40%), it asks how fast the economy will grow afterwards. It seems distinct from the "main" timeline this piece is analysing, as the other predictions assume that such an AI would only be possible well past the 2040s.

If it's not the case, then clearly the whole AI thing is easier than we've expected. Perhaps this means that alignment is easier as well, and so we'd be able to reap the benefits of strong AI assistants sooner.

  


# Part 2: Does This Story Make Sense?

To be blunt, no. No, it doesn't.

Reconciling the different predictions in a plausible-sounding way *necessitates* postulating some very specific things, too specific to be realistically predictable. In particular:

**1)** The timeline inconsistency between a college-level AGI, a polymath AGI, and a weak SAI. I've used a tiny detail to justify the seeming incongruity, but I pretty much invented the explanation *ex nihilo*. I'm not aware of any firm consensuses, in AI research circles, that some sort of meta-optimization is the quickest path to superhuman AIs, or that tool AIs would outperform general AIs. Indeed, there is some evidence to [the contray](http://www.incompleteideas.net/IncIdeas/BitterLesson.html).

And yet, I seem to see a very strong "market signal" implying this. Where is it coming from?

I suspect that the forecasts are simply inconsistent. That people tend to predict AGI somewhere in the 2040s-2050s (possibly for similar reasons as cold fusion always being 20-30 years away), but they're sensitive to the exact timing based on the exact capabilities expected of the AGI. This gives us the ~14 year difference between a college-level AGI and a polymath AGI. *Separately*, people expect SAI to quickly result from an AGI, *any* AGI, so we get short timelines on that.

The predictions aren't well-calibrated, however, meaning that the expected difference between the arrival of differently powerful AGIs can dwarf the expected time from *any* AGI to a SAI.

**2)** The confluence of factors where we develop SAI before solving the alignment problem, yet the world doesn't end. The first part certainly seems likely to me; the second one...

That unprecedented feat of regulation where the world's people and governments take the AI Risk seriously enough to efficiently ban dangerous research seems really unlikely right now. We aren't dealing that well with the global warming. We've failed terribly when coordinating against COVID-19, and that was the *easiest possible* mode of that problem.

Perhaps the forecasters think that the governments would become much better at regulation? Perhaps by adopting more features of totalitarian control states, enabled by various technological advances?

That still leaves the problem of rat races, the Cold War-style doomsday weapon competitions. And getting a SAI working for your nation is astronomically more useful than nukes. We'd need to additionally assume that either a) all extant superpowers would have such sensible, stable, and humanity-loving people in charge that they'd rather lose a geopolitical struggle than endanger the world (yeah, right), or b) by that point there would be only one superpower left, some sort of nascent world government that won't compete with *itself*.

That's a lot of assumptions piled upon one another. Again, I suspect the forecasts are simply inconsistent instead. (Indeed, [*World government by 2100?*](https://www.metaculus.com/questions/7329/earth-government-by-2100/) is sitting at 13%, discrediting b), and [*How much global warming by 2100?*](https://www.metaculus.com/questions/605/how-much-global-warming-by-2100/) is frowning at a) as well.)

Either we're unlikely to solve the alignment problem before SAI, or we're likely to survive. Not both.

**3)** The predictions about the economic growth. Again, they pretty much only work in conjunction with the "SAI by the 2070s" predictions if we assume that the governments acquire some supreme regulatory and coordination powers by that point.

Rather, I think the World GDP forecasts are simply not "in dialogue" with the AI forecasts. The one GDP forecast that was explicitly referring to AI impact predicted huge growth immediately after an AGI is developed. I suspect that if the GDP forecasts were taking the "SAI by the 2070s" forecasts into account, they would've been skewed towards early dates as well. (My dodge about it being a different timeline is only semi-plausible.)

  


# Part 3: Conclusion

The final picture turned out considerably more consistent than I'd expected. Nonetheless, it's still very patchy.

Taking the broad and literal view of the forecasts reveals major inconsistencies. But that doesn't mean there is no value to be extracted from them, or that this endeavour was fruitless. My takeaways:

1) When doing this sort of analysis, it's best to look at well-connected *families* of forecasts. For example, the forecasts regarding [what kind of organization is most likely to develop AGI first](https://www.metaculus.com/questions/8327/government-project-develops-first-agi/), or [what is the most threatening existential threat](https://www.metaculus.com/questions/?search=cat:series--ragnarok). You can be sure that the forecasters involved in them are mostly the same people, operating on the same information, and this could serve as an efficient survey of the attitudes in some domain. For-profits are seen as the most effective organizations, people aren't *despondent* regarding humanity's chances of survival yet view certain technologies as more dangerous than others, and so on.

2) On the other hand, it's also useful to compare "disconnected" forecast families to see what some groups of forecasters might be overlooking. For example, we may assume that the World GDP forecasts didn't properly integrate the AI forecasts. However, they could still be used to inform some ideas about the economic growth *in absence* of dramatic paradigm shifts.

3) Even if the resultant picture isn't useful by itself, *the process of assembling it* is. The process of writing this piece has prompted me to view the question of AI progress from new angles, to build a more well-connected causal model of it, to precisely track how some events would influence others. If nothing else, it's a good mental exercise.

4) On the flipside, weaving a concrete picture from a great deal of disparate forecasts seems like a good way to spot any issues with how the prediction aggregation works.

In conclusion, I'd like to note that these issues apply mainly to long-term questions. On short time scales, prediction aggregation works [quite well](https://www.metaculus.com/questions/track-record/).