## **Introduction**

Will we be surprised by progress in Artificial Intelligence? This is an important question, because if, for example, there is a frontier in AI that is close to falling (that we don’t know about) then the country or group who break that barrier may have a period of decisive strategic advantage over competitors. However, one of the nice things about prediction markets is that if we operationalise these terms properly we can have a pretty good stab at answering this question even though we don’t know which barriers might be close to falling:

  


This suggests that the community predicts a roughly 70% chance (at time of writing) that AI progress will lead to ‘discontinuities’ – leaps in capability far beyond the level a simple extrapolation of past improvements would predict (for example, the first transatlantic flight was radically faster than a projection based on the incrementally increasing speed of steam ships would imply - [here](https://www.lesswrong.com/posts/CeZXDmp8Z363XaM6b/discontinuous-progress-in-history-an-update) are some other good historical examples of the phenomenon). ‘Discontinuities’ aren’t exactly the same as ‘surprises’ – for example we’d be surprised if the first AI would only talk to humans if they dressed up in gorilla suits – but they are a pretty good proxy for the sort of surprises that we should worry about.

However, this essay is about being **REALLY** surprised by AI futures – are there potentially surprises even a prediction market cannot tell us about?

  


## Dr. Diabolical vs Metaculus

Here’s another question about surprises: What is the probability that every server that hosts Metaculus is simultaneously destroyed tomorrow, through a combination of physical and electronic attacks that effectively remove it from the internet?

Hopefully you agree with me the odds are very low – you would be very surprised if this happens. But perhaps I’ve spooked you with this scenario and you want the reassurance of a prediction market to make absolutely sure there’s nothing you’ve missed. This question seems perfectly amenable to a prediction market, so you post up on Metaculus and receive exactly the response you were hoping for – it is highly unlikely that Metaculus will be destroyed tomorrow.

What you might not know is that even as you read this essay, Dr. Diabolical sits in his island lair plotting his nefarious business. There are two facts about Dr. Diabolical you should know; Firstly, he cares very deeply about his aggregate score on Metaculus. Secondly, tomorrow he plans to destroy the entire internet as part of some grandiose scheme or another and will probably succeed (since he banished his arch-nemesis, Ultraman, to the shadow realm and the community is clear that they don’t expect him to return until at least next year…). Consider that Dr. Diabolical has very important information about this prediction market which ordinarily would be synthesised by Metaculus and used to give an accurate picture of the future. Dr. Diabolical cares very deeply about his Metaculus ranking and so would normally be delighted by the opportunity to pick up some easy points. However, in this particular case, Dr. Diabolical has no reason to participate in the market – he knows Metaculus won’t exist tomorrow and therefore nobody will be around to resolve the question.

We would be ‘radically surprised’ by Dr. Diabolical’s scheme, in the sense that it falls outside the scope of what a prediction market can make coherent statements about. But whereas Dr. Diabolical does not exist and so we probably don’t need to worry about his probability-bending effects on prediction markets, AI probably will exist at some point in the future and if AI might also have a probability-bending effect on prediction markets we should really worry about that now rather than on the day we wake up to discover Metaculus (and, to a lesser extent, the world) has stopped existing.

  


## I’m FOOMing about this

There are, very broadly, two basic paths to transformative AI. The first is called ‘FOOM’, which is usually understood as a proto-intelligence gaining enough awareness to design the next iterations of intelligences, which themselves design the next generation, and so on. Because we imagine computers could perform these operations at speeds limited only by their processors, FOOM imagines that progress from the first proto-sentience to superhuman intelligence could be completed very quickly, perhaps even a matter of hours. The second path I will call ‘Everything Else’ which is a catch-all category for all non-FOOM scenarios (for the sake of this essay). The only important feature about ‘Everything Else’ is that it means AI is developed slowly, maybe on the order of years or decades.

A FOOM scenario might cause Metaculus (and prediction markets more generally) to cease to exist in a variety of ways, not all of which are bad:

* AI may make prediction markets obsolete by serving as an ‘[oracle](https://www.lesswrong.com/tag/oracle-ai)’ which answers questions better than a prediction market
* An AI designed to be an oracle might not replace prediction markets, but instead discover that it is easier to hack Metaculus’ servers and resolve questions arbitrarily to appear better by comparison
* It becomes clear that prediction markets form an important component of bootstrapping an AI and governments ban online prediction markets
* AI allows us to transcend human existence in some way, relegating such parochial concerns as ‘Metaculus score’ to a subsection of the hive mind devoted to cataloguing our pre-posthuman follies
* A rogue AI destroys the whole world

Any of these FOOM AI scenarios is similar to the Dr Diabolical scenario – Metaculus ceases to exist so quickly that there is no time for the market to assimilate the information about its own destruction and award points accordingly. Therefore, even if you had information that made you certain an AI was going to be developed tomorrow, FOOM, and then spend the rest of its runtime trying to destroy Metaculus, you would have no way of synthesising that information into a prediction market. 

Note that this is importantly different from existential risks more generally – the important point is the speed of the FOOM, not the completeness of the destruction of the market. For example, currently the community gives almost no probability to the idea that a climate catastrophe causes large-scale human death. However, there are certain observations that we could describe in principle that would imply we were heading for runaway global warming which probably **would** kill most life on the planet. If those observations started falling more and more within the confidence intervals of high-quality research, then we’d expect the probability to rise incrementally, as – for example – people notice an opportunity to arbitrage questions which resolve sooner.

  


## Implications for prediction markets

I therefore think the Metaculus community is under-predicting radically surprising AI futures, because there is no way to discuss radically surprising futures in the context of a prediction market. That is, all questions about the future of AI are implicitly made up of different kinds of AI outcomes (“This one is slow and friendly, this one is fast and hostile, in this one AI is never invented…” and so on), but in only some of those outcomes are predictions ever going to end up on Metaculus, or any other prediction market.

# 

Consider, for example, the probability that FOOM occurs at all. As of the time of writing this essay, only about 8% of the community have predicted we are in a FOOM-like scenario (that is, the AGI to superintelligence transition taking less than a week). There seems to be a strong community consensus that we are in an ‘Everything Else’ scenario, probably with medium sort of take-off speeds giving a time between AGI and superintelligence measured in months but not in years.

  


My argument is that this does **not** mean that we can act as though there really is an 8% chance of a FOOM like scenario, and we shouldn’t interpret the outcome in a vacuum like we can with other estimates – there is an 8% chance of a FOOM which does not destroy Metaculus, and an unmeasured but higher chance of a FOOM which wipes Metaculus out completely in some way and therefore cannot be aggregated by Metaculus. In total therefore the chance of FOOM is >8%, but we just don’t know by how much: 

* It could be 8% almost exactly if a FOOM-plus-catastrophe scenario is very unlikely, so the hidden value is nearly 0%
* It could be nearly 100% if almost everyone knows a FOOM-plus-catastrophe is coming very soon but also knows they can’t therefore score points by predicting it; this implies only suckers remain in the market and the predicted value is noise.
* It could actually be less than 8% if we’re being strictly honest with ourselves – perhaps a slow-AI takeoff is more likely to lead to the total destruction of the internet somehow (perhaps a slow-AI has longer to infect all possible servers with a worm before starting its attack). The catastrophe could occur without FOOM, although it seems to me that FOOM is the most likely path for this sort of catastrophe to occur by.

There are some properties we can be reasonably confident this unknown probability has. For example, events which move the market closer to the FOOM end also seem very likely to be raising the hidden probability of FOOM-plus-catastrophe because FOOM and FOOM-plus-catastrophe seem very closely correlated (indeed, one is just a subset of the other). So, we can probably – over time – make sensible statements about whether ‘radically surprising’ futures are becoming more or less likely. Similarly, we could look at questions on AI catastrophe, such as the one below, and make the reasonable inference that the more likely catastrophe becomes the more likely FOOM-plus-catastrophe becomes. 

  


A second fairly important area to look at is ‘hockey stick’ predictions in unrelated problems, for example like the one below about the maximum compute in AI training in 2026. The community consensus is that it will be somewhere in the 4m petaFLOP-days, but you can see a few brave souls predicting as far out to the extreme right-hand edge as the graph can go. I don’t think these should be taken as literal estimates that 1bn petaFLOP-days is the right answer and that researchers will have just upped their game massively by 2026, but rather that the entire paradigm of AI training will have been totally overturned by something new in 2026, allowing us to throw huge numbers of petaFLOP-days at problems*. For our purposes, this helps to understand when and how AI questions might go ‘out of context’ and hence put upper and lower bounds on how ‘out of context’ the FOOM-plus-catastrophe scenario might be.

  


**Of course, this is slightly confounded by the fact that the new thing that comes along might be a FOOMing AI, which brings us back to square one for this particular problem – but I think the principle of using ‘hockey stick’ outcomes to frame other unknown information is probably sound.*

## Conclusions

Thinking about AI futures is enormously exciting. The community’s prediction is that human-machine intelligence parity is almost as likely to come before 2040 as after, suggesting I will probably live to see the first true Artificial General Intelligence.

  


Many entrants to this contest have taken this broad idea of excitement and built fascinating roadmaps to the future by piecing together different community predictions into a synthesis of a ‘community’ narrative. I wanted to offer a slightly different take which engaged as fully as possible with the entire Metaculus prediction spread - the idea that AI futures might exist which are literally unpredictable (in the sense that they cannot possibly be accurately assessed using the techniques of prediction markets) will systematically bias the results in every other essay towards believing that AI will be safe. The strong implication of this is that AI is more dangerous than we realise, by an amount we cannot know with high confidence.

I believe this is almost completely unique to AI – almost nothing else could destroy a prediction market fast enough for it to systematically distort the information aggregation function of that market. However, this doesn’t mean we should give up on prediction markets as a vehicle for thinking about AI futures, or even AI risk:

* First, there are many important things we can ask about AI that are not affected by this ‘radical surprise’ feature at all. For example, questions about the job market for AI researchers (of which there are a few on Metaculus) are of high importance to people making decisions about their career, and it is good that Metaculus provides an unbiased source of information about these topics
* Second, we actually can still say some things about AI risk even accounting for hidden ‘radical surprise’. We know it will correlate with certain other things that we can predict (even if we don’t know the exact correlation coefficient we can at least guess at the sign) and we know that we can get accurate information about ‘out of context’ events in some other AI scenarios (even if we don’t know how translatable these results are). This suggests we might be able to limit our exposure to ‘radical surprise’ by asking enough clever questions about correlates and hockey sticks.
* Finally, we don’t **have** to use prediction markets to solve every possible problem. Individual high-profile AI researchers who learned that such-and-such a technique could lead to FOOM-plus-catastrophe would probably use more old-fashioned channels to sound the alarm, and hence people who get their news from a combination of prediction markets and traditional sources would form a more nuanced picture about AI future than individuals who used either technique in isolation.

I completely agree with the community that we are likely to be ‘surprised’ by AI futures. But whether or not we are doomed to be ‘radically surprised’ (surprised in a way we cannot possibly account for using prediction markets) is an open question. On balance, I think we have enough evidence that the risk is <5% that we can be safe enough treating this as background noise in most prediction contests (I draw this conclusion mainly from the good consistency between expert non-market forecasts and the market forecasts implying that the unmeasurable-by-markets component must be small). However I also think it is sensible to shade interpretation of all AI questions in the tournament towards the more pessimistic end by a small amount, until we can be confident there are no ‘radical’ surprises lurking in our AI future.



---