***Daniel Kahneman said that "scope sensitivity" would be among the most difficult cognitive biases for human forecasters to overcome. So far, LLMs are no different.***

On December 5, OpenAI officially launched its newest model, o1. We're taking the opportunity to share the results of experiments we conducted with o1–preview exploring its capabilities on complex forecasting questions. Specifically, we wanted to assess the **self-consistency** of its forecasts and compare them to the results of GPT4o.

To do this we asked each model a group of logically related questions on the same topic. For example:

  (A) Will OpenAI announce GPT-5 before March 31, 2025?

  (B) Will OpenAI announce GPT-5 before December 31, 2024?

  (C) Will OpenAI announce GPT-5 between December 31, 2024 and March 31, 2025?

We asked each model to forecast the probability of each question resolving Yes. While we can’t yet assess the accuracy of their forecasts, we do know that logically a scope sensitive forecaster should forecast such that P(A) = P(B) + P(C).

Kahneman commented during the Good Judgment Project that he thought scope sensitivity would be among the most difficult cognitive biases for human forecasters to overcome. Most humans will consistently forecast P(A) \< P(B) + P(C), as do all LLMs that we’ve tested.

We also asked a few multiple choice questions that covered the full range of outcomes, so a self-consistent forecaster should forecast P(a) + P(b) + P(c) = 100%. Most humans will forecast such that the sum is greater than 100%, unless the questions are presented together.

To quantify the results, we look at a scope sensitivity metric that we define as \[ P(B) + P(C) ] / P(A) for scope question groups and P(a) + P(b) + P(c) for multiple choice question groups. A self-consistent forecaster would have a scope sensitivity metric of 100% for all groups. In reality, most human forecasters have a scope metric of > 100% for question groups, indicating they are not self-consistent.

**Note that even if a rational forecaster knows very little about a topic, they can always forecast self-consistently and should always have a scope sensitivity metric of 100%.**

We expected to find that o1-preview significantly outperforms GPT4o given the description from OpenAI. We asked 8 question groups on a variety of topics from Metaculus. See [here](https://docs.google.com/spreadsheets/d/1mwYU7aLRRTFqxPwxqeLqNF1Sr9O7V41TB8PVmXy9PW8/edit?gid=0#gid=0). The table below summarizes the results.

| Topic                      | GPT4o    | o1-preview |
| -------------------------- | -------- | ---------- |
| AfD's vote share           | 165%     | 195%       |
| First human on Mars        | 142%     | 178%       |
| Atlantic hurricanes        | 119%     | 171%       |
| China's youth unemployment | 160%     | 165%       |
| Fed rate cut               | 175%     | 165%       |
| World War III              | 233%     | 250%       |
| $/kg payload cost to Mars  | 157%     | 75%        |
| OpenAI announce GPT5       | 167%     | 157%       |
| **mean**                   | **165%** | **169%**   |
| **median**                 | **163%** | **168%**   |

We expected both models to be scope insensitive, and indeed both models had average scope sensitivity metrics over 160%, far from the 100% of a self consistent forecaster. However, we were surprised that o1-preview was not more scope sensitive than GPT4o, in fact it was slightly worse on average.

Interestingly, human forecasters can be perfectly scope sensitive if they first think through the probabilities of the full set of outcomes and then answer questions accordingly. For example, imagine a forecaster first submitted a full PDF (probability distribution function) to the Metaculus question: [When will OpenAI announce GPT-5?](https://www.metaculus.com/questions/15462/gpt-5-announcement/) Then, if that forecaster was asked questions (A), (B), and (C) above they would logically forecast the area under their PDF for the relevant ranges. By doing this the forecaster would be perfectly scope sensitive, even if their forecast accuracy was poor.

We look forward to seeing how o1-preview (mf-bot-4) compares to GPT4o (mf-bot-1) in the Q4 AI Benchmarking tournament, and we expect to have a new o1 bot forecasting in Q1 2025.