Today, we’re introducing a new tournament scoring system that’s clearer, more consistent, and that better rewards more forecasters.

The new system is simple: the sum of Peer scores determines a forecaster’s tournament rank, and their winnings are proportional to the square of that sum.

The new system applies to all new tournaments, and to very long term tournaments with few resolved questions. (Find a full list below.)

Why change?

Our tournament scoring was groundbreaking when we introduced it in 2021. It served us well over the last 3 years, aligning incentives and distributing more than $50,000 in prizes on dozens of Tournaments.

The new system offers a few key advantages:

• Simplicity. Many users told us that the old system was too complex. The new system is simpler.
• Consistency. The new system is based on the Peer score, just like medals.
• Prize distribution. Some past tournaments saw a single user win 90%+ of the prize pool, which discouraged participation for those who fell behind early. Back-testing on past tournament data tells us the new system would have rewarded more of the top forecasters.

How does it work?

In words:

You receive a Peer Score for each tournament question, with zeros for any you don’t predict. Your Total Score is the sum. If it’s positive, congratulations: you just won a share of the prize pool. Your share is proportional to the square of your Total Score. Your Total Score also determines your tournament ranking.

Not in words:

$$\text{your Total Score} = \sum_\text{questions} \text{your Peer score} \\ \text{your Take} = \max ( \text{your Peer score}, 0)^2 \\ \text{your prize share} = \frac{\text{your Take}}{\sum_\text{all forecasters} \text{forecaster Take}}$$

What does it mean?

To maximize your Total Score, always predict your honest belief.

Further:

• Forecasting all questions and forecasting early are now more important. Previously, predicting half of a tournament’s questions or joining halfway through would win you ~39% of the prize you would’ve otherwise won. Now it wins you ~25%.

• If we apply the new scoring to past tournaments, we find on average the top forecaster would have won a smaller prize, but the top 10 would have cumulatively won more prize. We expect far fewer cases where the top forecaster receives 90%+ of the prize.

We think those design choices have better incentives and reward users for forecasting more. Read about the trade-offs and alternatives we considered here: New Tournament Scoring: Trade-offs, Decisions.

Example

Imagine a tournament with 5 forecasters, 30 questions, and a prize pool of $1000.

• The first forecaster is excellent and always achieves a Peer score of +40.
• The second forecaster is just as good, but joined halfway through.
• The third forecaster is good and always gets +10.
• The fourth forecaster is bad and always gets -20.
• The fifth forecaster is quite poor and always gets -50.

Their scores and winnings are:

Current tournaments

Some tournaments will run for years to come. We don’t want to retain both scoring systems for longer than necessary. We also appreciate that you may have had certain scoring expectations at the time of your forecasts. To balance these, most tournaments will continue using the legacy scoring system, except for tournaments that:

1. are very long term (and unfinished).
2. have few or no resolved questions.

Here are the full lists:

Two special cases not mentioned above:

1. Biosecurity Tournament: the $15,000 for the 2024 questions will be awarded according to the old scoring system. The rest of the prize ($10,000) for the 2032 questions will use the new scoring system.

2. Forecasting Our World in Data: the $8,000 awarded in 2025 for the 1-year questions will use the old scoring system. The other $8,000 awarded in 2027 for the 3-year questions will use the new scoring system.

Note that the current (Q1 2024) Quarterly Cup will use the old scoring system, but future iterations will use the new system.

For simplicity, all Question Series will switch to the new system, since they don’t award prizes or medals.

If you're unsure which scoring system a particular tournament uses, open its leaderboard: if there is a "Coverage" column, then the tournament is using the legacy scoring. The new scoring system does not use the Coverage, and omits that column.

We look forward to reading your feedback, questions, and ideas below!