Your Notebook is now a Draft.

Once it's ready, please submit it for review by our team of Community Moderators. Thank you!


This content now needs to be approved by community moderators.


This essay was submitted and is waiting for review.

News» {{qctrl.question.primary_project.name}}

Introducing Simpler, Fairer Tournament Scores

by Sylvain {{qctrl.question.publish_time | dateStr}} Edited on {{qctrl.question.edited_time | dateStr}} {{"estimatedReadingTime" | translate:({minutes: qctrl.question.estimateReadingTime()})}}
  • Facebook
  • Twitter

  • Today, we’re introducing a new tournament scoring system that’s clearer, more consistent, and that better rewards more forecasters.

    The new system is simple: the sum of Peer scores determines a forecaster’s tournament rank, and their winnings are proportional to the square of that sum.

    The new system applies to all new tournaments, and to very long term tournaments with few resolved questions. (Find a full list below.)

    Why change?

    Our tournament scoring was groundbreaking when we introduced it in 2021. It served us well over the last 3 years, aligning incentives and distributing more than $50,000 in prizes on dozens of Tournaments.

    The new system offers a few key advantages:

    • Simplicity. Many users told us that the old system was too complex. The new system is simpler.
    • Consistency. The new system is based on the Peer score, just like medals.
    • Prize distribution. Some past tournaments saw a single user win 90%+ of the prize pool, which discouraged participation for those who fell behind early. Back-testing on past tournament data tells us the new system would have rewarded more of the top forecasters.

    How does it work?

    In words:

    You receive a Peer Score for each tournament question, with zeros for any you don’t predict. Your Total Score is the sum. If it’s positive, congratulations: you just won a share of the prize pool. Your share is proportional to the square of your Total Score. Your Total Score also determines your tournament ranking.

    Not in words:

    What does it mean?

    To maximize your Total Score, always predict your honest belief.


    • Forecasting all questions and forecasting early are now more important. Previously, predicting half of a tournament’s questions or joining halfway through would win you ~39% of the prize you would’ve otherwise won. Now it wins you ~25%.

    • If we apply the new scoring to past tournaments, we find on average the top forecaster would have won a smaller prize, but the top 10 would have cumulatively won more prize. We expect far fewer cases where the top forecaster receives 90%+ of the prize.

    We think those design choices have better incentives and reward users for forecasting more. Read about the trade-offs and alternatives we considered here: New Tournament Scoring: Trade-offs, Decisions.


    Imagine a tournament with 5 forecasters, 30 questions, and a prize pool of $1000.

    • The first forecaster is excellent and always achieves a Peer score of +40.
    • The second forecaster is just as good, but joined halfway through.
    • The third forecaster is good and always gets +10.
    • The fourth forecaster is bad and always gets -20.
    • The fifth forecaster is quite poor and always gets -50.

    Their scores and winnings are:

    Forecaster Total score Take %Prize Prize
    A 40×30=1200 1200²=1,440,000 76.2% $762
    B 40×30×½=600 600²=360,000 19.0% $190
    C 10×30=300 300²=90,000 4.8% $48
    D -20×30=-600 0 0% $0
    E -50×30=-1500 0 0% $0
    sum 0 1,890,000 100% $1000

    Current tournaments

    Some tournaments will run for years to come. We don’t want to retain both scoring systems for longer than necessary. We also appreciate that you may have had certain scoring expectations at the time of your forecasts. To balance these, most tournaments will continue using the legacy scoring system, except for tournaments that:

    1. are very long term (and unfinished).
    2. have few or no resolved questions.

    Here are the full lists:

    Tournaments retaining the old scoring system Tournaments using the new scoring system
    Alt-Protein Tournament Cultured Meat Tournament
    Keep Virginia Safe Tournament Climate Tipping Points
    Trade Signal Tournament 🛰 The Sagan Space Tournament 🛰
    NuclearRisk Tournament FRO-casting
    Virginia Lightning Round Tournament Conditional Cup
    Real-Time Pandemic Decision Making Respiratory Outlook 2023/24
    Global Trends 2022 Chinese AI Chips
    FluSight Challenge 2021/22 ACX 2024 Prediction Contest
    China and Global Cooperation
    Ukraine Conflict Any new tournament that opens in the future
    White Hat Cyber
    Economist 2021
    Humanitarian Conflict & Economic Risk
    FluSight Challenge 2022/23
    🔰Q4 2022 Beginner Tournament 🔰
    ACX 2023 Prediction Contest
    🔰Q1 2023 Beginner Tournament 🔰
    Keep Virginia Safe II
    📰 Breaking News Tournament 📰
    🔰Q2 2023 Beginner Tournament 🔰
    🏆 Q3 2023 Quarterly Cup 🏆
    FluSight Challenge 2023/24
    🏆 Q4 2023 Quarterly Cup 🏆
    Global Pulse Tournament
    🏆 Quarterly Cup 🏆

    Two special cases not mentioned above:

    1. Biosecurity Tournament: the $15,000 for the 2024 questions will be awarded according to the old scoring system. The rest of the prize ($10,000) for the 2032 questions will use the new scoring system.

    2. Forecasting Our World in Data: the $8,000 awarded in 2025 for the 1-year questions will use the old scoring system. The other $8,000 awarded in 2027 for the 3-year questions will use the new scoring system.

    Note that the current (Q1 2024) Quarterly Cup will use the old scoring system, but future iterations will use the new system.

    For simplicity, all Question Series will switch to the new system, since they don’t award prizes or medals.

    If you're unsure which scoring system a particular tournament uses, open its leaderboard: if there is a "Coverage" column, then the tournament is using the legacy scoring. The new scoring system does not use the Coverage, and omits that column.

    We look forward to reading your feedback, questions, and ideas below!

    Submit Essay

    Once you submit your essay, you can no longer edit it.