assembling accurate understanding predicting intelligent predictions generating precise understanding mapping quantitative contingencies forecasting calibrated insights mapping the future aggregating calibrated forecasts composing quantitative forecasts forecasting intelligent forecasts modeling predictive estimations mapping probable forecasts forecasting intelligent forecasts generating predictive estimations calculating accurate futures


The Bentham prize

By Tamay on Jan. 15, 2020, 6:58 p.m. GMT

We will be awarding bi-weekly prizes of $300, $200 and $100 in Amazon gift cards, as the first, second and third prize to the commenters who make the most valuable contributions to forecasting the questions in Metaculus' Animal Welfare series, in the periods:

Round 1: January 23, 00:00 GMT to February 6 (edit: changed to February 3rd), 23:59 GMT

  • For round 1, all comments in with the "Animal Welfare Series" category tag (i.e. all of these questions) will be considered toward overall contributions.

  • Round 2: To be announced.

  • Details to follow.
  • Evaluation Criteria

    Roughly, a good commenter is someone who contributes to our ability to make accurate and well-calibrated predictions. This might take the shape of a single beacon of good work, or a longer series of contributions.

    Examples of useful contributions are carefully thought-out public predictions, models, factorisations, comments that help improve questions, datasets, or links to relevant sources.

  • Public predicting
  • Regularly posting their predictions as comments, and sharing the reasoning that went in to produce it (using the “post prediction as comment” feature that appears after you’ve predicted). Good public predictions are usually accompanied by supporting evidence, analysis or references to other predictions made elsewhere. This 'evidence' may be as simple as "linear extrapolation suggests X", or "betting markets suggest Y", or as sophisticated as developing a Monte-Carlo simulation.

    Examples of public predictions

    Will the Chinese police or military intervene in Hong Kong in August 2019?

    25% notany made a prediction on Aug 14, 2019 at 12:50pm

    While the resistance is ongoing, number of people involved has declined. If China intervenes and sends in PAP (The Chinese People's Armed Police Force) they can control the streets but the political and economic disruption may be larger than the riots. More importantly they can't control other forms of less direct resistance that may emerge.

    Just to give a some numbers: In 2008 38% of the young (aged 18-29) living in the HK identified as Chinese. In 2019 it's only 3%.
    Democracy in Crisis: How many free countries in 2028?

    82 – 91 AABoyles made a prediction on Dec 16, 2019 at 5:28pm

    My intuition is that the world is trending better, and a local downturn in social and political will be resolved within the observed timeframe (much as the 2008 Great Recession wasn't a significant factor in the economic dynamics of 2018). Linear forecasting from Tamay's spreadsheet (which I've modified somewhat) suggests this is broadly true, but the implied forecast (~102) seems much too optimistic (as the community prediction seems to indicate). Accordingly, I place a large component around my gut point estimate (90).

    But I imagine I could be horribly wrong, and the world is slipping into a much less free future. Accordingly, I added a wide component in the 80-90 range to keep my score minimally positive in these bad futures.

  • Models and factorisations
  • Explicit analytical or statistical models can be powerful tools to guide one's forecasting. Simple techniques such as linear extrapolation can often help make good forecasts. Or simply listing all the relevant factors that need to be considered may prove useful.

    Examples of factorisations and modelling

    What percent of undergrads at select US universities will be enrolled in introductory ML classes?

    holomanga updated a prediction on Aug 17, 2019 at 10:51pm

    Linear fit to the data given (done in MATLAB) gives 4.8% to 6.0% in 2019.
    Will global malaria mortality rates be reduced by 90% when compared with 2015 rates, by 2030?

    Jotto updated a prediction on Jun 23, 2019 at 11:19pm

    There is a whole pile of factors that one could update on. Some candidates off the top of my head:
    1. Bed nets
    2. Resistance to pyrethroids (the insecticide used in the anti-malaria bednets)
    3. More housing development in malaria-riddled areas (has a similar effect as the bed nets)
    4. Improvements in anti-malaria medications
    5. Anti-malaria vaccines using new mechanisms, or tweaks to old ones
    6. The general global economy (e.g., a recession could reduce donations for the anti-malaria effort, and a further weakening of local economies in poor places would make it harder to weather the effects of contracting malaria)
    7. A gene drive to eradicate the types of mosquitos that give people malaria
    8. Changes to funding for the President’s Malaria Initiative, which is a major actor in this space
    9. Changes in the proportions between different at-risk groups. For example, suppose if refugees as a group were somewhat more likely to die of malaria than non-refugees, and suppose if five year olds as a group were even more prone to dying from malaria than refugees. Assume no correlation between those two factors (i.e., supposing refugees are equally likely to be five year olds as non-refugees). In that case, a scenario where the ratio between five year olds to refugees decreased would have an expected decrease in malaria mortality.
    10. Others
    It intuitively seems to me that the anti-malaria developments are unlikely to dramatically reduce malaria within 11 years. On the other hand, it seems very plausible that they will contribute to the recent downtrend continuing.

  • Compiling datasets
  • Creating and sharing datasets can be an invaluable public good. As such, we hope that our prize will result in you internalising some of its social value.

    Examples of dataset compiling

    Will the S&P 500 Index end 2020 higher?

    AABoyles provided an information source on Jan 2, 2020 at 11:45am

    Since 1927, $SPX has closed the year up 61 times and down 31 times.

  • Linking and distilling relevant sources
  • Examples of useful links

    Will the Democratic candidate for the 2020 US presidential election be chosen at the Democratic National Convention after only one round of voting?

    DanielFilan (☆☆) pointed to an analysis 10 hours 35 min ago

    FiveThirtyEight's Democratic primary model currently assigns a 14% chance of no candidate winning a majority of pledged delegates in the caucuses and primaries.

    What percentage of Americans will be considered obese or overweight in 2030?

    gjm commented on Dec 19, 2019 at 11:04pm… predicts that by 2030 ~49% of US adults will be obese. The abstract readable there suggests that the full paper has projections for mere overweightness as well, but doesn't say what those projections are; the full paper is paywalled. username

  • Writing new questions and improving existing questions
  • The writing of new questions that generate useful insight on existing questions, or related topics are appreciated.

    Moreover, existing questions can sometimes be ambiguous, or its resolution conditions may be under-specified or unclear. It is useful for this to be pointed out, and to receive suggestions on how the question could be improved.

    Examples of comments that helped improve questions

    Will any OECD country achieve a 10% or greater reduction in the national rate of obesity before 2030?

    PabloStafforini discussed the question resolution on Dec 15, 2019 at 10:02pm

    How does the question resolve if the definition of obesity changes, and the 10% condition is satisfied according to one definition but not the other?
    How many billions of poultry will be slaughtered worldwide, in the calendar year 2030, according to FAO estimates?

    katifish requested information on Dec 17, 2019 at 1:54pm

    I'm curious about the discrepancy between the results you get from searching the data at and the visualization linked in the resolution criteria. Just focusing on chickens, the visualization shows ~22.8 billion, whereas the database search give ~66.6 billion. Does anyone know why this is?

    This is not a checklist. We can imagine awarding the prize to someone who has written only a handful of comments, or to someone who has written dozens of one-sentence comments; to someone who has done only one of the above or someone who has done many. The key determinant will be (our estimation of) the value of the contributions made.

    For the first round of the contest, the judges will be Daniel Filan, Pablo Stafforini and Tamay Besiroglu.

    Terms and conditions
    • The Contest is open to everybody except except Contest judges
    • Sponsors reserve the right to cancel or modify this Contest in the event that an insufficient number of entries are received that meet the minimum judging criteria.
    • This Contest shall be construed in accordance with U.S. law. All Federal, state and local laws and regulations apply.
    • Winners are required to pay all appliable taxes on the prize and Metaculus assumes no liability for such taxes.
    • Void where prohibited by law. Metaculus reserves the right to refuse to award any Prize if doing so violates any applicable laws.
    • All decisions of the judges are final and the selection of Winners is at the sole and absolute discretion of Metaculus.