M

This Discussion is now a Draft.

Once it's ready, please submit it for review by our team of Community Moderators. Thank you!

Pending

This Discussion now needs to be approved by community moderators.

The Million Prediction Hackathon Winners

As you might remember, Metaculus passed the 1 million prediction mark this fall. To mark the occasion, we hosted our first ever hackathon. We invited data scientists, mathematicians, researchers, and forecasters in our community to analyze our complete dataset for the first time ever, and offered over $10,000 in prizes for the best projects.

We ultimately accepted 30 applications from individuals around the world, who worked solo and in teams on projects in the following categories:

  • Aggregation & Accuracy: When is the Metaculus forecast much more, or much less, accurate than average? What novel ways to aggregate forecasts might improve accuracy?
  • Evaluating Forecasters: What do top forecasters have in common? Are there early indications someone will be a top forecaster? What other scoring metrics may be useful?
  • Open Inquiry: Provided an opportunity to delve into the Metaculus data and provide a creative and innovative analysis.

We were impressed by the creativity and hard work displayed by all of the participants, and selecting the top projects was no easy task. We are thrilled to announce the winners; first place projects will receive $2000, second place $1000, and third place $500 in each category.

Aggregation and Accuracy

  1. Make Number Go Up; Three Ways to Improve the M-Forecast - Peter Wildeford

    Peter developed a new aggregation method, using the extremization-of-mean-log-odds-based method, with only certain predictions getting extremized, that outperforms the MP.

  2. Info Diversity and Extremization - Vasily Artyukhov

    Vasily looked into extremizing the CP by an amount that depends on temporal clustering of forecasts and found that extremizing more when forecasts are highly clustered helps somewhat.

  3. Exploring Alternatives to Metaculus Flagship Aggregation - Javier Prieto, Sarthak Agrawal

    Javier and Sarthak worked on a new aggregation method with weighting done via the inner product of a feature vector including reputation, recency, average update frequency and average update magnitude.

Evaluating Individual Forecasters

  1. Early Talent Spotting and Better Leaderboards - Vasily Artyukhov

    In his second project, Vasily looked into proxies for forecaster quality which are motivated by the idea that a good forecaster today is ahead of the community tomorrow and imputed default values on questions users didn't forecast on by setting up the problem as a sparse matrix factorization problem.

  2. Forecaster Median Empirical Ranking Reconstruction - Giancarlo Vercellino

    Giancarlo defined new measures of forecaster performance and used ML (random forests) to reconstruct these metrics from features like normalized time until resolution, category, number of predictions in that category, etc.

  3. Forecast-Til-You-Die Model - Younes Jeddi

    Younes adapted a consumer behavior model to a forecasting context. This model makes probabilistic predictions about how likely a forecaster is still active as a function of various quantities involving time since first/last prediction and frequency of predictions.

Open Inquiry

  1. Can Money Buy Accuracy? (Tournament Investigation) - Ellie Litwack

    Ellie conducted a deep dive into tournament behavior, comparing accuracy and participation on tournament questions versus non-tournament questions.

  2. Using Metaculus to Get Updates on the World - Aadil Kara, Sung Soo Moo

    Aadil and Sung Soo created a novel dashboard showing changes in the Community Prediction alongside news events and coverage.

  3. Predictors of Question Quality - Andrew Tweddle, Niklas Lehmann, Rike Becker

    Andrew, Niklas, and Rike performed a series of interesting regressions examining the relation of question features to question accuracy.

Congratulations to the winning teams and all of the participants for their outstanding efforts. We cannot wait to see what the next hackathon brings!

Categories:
Metaculus