As you might remember, Metaculus passed the 1 million prediction mark this fall. To mark the occasion, we hosted our first ever hackathon. We invited data scientists, mathematicians, researchers, and forecasters in our community to analyze our complete dataset for the first time ever, and offered over $10,000 in prizes for the best projects.
We ultimately accepted 30 applications from individuals around the world, who worked solo and in teams on projects in the following categories:
- Aggregation & Accuracy: When is the Metaculus forecast much more, or much less, accurate than average? What novel ways to aggregate forecasts might improve accuracy?
- Evaluating Forecasters: What do top forecasters have in common? Are there early indications someone will be a top forecaster? What other scoring metrics may be useful?
- Open Inquiry: Provided an opportunity to delve into the Metaculus data and provide a creative and innovative analysis.
We were impressed by the creativity and hard work displayed by all of the participants, and selecting the top projects was no easy task. We are thrilled to announce the winners; first place projects will receive $2000, second place $1000, and third place $500 in each category.
Aggregation and Accuracy
Make Number Go Up; Three Ways to Improve the M-Forecast - Peter Wildeford
Peter developed a new aggregation method, using the extremization-of-mean-log-odds-based method, with only certain predictions getting extremized, that outperforms the MP.
Info Diversity and Extremization - Vasily Artyukhov
Vasily looked into extremizing the CP by an amount that depends on temporal clustering of forecasts and found that extremizing more when forecasts are highly clustered helps somewhat.
Exploring Alternatives to Metaculus Flagship Aggregation - Javier Prieto, Sarthak Agrawal
Javier and Sarthak worked on a new aggregation method with weighting done via the inner product of a feature vector including reputation, recency, average update frequency and average update magnitude.
Evaluating Individual Forecasters
Early Talent Spotting and Better Leaderboards - Vasily Artyukhov
In his second project, Vasily looked into proxies for forecaster quality which are motivated by the idea that a good forecaster today is ahead of the community tomorrow and imputed default values on questions users didn't forecast on by setting up the problem as a sparse matrix factorization problem.
Forecaster Median Empirical Ranking Reconstruction - Giancarlo Vercellino
Giancarlo defined new measures of forecaster performance and used ML (random forests) to reconstruct these metrics from features like normalized time until resolution, category, number of predictions in that category, etc.
Forecast-Til-You-Die Model - Younes Jeddi
Younes adapted a consumer behavior model to a forecasting context. This model makes probabilistic predictions about how likely a forecaster is still active as a function of various quantities involving time since first/last prediction and frequency of predictions.
Can Money Buy Accuracy? (Tournament Investigation) - Ellie Litwack
Ellie conducted a deep dive into tournament behavior, comparing accuracy and participation on tournament questions versus non-tournament questions.
Using Metaculus to Get Updates on the World - Aadil Kara, Sung Soo Moo
Aadil and Sung Soo created a novel dashboard showing changes in the Community Prediction alongside news events and coverage.
Predictors of Question Quality - Andrew Tweddle, Niklas Lehmann, Rike Becker
Andrew, Niklas, and Rike performed a series of interesting regressions examining the relation of question features to question accuracy.
Congratulations to the winning teams and all of the participants for their outstanding efforts. We cannot wait to see what the next hackathon brings!