Metaculus
News
Analysis of 20,000 predictions
By Anthony on Oct. 15, 2016, 2 p.m. GMT
Metaculus users passed 20,000 predictions recently, and it seems time for a bit of analysis. Though there are predictions on about 300 questions total, this will concern the first 48 questions to resolve.
The figure above shows the ‘calibration plot’ for those 48 questions, based on the better predictors. This divvies up all of the predictions into bins of 1-5% likely, 5-10% likely, etc. For each ‘bin’, we plot what fraction of those predictions correspond to questions for which the answer was “yes.” What you’d really like is for all predictions to be either “0% probable” or “100% probable”, and that all of the 0% ones turn out not to happen, and all the 100% ones do happen. But that would require a time machine. Given our imperfect view of the future, we can at least hope that of the issues that are, say, 20% probable, 20% of them actually come true. If so, those percentages are actually meaningful, and can be used to make quantitative decisions.
New scoring system
By Anthony on Oct. 15, 2016, 1 p.m. GMT
We’ve recently rolled out a significant overhaul of the way scoring works, which we believe will lead to more accurate predictions, and more overall fun.
The update to the scoring system is accompanied by some behind-the-scenes changes that will allow an upcoming expansion of the types of questions we can pose. A full description of the scoring system can be found at the FAQ. Here’s what you need to know:
- You are still best off entering your best estimate of the outcome probability given all available information.
- Scoring is still based on a combination of being right, and also being more right than others, though you’ll see that in detail the numbers have changed a bit. What’s new is that your awarded points are a time-average of your point on-the-line at any given time, starting at question open and ending at question close. This means that you will maximize your points if:
- You predict as early as possible, since your points on-the-line is zero until your first prediction, and
- You update your prediction whenever you have reason to think your best estimate of the probability has changed.
- As before, you dial in your prediction using the slider, and you’ll see that points on-the-line change as you slide it. Once you make a prediction you will also see a “score history” tab that will show you how your points vary with time, in reflection of changes in your prediction, the community’s prediction, and the number of predictors.
Welcome to the Metaculus blog!
By Anthony on Oct. 15, 2016, noon GMT
At long last, we’ve launched this blog to host posts related to Metaculus, including:
- Announcements of updates and features
- Discussion of upcoming features
- Data and analysis of Metaculus’s prediction track record
- Analysis and discussion of results and predictions spanning multiple questions
- Discussions about forecasting and prediction in general
We hope you’ll enjoy the discussion, and look forward to your feedback!