Metaculus
News
New “activity” sorting metric
In a small, but we think quite useful, upgrade, sorting by “activity” now sorts in a more sophisticated way:
- Each question has an activity score A
- When a new prediction is made, this increments A by one.
- When a new comment is made, this increments A by 10.
- A decays exponentially with a half-life of 12 hours.
Sorting by “activity” now does a descending sort in A. In practice, questions with new comments should be at the top, questions with more new comments will be even higher, and questions with a lot of new predictions will also feature high on the list.
Also, a sort of all questions by activity is now the default under “featured.”
Let us know how this is working — we can easily tweak the parameters.
Feature request forum
We’ve got a long list of features we plan to implement in Metaculus, including:
- A better aggregation scheme in action
- Better sorting of “activity” and “interest”
- Predictions of numerical and multi-outcome questions
- A system to help better crowd-process suggested questions into launched ones
- More track-record elements for each user on the profile page
- and many more in the queue…
On top of our list, we’d love to hear from you! Relatively small tweaks might even get implemented in short order. Also, if there is something that frustrates or irritates you about the system, please let us know, and we’ll see if we can come up with a way to improve it (and welcome your ideas there too!)
Analysis of 20,000 predictions
By Anthony on Oct. 15, 2016, 2 p.m. GMT
Metaculus users passed 20,000 predictions recently, and it seems time for a bit of analysis. Though there are predictions on about 300 questions total, this will concern the first 48 questions to resolve.
The figure above shows the ‘calibration plot’ for those 48 questions, based on the better predictors. This divvies up all of the predictions into bins of 1-5% likely, 5-10% likely, etc. For each ‘bin’, we plot what fraction of those predictions correspond to questions for which the answer was “yes.” What you’d really like is for all predictions to be either “0% probable” or “100% probable”, and that all of the 0% ones turn out not to happen, and all the 100% ones do happen. But that would require a time machine. Given our imperfect view of the future, we can at least hope that of the issues that are, say, 20% probable, 20% of them actually come true. If so, those percentages are actually meaningful, and can be used to make quantitative decisions.
New scoring system
By Anthony on Oct. 15, 2016, 1 p.m. GMT
We’ve recently rolled out a significant overhaul of the way scoring works, which we believe will lead to more accurate predictions, and more overall fun.
The update to the scoring system is accompanied by some behind-the-scenes changes that will allow an upcoming expansion of the types of questions we can pose. A full description of the scoring system can be found at the FAQ. Here’s what you need to know:
- You are still best off entering your best estimate of the outcome probability given all available information.
- Scoring is still based on a combination of being right, and also being more right than others, though you’ll see that in detail the numbers have changed a bit. What’s new is that your awarded points are a time-average of your point on-the-line at any given time, starting at question open and ending at question close. This means that you will maximize your points if:
- You predict as early as possible, since your points on-the-line is zero until your first prediction, and
- You update your prediction whenever you have reason to think your best estimate of the probability has changed.
- As before, you dial in your prediction using the slider, and you’ll see that points on-the-line change as you slide it. Once you make a prediction you will also see a “score history” tab that will show you how your points vary with time, in reflection of changes in your prediction, the community’s prediction, and the number of predictors.
Welcome to the Metaculus blog!
By Anthony on Oct. 15, 2016, noon GMT
At long last, we’ve launched this blog to host posts related to Metaculus, including:
- Announcements of updates and features
- Discussion of upcoming features
- Data and analysis of Metaculus’s prediction track record
- Analysis and discussion of results and predictions spanning multiple questions
- Discussions about forecasting and prediction in general
We hope you’ll enjoy the discussion, and look forward to your feedback!