Metaculus
News
When will AI arrive?
By Anthony on Jan. 2, 2017, noon GMT
From the game of Go to self-driving cars, there is widespread agreement that progress in artificial intelligence has accelerated in recent years. Yet there is remarkably little consensus as to what this progress portends in the coming years or decades. This is especially true of the big question of if and when we will have general-purpose human (or super-human) level AI. I’ve seen estimates by credible AI researchers range from ten years to “never”, and everywhere in-between!
While difficult, this question is extremely important. Humans have (for better and worse) agency over the fate of planet Earth due to our intellectual capability and technology; as we create systems with comparable capabilities on various axes, it will profoundly change the course of society and perhaps life itself.
Toward making good decisions and allocating resources wisely, it is vital that we have some ability to predict how various types of AI systems and capabilities will unfold; that is the aim of the new “AI Milestones” question series on Metaculus.
Lots of newly-resolved questions
By Anthony on Jan. 1, 2017, noon GMT
Happy new year!
A whole boatload of questions resolved in the last couple of weeks of December (many of the form “Will X happen in 2016?”. Somewhat bizarrely, just about all of them (see the list here) resolved negative! In the great majority of cases, these negative resolutions were forecast correctly. I guess we all have to come up with some more likely things to predict about…
The leaderboard has now moved around a lot, and a couple have now attained “Predictor” status, homing in on “superpredictor.” We may even have a seer or a prophet soon!
I looked for a good numerical index with which to ask the question “will 2017 be better than 2016?”, but could not see a great candidate — if anyone else does please leave a comment, or just suggest a question onsite.
Me-calculus: Continuous predictions arrive
Although a lot can be learned from asking about whether or not various things will occur in the future, there are many cases where it would be much nicer to predict a number, such as a date by which something happens, or an amount of money, sales, citations, hits, etc., etc.
As of now, such questions can be posed on Metaculus. Rather than input a probability of yes, users may now define a probability distribution, i.e. a probability assigned to each possible value within a defined range. You get more points if you assign a higher probability to the actual realized number, but you have a fixed amount of probability to assign; thus you can make a narrow distribution if you are fairly certain of a particular value, and a wider distribution to reflect more uncertainty. See the FAQ for a lot of detail on how this works, and how these questions are scored at resolution.
Take the Research Impact Challenge
By Anthony on Dec. 27, 2016, noon GMT
One of the motivations behind creating Metaculus is to see whether researchers and others can get skilled at predicting the impact that newly-announced research will have, as I discussed here. When a new paper appears as a preprint or published research, and news stories start to appear about it, it can be very difficult for the public (or science writers, or even other researchers) to develop a useful sense of its impact and importance. Academic institutions and researchers, as well as journalists, are all motivated to maximize the perceived importance and interest of each story. (How many game-changing cancer, Parkinson’s and Alzheimer’s research breakthroughs have you seen in the news over the last years?)
One role Metaculus can play is to solicit and aggregate predictions as to how big a deal new science and tech stories will actually turn out to be; wouldn’t it be useful to know? It’s not easy to accurately quantify impact, and there are various metrics that could be used. Probably simplest (and widely adopted even if flawed) is by citations. So we’ve launched a number of questions asking how many citations some recent interesting and newsy papers will acquire over coming months. Here are a few:
Metaculus on the election
By Anthony on Nov. 15, 2016, noon GMT
Our community of predictors is poised to surpass 3,000, and we are continuing to experience steady growth, with close to 25,000 predictions now on the books.
Metaculus was not developed with political questions as its central goal, but there was no avoiding the 2016 election and its public interest. We ended up with a number of election-related questions that just resolved:
Importance of final predictions restored
By Anthony on Oct. 23, 2016, noon GMT
An unintended side-effect of the new scoring scheme, which averages over points on-the-line between question opening and closing, was that putting in a prediction right before closing makes hardly any effect. This is unfortunately both because it makes things less fun and exciting, and also because the community prediction at closing time is probably the most useful piece of information created by the community on each question, and should be optimized.
Thus in a slight tweak to the scoring scheme, we’ve now added extra weight to your last prediction; this makes the current scheme a hybrid between the original scheme (where all the weight was on the final prediction before closing) and the time-averaged scheme. The weighting is roughly 2/3 for the time-averaged part, 1/3 for the final prediction.
While the means to a best score is still to predict accurately, early, and often, this will hopefully help predictors better allocate their efforts.
New “activity” sorting metric
In a small, but we think quite useful, upgrade, sorting by “activity” now sorts in a more sophisticated way:
- Each question has an activity score A
- When a new prediction is made, this increments A by one.
- When a new comment is made, this increments A by 10.
- A decays exponentially with a half-life of 12 hours.
Sorting by “activity” now does a descending sort in A. In practice, questions with new comments should be at the top, questions with more new comments will be even higher, and questions with a lot of new predictions will also feature high on the list.
Also, a sort of all questions by activity is now the default under “featured.”
Let us know how this is working — we can easily tweak the parameters.
Feature request forum
We’ve got a long list of features we plan to implement in Metaculus, including:
- A better aggregation scheme in action
- Better sorting of “activity” and “interest”
- Predictions of numerical and multi-outcome questions
- A system to help better crowd-process suggested questions into launched ones
- More track-record elements for each user on the profile page
- and many more in the queue…
On top of our list, we’d love to hear from you! Relatively small tweaks might even get implemented in short order. Also, if there is something that frustrates or irritates you about the system, please let us know, and we’ll see if we can come up with a way to improve it (and welcome your ideas there too!)
Analysis of 20,000 predictions
By Anthony on Oct. 15, 2016, 2 p.m. GMT
Metaculus users passed 20,000 predictions recently, and it seems time for a bit of analysis. Though there are predictions on about 300 questions total, this will concern the first 48 questions to resolve.
The figure above shows the ‘calibration plot’ for those 48 questions, based on the better predictors. This divvies up all of the predictions into bins of 1-5% likely, 5-10% likely, etc. For each ‘bin’, we plot what fraction of those predictions correspond to questions for which the answer was “yes.” What you’d really like is for all predictions to be either “0% probable” or “100% probable”, and that all of the 0% ones turn out not to happen, and all the 100% ones do happen. But that would require a time machine. Given our imperfect view of the future, we can at least hope that of the issues that are, say, 20% probable, 20% of them actually come true. If so, those percentages are actually meaningful, and can be used to make quantitative decisions.