Metaculus
News
Metaculus is hiring!
*Update Jan 1, 2018: This position has now been filled.
Site updates: News, Tachyons, Tutorials, and more
By max.wainwright on Nov. 11, 2017, 6:10 p.m. GMT
There have been a lot of updates to the site over the past few weeks. Some are big and obvious, and some are less so. We hope that they collectively make Metaculus more fun and easier to use.
First, and most obvious to those reading this, we've brought the Metaculus blog onto the Metaculus site. Feel free to leave comments on news postings just as you would for Metaculus questions! You can always find the news in a link at the bottom of any page.
The biggest update, however, is the introduction of tachyons. Tachyons are Metaculus's in-game currency which can be used to go backwards in time to remove or change an old prediction, or to zoom forwards and sneak a peak at the Metaculus prediction before it closes. You can currently earn weekly tachyons just by logging in, and bonus tachyons by completing achievements or tutorials. We have more plans for tachyonic uses and how one might earn them more quickly or trade them with other players, but we'd love to hear your feedback and ideas too.
Cryptocurrency competition status
It's been interesting, educational, and frustrating in equal measures to see the cryptocurrency competition play out.
When will AI arrive?
By Anthony on Jan. 2, 2017, noon GMT
From the game of Go to self-driving cars, there is widespread agreement that progress in artificial intelligence has accelerated in recent years. Yet there is remarkably little consensus as to what this progress portends in the coming years or decades. This is especially true of the big question of if and when we will have general-purpose human (or super-human) level AI. I’ve seen estimates by credible AI researchers range from ten years to “never”, and everywhere in-between!
While difficult, this question is extremely important. Humans have (for better and worse) agency over the fate of planet Earth due to our intellectual capability and technology; as we create systems with comparable capabilities on various axes, it will profoundly change the course of society and perhaps life itself.
Toward making good decisions and allocating resources wisely, it is vital that we have some ability to predict how various types of AI systems and capabilities will unfold; that is the aim of the new “AI Milestones” question series on Metaculus.
Lots of newly-resolved questions
By Anthony on Jan. 1, 2017, noon GMT
Happy new year!
A whole boatload of questions resolved in the last couple of weeks of December (many of the form “Will X happen in 2016?”. Somewhat bizarrely, just about all of them (see the list here) resolved negative! In the great majority of cases, these negative resolutions were forecast correctly. I guess we all have to come up with some more likely things to predict about…
The leaderboard has now moved around a lot, and a couple have now attained “Predictor” status, homing in on “superpredictor.” We may even have a seer or a prophet soon!
I looked for a good numerical index with which to ask the question “will 2017 be better than 2016?”, but could not see a great candidate — if anyone else does please leave a comment, or just suggest a question onsite.
Me-calculus: Continuous predictions arrive
Although a lot can be learned from asking about whether or not various things will occur in the future, there are many cases where it would be much nicer to predict a number, such as a date by which something happens, or an amount of money, sales, citations, hits, etc., etc.
As of now, such questions can be posed on Metaculus. Rather than input a probability of yes, users may now define a probability distribution, i.e. a probability assigned to each possible value within a defined range. You get more points if you assign a higher probability to the actual realized number, but you have a fixed amount of probability to assign; thus you can make a narrow distribution if you are fairly certain of a particular value, and a wider distribution to reflect more uncertainty. See the FAQ for a lot of detail on how this works, and how these questions are scored at resolution.
Take the Research Impact Challenge
By Anthony on Dec. 27, 2016, noon GMT
One of the motivations behind creating Metaculus is to see whether researchers and others can get skilled at predicting the impact that newly-announced research will have, as I discussed here. When a new paper appears as a preprint or published research, and news stories start to appear about it, it can be very difficult for the public (or science writers, or even other researchers) to develop a useful sense of its impact and importance. Academic institutions and researchers, as well as journalists, are all motivated to maximize the perceived importance and interest of each story. (How many game-changing cancer, Parkinson’s and Alzheimer’s research breakthroughs have you seen in the news over the last years?)
One role Metaculus can play is to solicit and aggregate predictions as to how big a deal new science and tech stories will actually turn out to be; wouldn’t it be useful to know? It’s not easy to accurately quantify impact, and there are various metrics that could be used. Probably simplest (and widely adopted even if flawed) is by citations. So we’ve launched a number of questions asking how many citations some recent interesting and newsy papers will acquire over coming months. Here are a few:
Metaculus on the election
By Anthony on Nov. 15, 2016, noon GMT
Our community of predictors is poised to surpass 3,000, and we are continuing to experience steady growth, with close to 25,000 predictions now on the books.
Metaculus was not developed with political questions as its central goal, but there was no avoiding the 2016 election and its public interest. We ended up with a number of election-related questions that just resolved:
Importance of final predictions restored
By Anthony on Oct. 23, 2016, noon GMT
An unintended side-effect of the new scoring scheme, which averages over points on-the-line between question opening and closing, was that putting in a prediction right before closing makes hardly any effect. This is unfortunately both because it makes things less fun and exciting, and also because the community prediction at closing time is probably the most useful piece of information created by the community on each question, and should be optimized.
Thus in a slight tweak to the scoring scheme, we’ve now added extra weight to your last prediction; this makes the current scheme a hybrid between the original scheme (where all the weight was on the final prediction before closing) and the time-averaged scheme. The weighting is roughly 2/3 for the time-averaged part, 1/3 for the final prediction.
While the means to a best score is still to predict accurately, early, and often, this will hopefully help predictors better allocate their efforts.