Metaculus
News
Track record improvements
We've made some improvements to the Metaculus track record page: questions can now be filtered by date and question category, making it easier to drill down into the data. For example, it's easy to see that in the last 6 months the Metaculus prediction has outperformed the community prediction with a Brier score that's 33% lower. It's also easy to see that Metaculus performed especially well on biological questions (Brier score 75% lower than the community's), and both the Metaculus and community predictions missed the mark for cryptocurrency questions (both with scores > 0.24). In addition to the filters, there's now a histogram to clearly show the distributions of scores.
Cryptocurrency competition analysis
The results are in, and a big congrations go to our winners jzima and Barbarossa and our runners up AndrewMcKnight and ElliotOlds. But how did everyone else do? In short, not so well.
Our winner received 748 points, but the median score amongst all cryptocurrency players was only 7 points and the average score was way down at −73 points. Only 25 players broke 100 points. Player overconfidence combined with our log scoring system really hurt here: just one very overconfident prediction could easily send someone's score tumbling down from the top of the ranks. That said, there was a top tier of eight players who got at least 590 points (with a large gap separating the 9th player), and, as we'll see below, they used different confidence strategies to get there.
Cryptocurrency competition results!
After some internal deliberating and data crunching, we're finally ready to announce the results of the 2017 Cryptocurrency Prediction Competition. First we'll say a few words on the reason for delay, then the results.
As those following the competition know, we got stalled on four questions that were based on the highest or total amount amount of initial coin offerings (ICOs) in October and November. It turns out that the data for ICOs is terribly incomplete and unreliable, with numbers dramatically varying from one source to another. On top of this, it seems fairly clear that there is an order-of-magnitude uncertainty in the ICO funds actually raised by Paragon. In the end, we decided to resolve these questions per their letter, keeping with Metaculus's general policy. However, in recognition that these questions could be considered flawed because based on a misleading resource, we've decided to also issue prizes to the two predictors who would have won had we resolved these four questions as ambiguous.
So, without further ado, the winners are:
Lots of predictions from Fortune that could be Metaculized
Fortune mag just came out with a big list of predictions.
Metaculus is hiring!
*Update Jan 1, 2018: This position has now been filled.
Site updates: News, Tachyons, Tutorials, and more
By max.wainwright on Nov. 11, 2017, 6:10 p.m. GMT
There have been a lot of updates to the site over the past few weeks. Some are big and obvious, and some are less so. We hope that they collectively make Metaculus more fun and easier to use.
First, and most obvious to those reading this, we've brought the Metaculus blog onto the Metaculus site. Feel free to leave comments on news postings just as you would for Metaculus questions! You can always find the news in a link at the bottom of any page.
The biggest update, however, is the introduction of tachyons. Tachyons are Metaculus's in-game currency which can be used to go backwards in time to remove or change an old prediction, or to zoom forwards and sneak a peak at the Metaculus prediction before it closes. You can currently earn weekly tachyons just by logging in, and bonus tachyons by completing achievements or tutorials. We have more plans for tachyonic uses and how one might earn them more quickly or trade them with other players, but we'd love to hear your feedback and ideas too.
Cryptocurrency competition status
It's been interesting, educational, and frustrating in equal measures to see the cryptocurrency competition play out.
When will AI arrive?
By Anthony on Jan. 2, 2017, noon GMT
From the game of Go to self-driving cars, there is widespread agreement that progress in artificial intelligence has accelerated in recent years. Yet there is remarkably little consensus as to what this progress portends in the coming years or decades. This is especially true of the big question of if and when we will have general-purpose human (or super-human) level AI. I’ve seen estimates by credible AI researchers range from ten years to “never”, and everywhere in-between!
While difficult, this question is extremely important. Humans have (for better and worse) agency over the fate of planet Earth due to our intellectual capability and technology; as we create systems with comparable capabilities on various axes, it will profoundly change the course of society and perhaps life itself.
Toward making good decisions and allocating resources wisely, it is vital that we have some ability to predict how various types of AI systems and capabilities will unfold; that is the aim of the new “AI Milestones” question series on Metaculus.
Lots of newly-resolved questions
By Anthony on Jan. 1, 2017, noon GMT
Happy new year!
A whole boatload of questions resolved in the last couple of weeks of December (many of the form “Will X happen in 2016?”. Somewhat bizarrely, just about all of them (see the list here) resolved negative! In the great majority of cases, these negative resolutions were forecast correctly. I guess we all have to come up with some more likely things to predict about…
The leaderboard has now moved around a lot, and a couple have now attained “Predictor” status, homing in on “superpredictor.” We may even have a seer or a prophet soon!
I looked for a good numerical index with which to ask the question “will 2017 be better than 2016?”, but could not see a great candidate — if anyone else does please leave a comment, or just suggest a question onsite.