calculating predictive insights generating predictive futures formulating intelligent insights forecasting definitive forecasts modeling quantitative contingencies mapping the future formulating calibrated contingencies aggregating critical understanding delivering definitive contingencies computing contingent forecasts computing probable wisdom assembling critical insights crowdsourcing contingent futures mapping probable insights

Metaculus FAQ/Help

Table of Contents

What is the aim of Metaculus?
What sorts of questions are allowed, and what makes a good question?
How do a I make a prediction? Can I change it later?
What are the "open time", "close time" and "resolve time?"
Who creates the questions, and who decides which get posted?
How can I get my own question posted?
Who decides the resolution to a question?
Do all questions get resolved?
What happens if a question gets resolved in the real world prior to the close time?
What happens if a question's resolution criteria turn out to have been fulfilled prior to the opening time?
How is scoring calculated, and points awarded?
How do predictions and scoring work for numerical questions?
What are levels and ranks?
What is a Brier score, what is a Log score, and how are the various Brier and Log scores calculated?
How is the community prediction calculated?
What is the Metaculus prediction?
What language and special symbols are allowed in my questions and comments?
What do "credible source" and "by [date X]" and such phrases mean exactly?
Who created Metaculus?
Is Metaculus a prediction market?
What does it mean for Metaculus to be "partnered with" a person or organization on a question?
How can I help spread the word about Metaculus?

What is the aim of Metaculus?

Metaculus poses questions about the occurrence of a variety of future events, on many timescales, to a community of participating predictors — you! Like many mental capabilities, prediction is a talent that persists over time and is a skill that can be developed. By giving steady quantitative feedback and assessment, predictors can improve their skill and accuracy, as well as develop a quantified track record. Then, probabilities of future events can be reliably drawn by optimally aggregating predictions — counting more heavily those with domain expertise and a strong prediction track record.

Some events — such as eclipse timing and well-polled elections, can often be predicted with high resolution, e.g. 99.9% likely or 3% likely. Others — such as the flip of a coin or a close horse-race — cannot be accurately predicted; but their odds still can be. Metaculus aims at both: to provide a central generation and aggregation point for predictions. With these in hand, we believe that individuals, groups, corporations, governments, and humanity as a whole will make better decisions.

As well as being worthwhile, Metaculus aims to be interesting and fun, while allowing participants to hone their prediction prowess and amass a track-record to prove it.

What sorts of questions are allowed, and what makes a good question?

Questions must ask the predictor to choose between two mutally exclusive outcomes of future events.

They generally take the form "Will (event) X happen by (date) Y?"

A good question will almost certainly be unambiguously resolvable. For example, a committee of people with perfect precognition should be able to definitively agree upon the answer. A good question will also, of course, be interesting!

While the initial focus of Metaculus is on science and technology, its purview will be steadily expanding.

How do a I make a prediction? Can I change it later?

You make a prediction simply by sliding the slider on the question's page to the probability you believe most captures the likelihood that the event will occur.

You can revise your prediction at any time up until the question closes, and you are encouraged to do so: as new information comes to light, it is beneficial to take it into account.

You're also encouraged to predict early, however, and you are awarded bonus points for being among the earliest predictors.

What are the "open time", "close time" and "resolve time?"

Questions must contain a open time, a close time and a resolve time.

  • The open time is the time when the question is open for predictions. Prior to this time, if the question is active, it will have "upcoming" status, and is potentially subject to change based on feedback. After the open time, changing questions is discouraged (especially if the changes are likely to change people's predictions rather than just clarify things), and such changes are noted in the question body.
  • The close time is the time after which predictions are locked in.
  • The resolve time should be the earliest time at which the the world should resolve the question.

The close time must be at least one hour prior to the resolve time, but generally will be much earlier, depending upon the context. For example, in a live sporting event the close time might be at the beginning of the game, and the resolve time at the end. For The passage of a piece of legislation, there may be months in between: the close time should be safely before the legislation might possibly pass, and the resolve time should be after it would be likely for the legislation to be passed.

The window between the close and resolve time should be long enough to create interesting uncertainty, and decrease the chance that the question gets resolved prior to the close time.

Who creates the questions, and who decides which get posted?

Many questions are launched by Metaculus staff, but any logged-in user can propose a question. Proposed questions will be reviewed by a group of moderators appointed by Metaculus. Moderators will select the best questions submitted, and will (probably) edit the question to add links, background material, remove ambiguity, etc., prior to publishing the question.

How can I get my own question posted?

Here are some guidelines:

  1. If you have a basic idea for a question but don’t have time/energy to work out the details, you’re welcome to submit it; it will go in the queue for potential development and launch by the moderators. (This queue is pretty backlogged, but we do try to prioritize user submissions.)
  2. If you have a pretty fully-formed question, with at least a couple of linked references and fairly careful unambiguous resolution criteria, it’s likely that your question will be reviewed and launched quickly.
  3. Although you can presently just pick one topic for your question, the moderators may well add more.
  4. While formally open to all sorts of questions, Metaculus is currently topically biased toward scientific and technological advances. So suggested questions on other topics, especially that require a lot of moderator effort to get launched, will be given lower priority and may be deferred until a later time.
  5. We regard submitted questions as suggestions and take a free hand in editing them. If you’re worried about having your name on a question that is altered from what you submit, or would like to see the question before it’s launched, please note this in the question itself; questions are hidden from public view until they are given “upcoming” status, and can be posted by "anonymous" or by a moderator upon request.

Note that staff is limited, and there is a backlog of good questions, so if yours has not yet appeared, don't despair!

Who decides the resolution to a question?

After the closing date, questions are eligible to be resolved. The resolution to a question as yes,no, or ambiguous is chosen by Metaculus or a Metaculus-appointed moderator. An ambiguous resolution generally implies that there was some inherent ambiguity in the question, that real-world events subverted one of the assumptions of the question, or that there is not a clear consensus as to what in fact occurred.

Do all questions get resolved?

Currently, all questions will be resolved.

What happens if a question gets resolved in the real world prior to the close time?

When resolving a question, the Moderator has an option to change the effective closing time of a question, so that if the question is unambiguously resolved prior to the closing time, the closing time can be changed to a time prior to which the resolution is uncertain.

What happens if a question's resolution criteria turn out to have been fulfilled prior to the opening time?

Each question is assumed to have as an implicit assumption that the event it is asking about has not yet occurred as of the time of the question's opening. If it turns out that the event required for a question to resolve as true has already occurred prior to the opening time, so that there has in fact been no uncertainty (but just perhaps obscurity) as to the outcome of the question, then the question will resolve as ambiguous.

How is scoring calculated, and points awarded?

The score you receive depends upon your prediction, what actually happens, and what the rest of the community predicts. By sliding the slider, you can see what score you will get if the question were resolved as 'yes' or 'no' right now. There are five key things you need to know about the scoring:

  1. Your expected score is maximized if you provide the true probability. For example, if the question was whether a fair coin would come up heads, then in answering a series of such questions your score would be highest if you give 50% each time. You should always predict the probability you believe to best reflect the true likelihood of the event's occurrence.
  2. The scoring awards points both for being right and for being more right than the community.
  3. Since the score is partially based on other player's predictions, the points "on the line" will change with time.
  4. The score you are awarded is time averaged over the time for which the question is open; you receive no contribution to your score while a question is open but before you make a prediction, and the points you earn depend upon your predictions' accuracy and when they are made. This means both that you should make your first prediction early, so as to earn points over more of the question's lifetime, and also that you should update your prediction anytime new information comes to light that alters your best estimate of the probability.
  5. You also earn extra points for your final prediction — half of a question's point value at its closing time gets added to your score — so even if you come to a question late it's still worth it to make a prediction!

How do predictions and scoring work for numerical questions?

Whereas binary questions resolve as either 'yes' or 'no', resolutions for numerical questions lie on a continuum of values. Therefore, when you make a prediction, you need to specify a probability for each possible outcome (that is, assign a probability density function.) There are an infinite number of such functions, but to make life a little bit easier we restrict predictions to normal (Gaussian) distributions. You can adjust the center and width of your normal distribution predictions using the sliders on the question pages.

Here are the key things you need to know, with more details below.
  1. You should put the center of your distribution at what you think is the most likely value, then adjust the width of the distribution so that you attribute better than even odds to the true number falling into your range.
  2. Making your distribution wider or narrower reflects your confidence in the central value you've specified, and decides the stakes: a narrower distribution give more points if your central value is right, but more losses if it's very wrong.
  3. As for binary questions, the scoring awards points both for being right and for being more right than the community, the points "on the line" will change with time, and the score you are awarded is time averaged over the time for which the question is open, with one-third the value coming from your final prediction.

What are levels and ranks?

Your level is determined from your score, where each level required 100 more points to attain than the last. Thus, for example, Level 1 (Newbie) corresponds to score 0–99, Level 2 (Guesser) to 100–299, Level 3 (Guesstimator) to 300–599, etc.

However, while you can lose points, your level is yours to keep: if you lose enough points to go down a level, who will retain your highest level (but will still have to make up the points in order to get to a higher level.)

Your rank is simply where you appear in the sorted list of all player's point totals.

What is a Brier score, what is a Log score, and how are the various Brier and Log scores calculated?

The Brier score is a commonly-used scoring rule (sometimes also called "quadratic scoring") that compares a set of predictions to actual outcomes. For a single forecast of probability \(p\), it is computed as \(S=(p-f)^2\), where \(f=1\) if the event occurred, and \(f=0\) if not. For multiple forecasts, an average is taken over \(S\) for the individual forecasts. The Brier score ranges from 0 to 1, with 0 being perfect accuracy and 1 being perfect inaccuracy. If you were to guess 50% for every question, your mean Brier score would be 0.25; if you were to randomly select a number between 0 and 100% for each question, then in the limit of many questions your score would approach 1/3.

In your user profile, the "mean Brier score" is simply the mean of your Brier scores for all of the questions that have resolved (non-ambiguously), where \(p\) is taken to be your most recent prediction. The "mean community Brier score" is the score of the community, i.e. the score you would have had if you had just taken the community prediction in each case. The "community mean Brier score" is the mean over all predictions by all users on the same set of questions you answered. Because you answer a particular set of questions, all three of these scores are particular to you.

The Log score is another commonly-used scoring rule, which (relative to the Brier score) gives a larger penalty for being confident (i.e. predicting near 0 or near 100%) but wrong. For a single forecast of probability \(p\), it is computed as \(S=-\frac{1}{4}\log_2 p\) if the event occurred, and \(S=-\frac{1}{4}\log_2 (1-p)\) if not. The scaling is chosen such that it matches the Brier score for a 50% prediction.

How is the community prediction calculated?

The community prediction is the median (that is, middle value) of recent player predictions. It's designed to respond to big changes in player opinion while still being fairly insensitive to outliers.

Here's the mathematical detail. Each predicting player is marked with a number \(n\) (starting at 1) that orders them from oldest active prediction to newest prediction. The individual predictions are given weights \(w(n) \propto e^\sqrt{n}\) and combined to form a weighted community distribution function. The median community prediction is just the median of this distribution. The particular form of the weights means that approximately \(\sqrt{n}\) new predictions need to happen in order to substantially change the community prediction on a question that already has \(n\) players predicting.

What is the Metaculus prediction?

The Metaculus prediction is the Metaculus system's best estimate of how a question will resolve. It's based on predictions from community members, but unlike the community prediction, it's not a simple average or median. Instead, the Metaculus prediction uses a sophisticated model to calibrate and weight each user, ideally resulting in a prediction that's better than the best of the community.

At the date of its deployment, a cross-validated version of the Metaculus prediction had a Brier score of 0.095 and a Log score of 0.120, lower than even our best predictors.

What language and special symbols are allowed in my questions and comments?

Questions and comments should be in English and should refrain from profanity, insulting, derogatory, or offensive language. See the terms of use for more details. Comments and questions can both contain markdown.

What do "credible source" and "by [date X]" and such phrases mean exactly?

To reduce ambiguity in an efficient way, here are some definitions that can be used in questions, with a meaning set by this FAQ:

  1. A "credible source" will be taken to be an online or in-print published story from a journalistic source, or information publicly posted on a the website of an organization by that organization making public information pertaining to that organization, or in another source where the preponderance of evidence suggests that the information is correct and that there is no significant controversy surrounding the information or its correctness. It will generally not include unsourced information found in blogs, facebook or twitter postings, or websites of individuals.
  2. The phrase "By [date X] will be taken to mean prior to the first moment at which [date X] would apply, in UTC. For example, "By 2010" will be taken to mean prior to midnight January 1, 2010; "By June 30" would mean prior to midnight (00:00:00) UTC June 30.

Who created Metaculus?

Metaculus originated with two researcher scientists, Anthony Aguirre and Greg Laughlin. Aguirre, a physicist, is a co-founder of FQXi.org, which catalyzes breakthrough research in fundamental physics, and of Futureoflife.org, which aims increase the benefit and safety of disruptive technologies like AI. Laughlin, an astrophysicist, is an expert at predictions from the millisecond predictions relevant to high-frequency trading to the ultra-long-term stability of the solar system.

Is Metaculus a prediction market?

Sortof. Like a prediction market, Metaculus aims to aggregate many people's information, expertise, and predictive power into a single high-quality estimation of the probability that something will occur. However, prediction markets as such operate using real or virtual currency, which is used to buy and sell shares in "event occurrence." The idea is that people buy (or sell) shares if they think that the standing prices reflect too low (or high) a probability. Metaculus, in contrast, directly solicits predicted probabilities from its users, then aggregates those probabilities. We believe that this sort of "prediction engine" or "prediction assembler" has both advantages and disadvantages relative to a "prediction market."

What does it mean for Metaculus to be "partnered with" a person or organization on a question?

A partnered question means that the partner individual or organization has approved being listed as a question partner, may have participated in the question development, is interested in the predictions on the question (and may use them in its own decision-making processes), and may aid in promoting the question; it does not necessarily imply any payment or formal agreement between the partner and Metaculus.

How can I help spread the word about Metaculus?

Metaculus will get more fun and more interesting to the extent that it grows to include more and more predictors, so we encourage participants to spread the word to people who they think may enjoy predicting, or just be interested in how the questions develop. Some of the most useful mechanisms are:

  1. Post particular questions you like to Twitter, Facebook and Reddit, using the "share" button on each page, which sets up a default tweet/post that you can edit.
  2. Follow us on Twitter, then retweet Metaculus tweets to your followers.
  3. Follow our Facebook page, and share posts you like.
  4. Contact us for other ideas.