M

White Hat Forecasting Challenge FAQ

What is Metaculus?

Metaculus is an online forecasting platform and aggregation engine that brings together a global reasoning community and keeps score for thousands of forecasters, delivering machine learning-optimized aggregate forecasts on topics of global importance. The Metaculus forecasting community is often inspired by altruistic causes, and Metaculus has a long history of partnering with nonprofit organizations and university researchers to increase the positive impact of its forecasts.

What is forecasting?

Forecasting is a systematic practice of attempting to answer questions about future events. However, it’s distinguished from other forms of prediction:

First, questions are carefully specified (or operationalized), and forecasters try to give precise, quantified answers (or forecasts) that also quantify their uncertainty.

Second, these forecasts are often aggregated into a single prediction. You can think of this as taking the average of the predictions, though in reality this aggregation is more sophisticated. Usually, this aggregate is more accurate than any individual forecast! This principle is known as wisdom of the crowds, and it becomes more intuitive when you consider that each forecast is based on partial information, while also ignoring some important considerations. The aggregate then integrates all of these disparate pieces of information. It can also cancel out individual biases—provided that the group as a whole is not biased in any one direction.

Third, the success of forecasters is usually measured. When a question is resolved (i.e., we reach the point in the future where the answer is known), forecasters are scored based on the accuracy of their previously made predictions.

These scores can accumulate from many forecasts on many questions over a long period of time and become a personal metric of how good a given forecaster is at predicting the future. This metric can later be used as an input to the forecast aggregation process, allowing us to give greater weight to predictions by forecasters with better track records. It also provides aspiring forecasters with important feedback on how they did and where they can improve.

When is forecasting valuable?

Forecasting is uniquely valuable primarily in complex, multi-sectoral problems or in situations where a lack of data makes it difficult to predict using explicit or exact models.

In these and other scenarios, aggregated predictions of strong forecasters offer one of the best ways of predicting future events. In fact, work by the political scientist Philip Tetlock demonstrated that aggregated predictions were able to outperform professional intelligence analysts with access to classified information when forecasting various geopolitical outcomes. More background on forecasting can be found in Tetlock’s book Superforecasting.

Why should I be interested in forecasting?

Philip Tetlock’s research has shown that great forecasters come from various backgrounds—and oftentimes from fields that have nothing to do with predicting the future. Like many mental capabilities, prediction is a talent that persists over time and is a skill that can be developed. Steady quantitative feedback and regular practice can produce greatly improved forecasting accuracy.

What is a recent example of how Metaculus forecasting is helping people?

In the midst of the COVID-19 epidemic, the Virginia Department of Health needed to determine state-wide staffing, testing, and genomic sequencing levels, but was hampered by the high degree of uncertainty about the effects of contrasting public health policies and community interventions. Metaculus and the VDH entered into a first-of-its-kind partnership, launching the yearlong Keep Virginia Safe Tournament and eliciting thousands of predictions from hundreds of forecasters. Forecasts provided an early warning of the impact of new variants and contributed to shaping the public health response, including determining statewide COVID-19 genomic sequencing levels, filling a gap in guidance at the federal level.