Metaculus

Metaculus is excited to once again be partnering with Professor Thomas McAndrew and the Computational Uncertainty Lab at Lehigh University, with grant support from MIDAS (Modeling of Infectious Disease Agent Study) to generate actionable consensus forecasts on COVID-19 in the U.S. Your participation is greatly appreciated — it is always great to be able to harness our forecasting prowess in service of the greater good!

The goal of this project is to provide probabilistic predictions of the U.S. COVID-19 outbreak to support public health decision making at the federal and state level.

Predictions you make will be aggregated into a consensus forecast: an aggregate of individualized human predictions---and a subset of questions will be used to build a metaforecast: a combination of predictions from computational models and humans.

We plan to ask between 4 and 8 questions related to public health targets each month. This will take place on a monthly basis from this month through June 2021.

Questions will be posted at least 24 hours before they are officially opened for forecasting. During this time, we encourage forecasters to review the questions and ask for clarifications if needed by using the comment box on the Metaculus platform to ask questions under the corresponding question.

After the 24 hour review period, questions will be opened for forecasting. As usual, you will have the opportunity to review/revise your probabilistic predictions as many times as desired up until 10 days after questions are opened for forecasting.

We feel an important component to predictions based on human judgment is our ability as humans to provide a rationale alongside a quantitative forecast. We strongly encourage you to provide a rationale behind your forecast and/or supportive information to (i) help fellow forecasters and (ii) provide insight into how you made your prediction.

At the end of each month we will share a summary report with the Council of State and Territorial Epidemiologists, members of the Centers for Disease Control and Prevention, all members of MIDAS (Modeling of Infectious Disease Agent Study), and make this report available for public consumption. The summary report will include: (i) a synopsis of consensus predictions, (ii) a line list of each question and the consensus prediction, (iii) plots of the consensus probabilistic predictions for each question, (iv) methodology for how predictions are combined, and (v) a detailed list of each question, the background text, and resolution criteria.

Each question has a resolution criteria---the method we will use to determine the ground truth. Before a question is resolved, forecasters can view their individual probabilistic prediction overlaid against the consensus prediction. After a question is resolved, you will receive points that are based on your absolute performance and your performance relative to other forecasters. Details on scoring can be found in the Metaculus FAQ (https://www.metaculus.com/help/faq/#howscore) and the Metaculus scoring system page (https://www.metaculus.com/help/scoring/). In addition, when a question is resolved you receive a personal email that details the ground truth, your prediction, the points you were awarded.

Forecasters can see their performance across all questions that have resolved on their profile page — here, they will be able to access their Brier and Log scores averaged over all questions.

A document that outlines data security, privacy, and study procedures can be found here.

If you have any questions about the study, please email Tom McAndrew directly (mcandrew[at]lehigh[dot]edu).

Acknowledgment of Support: This research is supported through the MIDAS Coordination Center (MIDASNI2020-1) by a grant from the National Institute of General Medical Science (3U24GM132013-02S2).