This essay is contributed by Justin Crow, MPA. He is the Foresight & Analytics Coordinator in the Office of Emergency Preparedness at the Virginia Department of Health, and a Superforecaster.

It  is a truism in military strategy that no plan survives first contact with the enemy. A similar claim could be made about forecasts: few of today’s forecasts will survive the weeks and months ahead. Good forecasting requires continual updating as circumstances change — sometimes due to people reacting to the forecasts themselves — and new information becomes available. If forecasts we generate today are likely to prove less accurate than future ones, what value does generating a forecast today produce? We generally think of forecasts as oriented toward some future state. People may act on a forecast now but, outside of swing-trading on prediction markets, the benefits of those actions do not accrue until the future state is realized, and then only if the forecast proves accurate. However, as in planning, the act of forecasting is indispensable, and provides many practical benefits along the way.

In the early stages of an emerging public health crisis, aggregate human forecasting may be especially useful. The question generation process helps people engage with the potential paths an outbreak may take. The forecasting process supports rapid collection and analysis of information and data sources. It provides a quick early assessment of new situations, and real-time integration of new information into the understanding of that situation. This post examines the value-add of each of these factors in a real world situation.

A Classic Example

Coronavirus disease 2019 (COVID-19) was an unprecedented threat to public health. As COVID-19 began to appear in the United States, policymakers and public health officials faced a great deal of uncertainty. COVID-19 had quickly overwhelmed health systems in China and Italy. Public health agencies, globally and throughout the United States, began setting up field hospitals to manage the expected influx of COVID-19 patients. By May 2020, however, most of these facilities in the United States were shut down without seeing a single COVID-19 patient. While this was good news, standing up these facilities unnecessarily diverted essential resources, staff time, and attention during a critical period.

Virginia also started down this path and, working with the US Army Corps of Engineers, began planning for three to four “alternate care facilities” at an estimated cost of $120 to $140 million. However, Virginia paused work on these facilities in mid-April 2020. Why was Virginia able to pivot from field hospitals when so many other states and countries did not? One reason was that Virginia partnered with the University of Virginia’s Biocomplexity Institute (UVA-BI) to create state- and local- projections of the course of the pandemic in Virginia. UVA-BI’s projections showed that public health measures were successfully “flattening the curve”, that Virginia would not immediately experience overrun hospitals as had occurred in China and Italy, and that they could provide enough warning if the situation changed.

Following this success, Virginia continued to partner with UVA-BI. The projections provided by UVA-BI helped state public health officials and health sector partners anticipate and plan for resource needs. Staff in these sectors also mentioned that this work helped them mentally prepare for coming surges, and breathe easier when cases were low. 

In Summer 2021, Virginia expanded its foresight work and partnered with  Metaculus in the Keep Virginia Safe tournament, and the Real-Time Pandemic Decision Making tournament. When a global monkeypox outbreak occurred in early 2022, Virginia was unique among states in having sustained partnerships with both an expert disease modeling group and an aggregate human forecasting platform. This post explores why and how the Metaculus forecasting platform was especially useful during the monkeypox outbreak.

Foresight and Monkeypox

Unlike COVID-19, monkeypox is not an unknown disease. Monkeypox is endemic in the Democratic Republic of Congo, and appears regularly in other Central and West African countries. Nigeria has been dealing with a particularly stubborn outbreak since 2017, and the virus has been exported sporadically from there. Prior to that, the largest outbreak outside of the endemic region occurred in the United States in 2003. The 2003 outbreak, which infected 71 people, was linked to imported pet rodents.

The 2022 outbreak has been very different. From the earliest days, it was evident this was a large, multinational outbreak. The Global.health public dataset, which tracks publicly reported cases, shows there were 290 cases reported in 20 non-endemic countries within the first seven days of data collection. Including endemic zones, monkeypox cases were identified on every continent. Coming in the midst of the ongoing COVID-19 pandemic, this unusual outbreak created a great deal of uncertainty and anxiety. Virginia’s public health sector, which is responsible for responding to disease outbreaks, was no exception. 

Quantitative Modeling or Aggregate Human Forecasting?

Quantitative modeling and aggregate human forecasting both have essential roles to play in anticipating the course of a disease outbreak. These roles often overlap or complement one another. In some circumstances, however, distinct use cases arise. Aggregate human forecasting can be especially useful during smaller outbreaks, or early in epidemics and pandemics. Quantitative models, by their nature, perform best when there is a large amount of data available and key parameters are known. Early in a disease outbreak neither of these conditions exist. 

Although large compared to previous monkeypox outbreaks, there were just over 1,600 confirmed or suspected cases of monkeypox globally through June 12, 2022, and just 53 in the United States. By comparison, at the time Virginia’s policy-makers were using UVA-BI models to make decisions about alternative care facilities in mid-April 2020, Virginia alone had over 50,000 confirmed and probable COVID-19 cases.  

Even for established diseases such as monkeypox, information on key parameters may be lacking or uncertain, as new variants, new vectors, and new environments may result in new disease characteristics. The sudden appearance of monkeypox cases and outbreaks across the globe raised questions about all of these factors. Quantitative modelers often deal with this uncertainty by adjusting model parameters to examine the implications of different possibilities in scenarios. Although scenarios are useful, decision-makers must weigh the likelihood of each scenario playing out in order to determine a course of action. Human forecasters, by contrast, are able to quickly balance and update sparse information on multiple parameters within their forecasts. 

The Question Generation Process

Whether using models or human judgment, the act of developing a forecasting regimen forces us to think rigorously about what actions may we need to take, and what information may inform those actions. The question development process helps public health officials engage with the current crisis and anticipate the paths it may take in a systematic way. Key decision points are identified, and linked with information streams for question resolution. For public health agencies, the most immediate questions revolve around operations. What resources may need to be mobilized, at what level, and what may drive those resource needs? Is the number of cases important, or is hospital capacity at risk? Which mitigation strategies may be employed, and how can we best target those strategies? 

Monkeypox generally has low transmission rates and low severity outside of the endemic zone, and a vaccine is available for prevention and treatment. Cases, which drive contract tracing and case investigation efforts, were likely to be the most important outcome.

Similarly, cases would drive the number of vaccines needed under the current ring-vaccination strategy — unless that strategy changes. 

Importantly, in May, state health agencies needed to consider whether to mobilize resources immediately, or whether a wait and see approach would be a more effective strategy. So the number of states impacted, along with an idea of the risk of a traveler-imported case was important as well.

Rapid Information Aggregation

One of my favorite things about forecasting is that it requires broad knowledge of the topic in question. I am never comfortable with making a forecast on a new topic until I have done at least a couple of hours of research. Of course, this is not nearly enough to create an expert in any field, but it is often enough to allow you to understand what the experts think is important about a topic, understand what is creating uncertainty right now, and to identify important information streams. 

Rapid information aggregation begins during the question development process. Question developers need to understand many of the same things forecasters do to develop relevant questions. In the first days of the monkeypox outbreak, staff from Metaculus, UVA-BI, and VDH all worked together to compile information and resources on monkeypox and the current outbreak. Importantly, forecasting questions also require rigorous sources for question resolution. Global.health’s public dataset, identified during the question development process, has been an important resource for tracking the outbreak.

link

Once questions are launched, forecasters themselves surface important information and resources. Forecasters ferreted out information on how monkeypox is spreading, the World Health Organization’s process and track record for declaring Public Health Emergencies of International Concern, and new genomic surveillance findings. Following the comments in relevant questions is a good way to keep informed on a topic.

Rapid Early Assessment

Forecasters weigh the evidence they collect and their collective judgment is aggregated into community forecasts. Unlike Eisenhower’s opinion of plans, these forecasts provide a useful and very rapid early assessment of an outbreak. Within a few days, Metaculus forecasters were able to provide state public health agencies with clear insight on how monkeypox was likely to impact operations, along with the risk of a broader pandemic. 

Metaculus forecasters estimated that nearly half of states would  see a monkeypox case by July 1st, but case numbers were likely to be low in that time frame — the median forecast was just over 100. Similarly, Metaculus forecasters expected monkeypox to be widespread, but with cases in the thousands, not tens or hundreds of thousands. Importantly, Metaculus forecasters gave only a small chance that the CDC would recommend vaccination for at least 10% of the US population this year. Overall, there was a reasonable chance that states would need to respond to monkeypox, especially those with large, connected travel hubs, but the impact on operations would likely be minimal barring any major changes.

Forecasting for Situation Monitoring

As expected, these forecasts have matured as more information about the course of the outbreak has become available and the time-frame of forecasts has shortened. Forecasts for the number of cases and jurisdictions affected have edged up, while forecasts for a widespread vaccination campaign have edged down. In some cases, the range of forecasted outcomes has tightened as well. 

While these updated monkeypox forecasts are likely to be more accurate, they also provide a valuable resource. New information comes in constantly, often at a furious pace during a crisis. However, most news also tends to be piecemeal, coming in small drips.  The impact can be unclear, or even contradictory. In the first months of the monkeypox outbreak, for instance, different reports linked the current strain to cases identified in 2018, noted two different strains were identified in the US, and discussed possible “microevolution” of the virus. Assessing the importance of these developments and their collective ramifications is a difficult and time-consuming task. Aggregate human forecasting excels at collating this information into forecasts, providing value beyond current epidemiological data and news streams. Forecasters also excel at understanding and assessing the impact of big news events, often incorporating the risk of such events before they occur.

In contrast, it is difficult for busy practitioners to take this all in and develop a full picture of changing circumstances and how they may impact operations. Importantly, the data point provided is simple and directly relevant: number of cases expected, or the chance of a large vaccination campaign, for instance. Ideally, questions are tailored to operational needs, and new questions added as circumstances change. 

Forecasting for Situational Awareness in an Emerging Public Health Crisis

It may be true that few plans work out as expected. But few leaders would recommend going into a crisis situation without a plan. Planning forces us to take stock of assets and vulnerabilities, and to consider risks, contingencies, and potential responses. The plans themselves may need to be adapted beyond recognition, but the knowledge gained during the planning process allows for quick response and redeployment in changing situations. 

link

Similarly, good forecasters do not expect to be accurate all of the time. Our goal is to be well-calibrated — on “the right side of maybe” — the expected amount of time. This is useful, and in addition, the process of forecasting itself can provide a benefit similar to that of planning, in real time both at the beginning of and throughout an ongoing crisis. Data and news streams are helpful, but public health officials are usually too busy responding to crises, implementing and adjusting plans, to properly assess and integrate new information that arrives in the form of disparate news stories. Aggregate human forecasting can fill the gap, helping practitioners understand how a changing situation may affect operations in a simple and directly relevant manner.

---

¹A substantial share of quantitative modeling’s role does not fall directly into the foresight realm, including causal modeling, cost-benefit analysis, policy analysis, and program design.

²UVA-BI and Metaculus are collaborating on a project to use Metaculus forecasts to estimate and update model parameters.