
Forecasting for Scientific & Technological Exploration
11
*By Anand Muthusamy (Convergent Research), Molly Hickman (Metaculus), Trevor Martin (Independent), Deger Turan (Metaculus)* Global research and development (R\&D) makes long-horizon bets on highly path-dependent progress, but the assumptions underlying these bets are rarely made explicit. This piece argues for bringing public, systematic forecasting into scientific planning and resource allocation. In this piece, you will: (1) Learn where and how research & development can leverage forecasting. (2) Review workshop formats to turn expert views into resolvable forecast questions. (3) Preview a pilot use case in protein function prediction. ## 1. Asking \$3 Trillion Questions Each year, humanity spends roughly \$3T on R\&D, betting on which paths will yield future prosperity. Committees and leaders across companies, universities, and governments allocate millions to billions based on their hypotheses for those avenues in research, and which will amount to breakthrough-level progress. **Every strategic decision rests on two types of assumptions: what is feasible today, and how different fields might converge to unlock new capabilities tomorrow.** These assumptions about capabilities over time are ubiquitous but, remarkably, often remain unwritten, under-quantified, and therefore, under-examined. Late-stage industrial development may be tightly coupled to product development and disciplined by markets; however, early-stage R\&D has inherently fewer constraints. Long-horizon efforts are particularly needed in domains where progress is highly path-dependent and the most enabling outputs are public goods (e.g., large datasets, measurement standards, testbeds, and core facilities). These efforts require patient, sustained investment to compound over time and create entirely new categories of technology and markets. But many funding methods tend toward fragmented, one-off projects and short funding cycles. Most scientists and operators in R\&D acknowledge that resource allocation is inefficient at best and frequently misguided by poor incentives or biases related to “citations as currency” and individual career dynamics. **Boom-and-bust cycles, whether in capital markets or research hype, may not efficiently support exploration-exploitation sequences** (e.g., AI winters and biotech downturns). While there is plenty of interest in moonshots, we lack the institutional coordination to actually realize the next Apollo or Manhattan projects. **All investment has a temporal dimension, and R\&D is no different. It does not just matter what you invest in, but when. Science is the “art of the soluble,” per Peter Medawar**, the immunologist and Nobel laureate, and progress depends on asking the right questions that can plausibly be answered now. If you can test an interesting scientific hypothesis quickly, you should do it! Collecting and sharing data is often the best feedback mechanism for refining our theories of change, and helping us make better investments in science projects moving forward. But what if you cannot test your idea out quickly and cheaply? What if the tools do not yet exist, or the dataset you need would require a coordinated effort or be extremely expensive to collect? What if the field has wildly varying ideas of the “best question to ask right now?” Or, more radically, what if advances in another field totally unblock or make obsolete the question at hand? To address these challenges, we propose that systematic forecasting ought to support coordinated scientific efforts. Forecasting maps a “cone of possibilities” that radiates out from the present moment. **Physical constraints define all “possible” futures, but we must define paths to “preferred” futures (Figure 1)**. In this process of forecasting, we can discover levers that will allow us not only to reach, but also to hasten the arrival of those preferred futures. Some of these levers will be unexpected; paraphrasing a prior report in the field, the essence of forecasting for science can be summed up as: **“When will we have that technology we want? 10 years from now, if we do it right. Sooner, if we do it crazily.”** Convergent Research and Metaculus aim to quantify these timelines and levers: What specific interventions (e.g., funding drives, parallel tracking, regulatory shifts) would be required to pull that future forward? [<img height="900" width="1600" src="https://substackcdn.com/image/fetch/\$s_!8D7F!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc967e7b3-18dc-4757-9e31-23ff104bfae5_1600x900.png" />](https://substackcdn.com/image/fetch/\$s_!8D7F!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc967e7b3-18dc-4757-9e31-23ff104bfae5_1600x900.png) **Figure 1**: The “futures cone,” often used in strategic foresight; planning often focuses on the preferable and projected (or “probable”) slices ([Hancock and Bezold 1994](https://www.researchgate.net/publication/13166132_Possible_futures_preferable_futures)). Image source: [Speculative Design and a Cone of Possibilities (Delve 2025)](https://www.delve.com/insights/speculative-design-and-a-cone-of-possibilities) *** ## 2. Enriching the Scientific Method with Forecasting Breakthroughs in science and technology do not occur [*ex nihilo*](https://en.wikipedia.org/wiki/Creatio_ex_nihilo)[.](https://en.wikipedia.org/wiki/Creatio_ex_nihilo) They are often preceded by [measurable trends](https://arxiv.org/pdf/1503.00673) involving the maturation of precursor technologies or the coordination of critical actors. **Even “sudden” breakthroughs usually follow visible precursors**: the accumulation of experimental results, new technologies exposing phenomena that were previously unmeasurable or undetectable, distant fields connecting, and resources shifting to new institutions. For example, AlexNet’s 2012 ImageNet victory built on decades of neural network research and GPU availability; CRISPR-Cas9 emerged from basic microbiology converging with synthetic biology and protein engineering; the Human Genome Project succeeded as sequencing costs dropped and computational biology matured. Scientific progress is inherently uncertain, iterative, and collective; navigating it benefits from structured foresight. By making implicit assumptions explicit and mapping capability convergence, probabilistic forecasting can help manage uncertainty and identify bottlenecks on a rolling basis. We expect enriching the scientific method with forecasting will yield three benefits: ### 2.1 Surfacing Promising Outlier Views The *shape* of the forecast distribution can matter more than the median. Tightly clustered forecasts can suggest convergence. Still, they warrant scrutiny: is this a genuine consensus based on evidence, or does this tight distribution reflect herding or groupthink in beliefs? When forecasters split into distinct groups, with some expecting incremental progress and others expecting discontinuous breakthroughs, it signals a fundamental disagreement about how the field will evolve. A high-probability forecast could reflect two different beliefs: (1) the outcome will happen naturally given current trends, or (2) the outcome is technically feasible but only if actors prioritize it. Distinguishing these requires analyzing the forecast distribution and asking forecasters to explain their reasoning. Outlier forecasts can reveal that the initial question was poorly framed or is not considering a dramatically different level of abstraction. [Deep learning for protein structure prediction](https://www.nature.com/articles/s41586-021-03819-2) illustrated this: the winning question was not “How do we incrementally improve template-based methods?” but rather “Can neural networks learn sequence-to-structure mapping directly?”, pointing to a shift in approach that unlocked capabilities that previously seemed decades away. Outlier forecasts can also be treated as structured options. A 10% chance of a 100-fold improvement might justify more exploratory investment and attention than a 90% chance of a 2-fold improvement, especially if the field is already progressing incrementally but lacks tools for step-changes. Public forecasting carries its own risks such as premature consensus, social pressure toward the median, or optimizing for forecastable milestones rather than breakthroughs. Workshop design and hidden forecasting periods can not fully eliminate but can mitigate these dynamics. ### 2.2 Reframing Questions for More Effective Scientific Reasoning Every scientist carries their own implicit theory of progress, but these theories are not often detailed publicly and get watered-down in their scientific publications. By registering probabilistic predictions on specific timelines, we can help make these assumptions explicit, which lets us see clearly where and when we are surprised, which of our models hold up better, and which of our assumptions consistently fail. Experiments guided by registered forecasts can generate insight about the scientific process that would seldom otherwise be accessible. In the case of failure, they inform us of our technical capabilities: whether certain approaches were technically flawed or just poorly supported, whether the previously identified bottlenecks were misunderstood, and/or whether some timelines were overestimated (planning fallacy). Each surprising resolution also acts as a feedback mechanism for our forecasting methods: either our probability estimate was off, or our underlying assumptions about how fields converge and capabilities scale were wrong. Over iterations, this process sharpens both the forecasts and the world model underlying them. ### 2.3 Creating Benchmarks and Formal Verification Methods A well-operationalized forecasting question must have three elements: clear definitions of key terms, objective criteria for success, and verifiable resolution conditions. These requirements reveal a problem: many scientific disciplines lack agreed-upon metrics for progress. But this gap is precisely where forecasting adds value. The process of defining measurable milestones forces a field to clarify what “solved” actually looks like. When experts work jointly through operationalization, they can create benchmarks that can become standard within their field. [CASP](https://predictioncenter.org/) (Critical Assessment of Structure Prediction) exemplifies this: by establishing a biennial competition to rank protein-folding methods, CASP created a shared yardstick that accelerated the research ecosystem’s progress toward models like AlphaFold. The same method can be generalized across domains at the stage where they naturally yield well-operationalized forecasting questions. Some fields and problems may not offer problems as simple, bounded, and scorable as those in CASP. Other fields may be nascent or hinge on large, complex datasets, and thus not support the labor needed for scoring. We think that developments in AI can help us with this problem: as AI methods for handling multi-modal and unstructured data improve, we envision the creation of automated, formal verification systems that parse publications and datasets in real-time to update forecast probabilities. Moreover, the fact that future agents can be trained to follow standard forecasting workflows and to employ research tools will enable repeatable verification. Over time, forecasts can resolve with less manual overhead, and benchmark signals can update in near real-time, letting communities focus less on building bespoke tools to measure against benchmarks, and more on setting priorities and picking the most informative next experiments. *** ## 3. Early Prediction Efforts Have Paved the Way for Systematic, Public Forecasting Forecasting is broadly used in government (especially defense & geopolitical activities), computational sciences (e.g., AI scaling laws and compute build out), pharmaceutical development (around development pipelines & revenue targets), as well as in nearly every other major industry in some form. Yet the models used in these circumstances remain largely proprietary because they are classified (government defense), competitively sensitive (pharma, tech, or finance), or too complex and specialized for external use. This secrecy has a cost. **When forecasting stays siloed, fields lack shared visibility into each other’s progress and bottlenecks, and individual researchers and funders can’t benefit from broader expert input.** There are two approaches to aggregating forecasts: prediction markets (e.g., Polymarket, Kalshi, and Manifold) and prediction aggregators (e.g., Metaculus and Good Judgment). Prediction aggregators are better suited for scientific forecasting: they handle extreme probabilities (where market mechanics break down), accommodate long horizons without tying up capital, and can collect independent expert judgments before groupthink sets in, through the use of hidden periods (i.e., collect each forecaster’s opinion individually before revealing the crowd aggregate, which is impossible with markets by definition). For more on prediction markets versus aggregators, see [our comparison](https://www.metaculus.com/notebooks/38198/metaculus-and-markets-whats-the-difference/). The following are some forecasting platforms which have largely focused on the likelihood or impact of certain breakthroughs and projects in science and technology: * [Good Judgment Open](https://www.gjopen.com/questions) (GJO, founded 2015): A crowdsourced forecasting platform which asks users to predict geopolitical and scientific outcomes. GJO pioneered techniques for identifying “superforecasters” (users with exceptional accuracy) and demonstrated that aggregating non-expert judgments can outperform individual experts—a finding which has broadly elevated institutional interest in forecasting. * [IARPA ForeST](https://www.iarpa.gov/research-programs/forest) (Forecasting Science and Technology, 2014-2016): An IARPA research program that developed methods to detect and forecast scientific breakthroughs. The program aggregated predictions from 10,000+ scientists, mined possible forecasting questions from scientific literature, and used several methods for eliciting forecasts, including a prediction market called SciCast. * [RAND Forecasting Initiative](https://www.randforecastinginitiative.org/) (RFI, founded 2022): Crowdsourced forecasts on biotech risks and biosecurity. Examples include: predicting timelines concerning developments (e.g., enhanced-risk pathogens), forecasting biotech workforce trends, and estimating funding gaps relevant to pandemic preparedness. Intended to help policymakers, funders, and intelligence agencies align resource allocation with estimated risks. * [Long Now Bets](https://longbets.org/) (founded 2003) & [Manifold Markets](https://manifold.markets/) (founded 2021) & other similar platforms: Public prediction markets that allow anyone to wager money (play money in the case of Manifold) on future outcomes. While many bets are social or financial (sports, politics), a subset addresses scientific questions. Unlike GJO or IARPA ForeST, these are more recreational. * [Metaculus](https://www.metaculus.com/about/) (founded 2015) & [Federation of American Scientists](https://fas.org/) (FAS): Metaculus is a forecasting platform focusing on science and technology; FAS collaborated with Metaculus in an “FRO-casting tournament” to test whether forecasts could prioritize Focused Research Organization (FRO) proposals. The tournament invited subject-matter experts and professional forecasters to score FRO concepts on tractability and impact. The experiment aimed to develop a mechanism for science funders to leverage forecasting in allocating grants, thereby treating funding decisions as portfolio optimization problems. These efforts were devised in part to answer questions such as “Will X happen by a certain date?” or, on occasion, “How big is the impact of Y?” \*\*They were not intended to produce *integrated forecasting systems—those that can account for contingencies or new developments, targeted interventions, base rates of success, and cost curves.* *** ## 4. Forecasting for Science Weaves Elements of Progress Into a World Model ### 4.1 The Nature of Scientific Progress Lends Itself to Classic Forecasting As Metaculus defines it: > *Forecasting is a systematic practice of attempting to answer questions about future events*. The method requires well-defined and operationalized questions, along with quantified confidence scores to track progress objectively (see[ Metaculus’ FAQs](https://www.metaculus.com/faq/) for more). Forecasting is a repeatable and scalable technique in other fields, and we propose adapting it for science and technology. **The focus on uncertainty in forecasting naturally complements traditional planning and roadmapping.** Traditional roadmaps often commit to a sequence of steps toward a preferred future. Forecasting asks: what is the spectrum of futures we might plausibly face, and how likely are they? [Scoblic and Tetlock](https://www.foreignaffairs.com/articles/united-states/2020-10-13/better-crystal-ball) explored this intersection directly. They note that strategic foresight and probabilistic forecasting treat the future differently: foresight works with a portfolio of *plausible* narrative scenarios, while forecasting pins down specific events and assigns numeric probabilities to each. Their proposal is to combine the two, developing clusters of forecastable questions that signal which scenario is emerging, so policymakers can place smarter bets sooner. **Since science and technology operate under natural laws and physical constraints, they are amenable to the kind of model-based forecasting that has proven previously successful in some niches**. Weather forecasting offers an analogy: it deals with chaotic, complex systems but has evolved to make physically and historically grounded projections that improve over time. We can build world models with these properties across science and technology that integrate physical constraints, tool capabilities, economic trends, and socioeconomic factors such as research priorities and government incentives. **That said, forecasting isn’t equally applicable to all scientific problems. It’s most valuable where complex coordination is required and progress can be measured in clear, quantitative terms.** **We propose five dimensions for identifying which problems are good candidates for forecasting:** 1. *High contingency*: Forecastable science problems are those that require multiple fields to solve. In this case, their progress is tightly coupled through executing experiments, aligning standards, and funding bottlenecks in the right order. If a few labs working in isolation could plausibly succeed without outside advances, then contingency is low. 2. *Medium horizons (2–10 years)*: development timelines long enough for cost/throughput curves and capability ramps to influence progress but plausibly short enough to not experience a scientific revolution. On \< 2-year timelines, progress is primarily driven by operations and execution, so roadmaps are much more useful than forecasting. On \~10-year timelines, assessing progress looks more like worldview bets; as such, these should be broken down into 2–5‑year resolutions. 3. *Experiment‑dependent*: where progress depends on data collection and iterated experimentation as opposed to theoretical proofs that unlock progress. That makes the rate limiters material and legible. Purely theoretical problems with no empirical choke points rarely benefit from this approach or must be clearly tied to physical limiters (e.g., datacenters and compute power for AI). 4. *Tool‑sensitive*: where there are identifiable instrumentation or method bottlenecks (e.g., readout fidelity, synthesis cost, sample prep throughput) that move along [learning curves](https://ourworldindata.org/learning-curve). 5. *Measurable milestones*: where one can define externally resolvable achievements with metrics, thresholds, and dates that pass the clairvoyance test, i.e. someone who knows the future should be able to resolve it unambiguously. Quantitative milestones allow for less ambiguous tracking. ### 4.2 Graduated Approaches to Forecasting Scientific Progress The most ambitious, longest-term technical efforts naturally require broader expertise and proportionally greater predictive power for useful forecasts. **But we do not need to achieve the strongest level of forecasting to start benefiting**: simply documenting implicit bets about tool availability or cost curves can already enhance resource allocation. Full, field-wide modeling remains aspirational (but achievable for certain well-bounded domains, where the input fields are defined and rank highly on the five dimensions of forecastability). We propose three degrees of gradation in forecasting for science and technology: * *Judgment-based*: This is what scientists are usually doing, or almost doing but have not yet made systematic and public. We already carry value judgements, worldviews, cost-benefit curves, and likely paths our field may take, and a sense of our level of confidence for each. Even in the complete absence of traditional quantitative models, we can encode our beliefs in terms of probabilities and enable more nuanced conversations (see “[A Case for Nuanced Risk Assessment](https://www.metaculus.com/notebooks/27471/a-case-for-nuanced-risk-assessment/)” for more arguments and examples). * *Data-informed*: at this level, the field has reached formalization in concepts and rules. We can project learning rates and cost curves for well-defined problems like [CASP scores](https://predictioncenter.org/casp15/index.cgi), DNA synthesis costs, etc., and create benchmarks for measures of progress. We make forecasts based on observable trends and established metrics, but these projections depend on continuity (i.e., they work well *until* a paradigm shift occurs). * *Data-driven*: We can predict the rate of progress with high confidence in that field, including cases where those in a field have not reached consensus (i.e., we can account for different worldviews/paradigms clashing). To take but one example: astronomy is a clear case where larger or more powerful telescopes and detectors (e.g., LIGO) have been predicted to be sufficiently capable to break past the uncertainty needed for a potential breakthrough. ### 4.3 Making World Models Explicit & Interrogable Through Forecasting Other domains have found ways to make their forecasting models explicit and to share them to facilitate coordination. [In climate science, integrated assessment models](https://unfccc.int/topics/mitigation/workstreams/response-measures/modelling-tools-to-assess-the-impact-of-the-implementation-of-response-measures/integrated-assessment-models-iams-and-energy-environment-economy-e3-models) couple physical dynamics, technology and energy systems, and economic and policy choices to explore long‑run scenarios. In semiconductors, the [International Technology Roadmap for Semiconductors](https://en.wikipedia.org/wiki/International_Technology_Roadmap_for_Semiconductors) plays a similar role by providing a projection of transistor density, lithography limits, and cost curves that coordinated fabs and designers for decades. Scientific exploration needs something different. Breakthrough research often involves high-contingency questions that cross domains, with externalities across domains as varied as supply chains, instrumentation, funding, and theory. To issue good forecasts against these sorts of questions, we need something less rigid than a fixed quantitative model but more structured than informal conversation. *Structured forecasting* provides this bridge. By translating vague intuitions into concrete, dated, measurable predictions, we can build a shared map of the problem space. The model isn’t predetermined; rather, it emerges from the questions that experts think matter and how their forecasts on those questions interrelate. This approach handles cross-domain contingency as different experts contribute forecasts on their piece of the puzzle (e.g., instrumentation, supply chains, theory, funding) and the connections reveal how progress in one area enables or blocks breakthroughs elsewhere. The resulting network serves as an artifact around which discussion can be centered. Then, when we get surprised, we can see which assumptions broke and update accordingly. **Over time, we expect models built from structured forecasting exercises to become more systematic, closer to reasoning engines that encode domain knowledge explicitly.** Hard constraints from physics or information theory, soft rules like learning curves and cost scaling laws, adjustable parameters for funding or policy choices can all be embedded in the model’s architecture, automatically updating forecast probabilities as dependencies resolve. The result would integrate physical laws, empirical trends, and expert judgment into a single coherent framework for exploring scientific futures. **Figure 2 shows one implementation: Metaculus’s MetaNet tool (beta).** [<img height="595" width="728" src="https://substackcdn.com/image/fetch/\$s_!co_j!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9886dd18-ffec-4f16-96cd-18612b926fcf_1600x1309.png" />](https://substackcdn.com/image/fetch/\$s_!co_j!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9886dd18-ffec-4f16-96cd-18612b926fcf_1600x1309.png) **Figure 2**: MetaNet example output. Each node represents a specific, resolvable question with a date and probability estimate. The tool allows you to chain these nodes together, based on their relationships. For example, “solid-state electrolytes become manufacturable under a given process at scale at \<\$X/kWh by 2030” would increase the probability of “EV pack costs fall below \$Y/kWh,” which in turn makes additional factory build-outs more likely. Separate nodes like export controls, permitting delays, or critical-mineral supply shocks can dampen those pathways. Two experts might forecast the same milestone differently, or may even disagree about the ontology of the field (i.e., which factors matter and how they interrelate). That disagreement can be encoded in the model and is valuable! It makes assumptions visible and interrogable rather than hidden in competing grant proposals or roadmaps. Making world models legible through forecasting gives us a publicly verifiable system to facilitate *nuanced, dynamic conversation* about the future that’s grounded in specific, resolvable claims. The example above is still early-stage, though MetaNet already includes LLM tools that can suggest relevant questions and connections, and nodes can draw from live Metaculus community predictions when they exist. In the near future, LLM-based tools can enable auto-updating probabilities as new evidence arrives, synthesizing disagreements across forecasters, and reformulating the forecast under different user-defined assumptions. We are excited by the potential to run MetaNet models as quantitative models, playing out many scenarios. *** ## 5. Solving the Problem of Specialized Knowledge by Bringing Together Insiders and Outsiders Scientific forecasting faces unique challenges when operating on and projecting out from the frontier of what is known. The nature of science and practices in the scientific process provides helpful structural features not found in forecasting for social and geopolitical events: tool and model capabilities are reported along with their metrics, providing measurable trajectories; reproducibility (or lack thereof) latent in the literature can continually update our confidence in a finding or approach; instruments and services, and their capabilities, are documented by vendors and contract research organizations; scientists provide mental world models implicitly in their paper introductions and conclusions. **Furthermore, the same digital infrastructure that is currently accelerating science also makes scientific progress both easier to observe and more accessible in real-time.** Communication feedback loops are shortening via preprints, social networks, evolving standards (such as pre-registration), and open datasets. Markets create historical cost curves for the inputs to R\&D such as semiconductor manufacturing, DNA synthesis, and clinical trials, among many other ventures. In several AI-for-science domains, researchers have reported empirical ‘scaling’ relationships in [structural biology](https://www.pnas.org/doi/10.1073/pnas.2016239118), [chemistry](https://www.nature.com/articles/s42256-023-00740-3), [materials science](https://arxiv.org/html/2509.21811v1), and [math](https://scalingintelligence.stanford.edu/pubs/large_language_monkeys.pdf), while improving our ability to extract reliable insights from these models. We note that these trends are useful for near-term forecasting, as they can bend or break when bottlenecks shift. ### 5.1 To reach the frontier of tacit knowledge, we propose gathering subject-matter experts (SMEs) in workshops to develop frameworks for forecasting. This approach can be self-motivating: SMEs usually want to shape the future of their field and benefit from effective collaboration with colleagues. Scientific culture is broadly collaborative, and emerging tools will reduce the burden of participating in forecasting efforts. However, there are real competing incentives for secrecy. Researchers may guard unpublished data or model releases due to publishing priorities or commercial interests. But information exchange in early R\&D is more porous than it might seem. Private and public organizations attend the same professional conferences. Resource allocation leaves visible traces: hiring patterns, patents, and physical activities like building new facilities. Governmental actors already make projections on scientific progress and defense capabilities in adversarial informational environments. Helpfully, forecasting doesn’t require perfect information to be useful. ### 5.2 We are developing and testing a systematic workshop process, drawing on three established forecasting frameworks: * *Roadmapping*: A collaborative process where stakeholders map the sequence of capabilities, milestones, and dependencies needed to reach a technological goal, then work backward to identify intermediate steps and coordinate investment. [Convergent Research has discussed this approach](https://www.essentialtechnology.blog/p/scientific-roadmapping) when scoping Focused Research Organizations around well-defined scientific challenges. For example, they have published public roadmaps for [far-UVC infrastructure](https://www.convergentresearch.org/resources/convergent/solid-state-far-uvc-roadmapping-workshop-report) and [metagenomic sequencing methods](https://sequencing-roadmap.org/roadmap.pdf) for mitigating pandemic risks. * *Human judgment forecasting*: A forecasting approach that trains individuals in probabilistic thinking, actively open-minded reasoning, and debiasing techniques, then aggregates their calibrated predictions. IARPA’s [ACE tournament](https://www.iarpa.gov/research-programs/ace) demonstrated that the aggregate judgments of top forecasters performed comparably to the aggregate of intelligence analysts on short-run geopolitical questions ([Leech and Yagudin](https://arbresearch.com/files/comparing_forecasters.pdf)). * *Delphi Method*: A structured technique where experts forecast independently across multiple rounds, with each round revealing the distribution of predictions and supporting arguments. This allows forecasters to refine estimates based on collective reasoning while preserving individual judgment. [The RAND Corporation pioneered this method in the 1950s for defense forecasting](https://www.rand.org/topics/delphi-method.html). ### 5.3 The goal of the workshop is to sketch out the world model. We want to know how the field works and [“carve it at its joints.”](https://mitpress.mit.edu/9780262516266/carving-nature-at-its-joints/) **We propose workshops driven by hybrid teams; SMEs bring in their tacit, frontier knowledge and professional forecasters bring in best practices and experience** (e.g., skill in decomposing questions and feeling out base rates). Workshop design adds a final layer of best practices: scope by field maturity (i.e., incorporate philosophical discussions for pre-paradigmatic fields and seek comprehensive coverage of more mature fields), stress-test questions in solo and group formats, pair insiders with outsiders, and set collaborative incentives and norms (Chatham House rules, authorship credit) so that participants can be transparent and invested in the process. [Empirically](https://pubmed.ncbi.nlm.nih.gov/24659192/), non-professional forecasters can improve their skill (e.g., with [these tactics](https://www.cambridge.org/core/journals/judgment-and-decision-making/article/developing-expert-political-judgment-the-impact-of-training-and-practice-on-judgmental-accuracy-in-geopolitical-forecasting-tournaments/123EB18425391D05FA6581FDBB3F309F)). We start with asynchronous question formulation: SMEs define what progress looks like in their field, identify bottlenecks, and surface foundational disagreements. Once this formulation is complete, SMEs, with the help of the established forecasters, turn these interesting but un-operationalized questions into ones that can pass the clairvoyance test. For example\*\*,\*\* the question “Will we cure cancer?” might, after careful consideration, yield: “What is the probability that 5-year survival exceeds 80% for stage III subtype S by year X under a preregistered protocol?” Forcing this level of specificity often reveals that experts don’t even agree on what “solved” means, which is already a valuable output. **As questions are iteratively generated and operationalized, assumptions emerge, filling out the early shapes of a world model for that field.** The following processes can aid in this regard: 1. *Map solution paths:* Break the desired target into sub-problems and candidate paths; propose natural milestones. 2. *Stress-test with SMEs:* Ask whether those sub-problems actually “carve at the joints” and refine until they’re both technically meaningful and forecastable. 3. *Run iterative workshops:* Bring SMEs and professional forecasters together to refine the questions, elicit probability estimates, and repeat until no major surprises surface and a provisional ontology stabilizes. 4. *Open prediction:* Expand beyond the core group to adjacent-field scientists and, for a subset of questions, to the broader forecasting crowd. 5. *Public resolution and feedback loops:* Make questions and criteria public, resolve them on 0.5–5-year timelines, and update the world model as evidence accumulates. 6. *World-model synthesis:* Consolidate the results into a coherent, evolving model that funders, operators, and researchers can interrogate and react to. *** ## 6. Protein Engineering is One Ideal Pilot Domain for Forecasting To test whether systematic forecasting can improve scientific coordination, we need a domain that meets the criteria for tractable forecasting with sufficient ongoing activity to potentially shape. **Protein engineering is a strong fit: AI and lab automation are accelerating progress, yet experts disagree sharply on when and where approaches will plateau.** The potential outcomes present risks and rewards relevant to the general public in terms of personalized medicine (e.g., [viruses for gene therapies](https://www.nature.com/articles/s41576-019-0205-4)), [agriculture](https://www.cell.com/fulltext/S0092-8674\(21\)00005-2), and [biosecurity](https://www.science.org/doi/10.1126/science.ado1671). After decades of effort, protein structure prediction has matured into a practical design utility thanks to experimental data and breakthroughs in deep learning. We can now predict structure from sequence and increasingly design sequences for desired structures. But to realize the impacts of protein engineering, we must create proteins that reliably complete certain jobs, some of which are not found in nature. This is the realm of *function* prediction (i.e., given a sequence, what job does this protein do?) and *de novo* design (e.g., given a desired function, what sequence would produce a protein to accomplish it?). **Generative protein design methods are increasingly successful but the frontier is jagged**. [Antibody design](https://www.nature.com/articles/s41586-025-09721-5) has advanced rapidly, driven by massive commercial value, abundant training data, and relatively constrained design space. Outside this and a few other domains, progress is slower and executed artisanally. For example, protein engineering has already yielded fantastic machines such as enzymes that catalyze [formation](https://www.science.org/doi/10.1126/science.aah6219) or [cleavage](https://www.science.org/doi/10.1126/science.adi5554) of carbon-silicon bonds, reactions scarcely found in nature, if at all, and [optogenetic tools](https://www.nature.com/articles/s41592-019-0583-8) that allow us to control neurons with light. However, these examples relied on natural starting points and involved extensive years-long evolution campaigns. **More functions await:** proteins that perform logic operations in cells, template the assembly of inorganic molecules, operate in organic solvents, carry out reactions at room-temperature (that otherwise take industrial reactors), and form useful materials for clothing, protective gear, or medical equipment. Fully realized, protein design could look like a general-purpose nanotechnology. To test forecasting within protein engineering, we propose the protein function prediction problem as an organizing question. [CAFA](https://biofunctionprediction.org/cafa/) (Critical Assessment of Protein Function Annotation) already provides a framework for formalizing progress, and forecasting can build on this infrastructure while pushing toward harder, open-ended problems in design. **This problem meets our five forecastability criteria:** 1. *High contingency*: Progress depends on contributions from multiple fields such as structural biology, biophysics, machine learning, synthetic chemistry, and laboratory automation. One area (e.g., generalizing generative design to scaffolds outside of the training data) may stall due to lack of progress in another (e.g., new high-throughput assays for novel functions). Evolution of commercial needs may pull certain scaffolds or functions forward, concentrating activity for better or worse. 2. *Medium time horizons (2-10 years)*: Structure prediction has been a decades-long grand challenge that became generally tractable within \~years after deep learning methods matured. Whether function prediction and *de novo* design follow a similar arc is genuinely uncertain: the problem is harder in a sense (function depends on dynamics and context) but starts from a stronger foundation (today’s abundant structural data and mature ML tooling). 3. *Experiment dependent*: Progress requires biophysical validation and, in many cases, large-scale training data collection. New tools may need development and validation of their own, adding lead time. These cycles compound: iterating through new design frameworks takes years, not months, which is long enough for forecasts to be useful for resource allocation. 4. *Tool-sensitive*: Identifiable bottlenecks (e.g., DNA synthesis costs, cryo-EM quality and throughput, compute efficiency in molecular dynamics, machine learning algorithms) move along learnable cost curves and corresponding gains in productivity. These learning curves are helpful for near-term logistics of projects. Tools that deliver step changes in experiments via massively improved economics (e.g., full self-driving labs) or new capabilities (e.g., time-resolved structural methods) change the scientific focus of entire research plans. 5. *Measurable milestones*: Unlike some disciplines, protein engineering can be bounded and quantified in non-controversial terms (compare this with, say, “solving” neuroscience or intelligence). CASP and CAFA provide examples of how the field already achieves this. Functions can be described in terms of biophysical concepts (e.g., allosteric coupling) or physical metrics (e.g., binding affinity and energetics). In-field experts disagree on a host of questions about not only the future trajectories of the field but even the capabilities of current structure and design models. Many of these debates center on generalizing approaches beyond training data, data vs. algorithm bottlenecks, and the role of lab automation. We provide example questions that a workshop could use to *begin* discussions: > By 2027, will any protein function prediction model that is trained only on data publicly available as of 2025 achieve a ≥ 30% relative improvement over the January 2025 state‑of‑the‑art on a preregistered CAFA‑style benchmark metric? Public data includes all sequence, structure, and functional annotation datasets released on or before that date; no experimental measurements generated after that date may be used for training or fine‑tuning.\*This question addresses the data vs. algorithm debate for function prediction and uses a medium-term timeline to isolate true advances while providing some time for validation of the work. “CAFA-style” will require further refinement by SMEs. This question could be adapted for structure prediction and broken down by protein class as well to address model advancements across structure and function. Additional iterations could include gradations for additional new data enabling improved model capabilities. > By 2030, will any design method demonstrate ≥20% initial hit rate on a prospective panel of ≥100 enzyme designs for ≥50 reactions with no known natural enzyme nor efficient industrial process, where success is defined as measurable catalytic activity (kcat/KM ≥ 10² M⁻¹s⁻¹) confirmed by independent replication?\*Gives a more concrete check on model capabilities than the first question (at least for enzymes) and progress towards industrial application with a clear, legible scaffold today. SME input is needed to specify the reactions to best capture the most salient scientific and industrial problems and thresholds for efficiency to define what is useful. > *By 2030, will de novo designed protein assemblies for gene delivery, defined as any engineered protein or protein multimer that encapsulates and delivers nucleic acid cargo to mammalian cells, with \<10% sequence identity to any natural viral capsid, membrane fusion protein, or nucleic acid binding domain (as of December 2025) account for ≥30% of new gene therapy clinical trials initiated that year (per* [*ClinicalTrials.gov*](http://clinicaltrials.gov/)\*)? Chemical modifications (e.g., lipids, polymers, and sugars) to the proteins are included provided they compose \<20% of the total molecular weight.\*Gives a concrete check on progress towards industrial application with an open-ended scaffold (no one knows what could displace today’s viruses, but perhaps hints today indicate some confidence). This question gives an example of a higher level of specification. A potential weakness is introducing too much market forecasting by asking how competitive these non-natural protein vehicles will be against a slew of other delivery vehicles. The primary strength is a tightly scoped problem that stokes the imagination from the infinite chemical space of proteins. > By 2031, will any de novo designed protein (not minor variants of silk, collagen, elastin, etc.) be produced at ≥1 kg scale and formed into a material with (a) tensile strength ≥100 MPa or toughness ≥50 MJ/m³ and (b) at least one property not achieved by natural protein materials related to but not limited to conductivity, stimuli-responsive shape change, or self-healing with ≥80% strength recovery?\*This question deals in new-to-nature structures and functions but grounds progress in physical metrics (as opposed to medical or market terms). Specifying “one property not achieved by natural proteins” is meant to ensure substantial deviation from existing functions, but opens up ambiguity. We may need a formalism to distinguish functions at least by thresholds but, more satisfactorily, by category. These types of questions provide an initial survey of the field. The specific bottlenecks, capabilities, needs, and contingencies with other fields will seed the next round for forecasting eventual world model building for biology, spilling over into chemistry, materials, and medicine. **We anticipate the initial forecasting process itself would yield several valuable outputs:** * *Clarified success criteria*: The field often uses phrases like “solve the protein folding problem” or “solve design” without clear and shared definitions. The forecasting process requires operationalization of questions: solve for which structures and functions? At what success rate? Under what time and cost constraints? These clarifications alone would sharpen scientific dialogue and create a common language for coordinating efforts. * *Benchmark development*: Questions that prove difficult to operationalize often point to underdeveloped evaluation methods. For protein design, there may not be a consensus panel of new reactions for enzymes or conformational changes for allosteric coupling and switches to test. Furthermore, “solving design” for functions new-to-nature may be an enormous space that is challenging to quantify or decompose. Grappling with this challenge may push the field toward definitions grounded in chemistry and physics, specifying mechanisms and processes, for example. * *Bottleneck identification*: By mapping dependencies between capabilities (e.g., improved design requiring a certain experimental data set or better molecular dynamics), we identify which efforts would have multiplier effects instead of being blocked by upstream limitations. Bottleneck analysis is enriched by accounting for time and cost constraints. For example, DNA synthesis and sequencing still add tremendous costs to projects and functional assays that do not involve binding and barcoding (making large screens feasible) are challenging to automate. This analysis eventually makes explicit how bottlenecks in other fields relate to protein design. * *Cross-field coordination*: Making these bottlenecks legible to experts in adjacent fields invites contributions from researchers who might not otherwise know how exactly to engage protein engineering. A roboticist may have a solution for handling disordered or aggregation-prone proteins in assays. A machine learning scientist may have a clever way of squeezing more out of existing datasets with some context on how they were collected or how they fit into the larger biological context. Given the scientific and market value at stake in protein engineering, we suspect many researchers in adjacent fields would engage with legible forecasts. Collectively, these outputs constitute the first layer of a world model for the field: a shared, updatable map of capabilities, dependencies, and uncertainties that can inform coordinated projects. *** ## 7. Join us in Forecasting for Science We believe forecasting results should be a shared infrastructure as essential as open datasets and tools. Metaculus is actively developing and putting into practice tools and workflows for world modeling, to link progress across fields and make assumptions explicit (e.g. [SCRY-AI](https://www.aria.org.uk/opportunity-spaces/mathematics-for-safe-ai/?cardId=scry-ai#opportunityseeds), funded by ARIA; our [vision](https://www.metaculus.com/notebooks/29025/what-metaculus-is-building-next/) for forecasting for decision support). But the real power comes from building a coalition to improve decision-making: scientists and operators in R\&D anchoring efforts in their domains, ARPA-style program managers and SMEs sharpening assumptions, funders supporting metascience, and policymakers and industry players fostering coordinated early-stage ecosystems. With these partners, this approach can scale, turning today’s loosely coordinated efforts into systematic foresight that helps guide R\&D. **We are interested in bringing forecasting to other domains and are looking for scientists, funders, and policymakers interested in collaborating.** **Email** [**hello@metaculus.com**](mailto:hello@metaculus.com) **and** [**anand@convergentresearch.org**](mailto:anand@convergentresearch.org) if you’d like to partner with us on applying systematic forecasting to questions in science and innovation, or if you would simply like to discuss the approach. If you are interested but not yet sure what a partnership would look like, we would still welcome your message and can help identify a solid starting point. *** ## Acknowledgements: Joseph Fridman Christian Williams Laurie Smith Maya Gavin Fabrizio Ticchiarelli-Marjot Anders Sandberg *The arguments in this piece do not necessarily reflect the views of those listed nor their employers.*