Self-resolving question method discussion

{{qctrl.question.commentStr()}} {{qctrl.question.estimateReadingTime()}} min read

Your submission is now in Draft mode. Once it's ready, please submit your draft for review by our team of Community Moderators. Thank you!

We now have a few "self-resolving" questions onsite, and seeing how they play out suggests some changes to the method. So I'm proposing a new possible scheme, and looking for feedback.

First, the rationale for these types of questions:

  • There is a set of propositions of widespread interest whose truth value is in dispute, even though there probably is some fact of the matter.

  • Often (though not always) the truth value becomes widely agreed-upon and manifest at a later time, such a great majority of reasonable people will agree.

  • These issues can be very hard to cleanly operationalize as a usual Metaculus question.

So the idea is to look for a method where predictors can converge on a resolution as more information comes to light, with a resolution triggered at some point when a threshold certainty is achieved.

The tricky part is for this to be real confidence in the truth value rather than just a self-reinforcing convergence.

So here's my new proposal:

  1. The question comes in the form of a proposition that will be judged to be either true or false.

  2. The question opens for some Initial Period. During this time the community prediction is hidden. (I'm on the fence as to whether to keep the scoring value hidden, and prefer not to hide comments.)

  3. Next comes a Convergence Period. During this period the community prediction becomes visible, and new predictions are still allowed.

  4. At the end of the Convergence Period:

  • If the community prediction has converged sufficiently closely (say <= 2% or >= 98%) the question resolves in that manner.

  • Otherwise, the question goes to a 3-person panel of unpredictable nature, whose unanimous ruling resolves the question one direction or another, and non-unanimous ruling renders it ambiguous.

  • However, via a random number generation there is a P ~ 10% chance that even if the question resolves via the first mechanism, it will be turned to the panel anyway. (Optional alternative is for this to be at the question author's discretion according to a method of their own.)

Metaculus help: Predicting

Predictions are the heart of Metaculus. Predicting is how you contribute to the wisdom of the crowd, and how you earn points and build up your personal Metaculus track record.

The basics of predicting are very simple: move the slider to best match the likelihood of the outcome, and click predict. You can predict as often as you want, and you're encouraged to change your mind when new information becomes available.

The displayed score is split into current points and total points. Current points show how much your prediction is worth now, whereas total points show the combined worth of all of your predictions over the lifetime of the question. The scoring details are available on the FAQ.

Thanks for predicting!

Your prediction has been recorded anonymously.

Want to track your predictions, earn points, and hone your forecasting skills? Create an account today!

Track your predictions
Continue exploring the site