M
Discussion
  • Facebook
  • Twitter
    •

    • This Discussion is now a Draft.

    Once it's ready, please submit it for review by our team of Community Moderators. Thank you!

    Pending

    This Discussion now needs to be approved by community moderators.

    Self-resolving question methods

    Created by Anthony on {{qctrl.question.created_time | dateStr}} • Edited on {{qctrl.question.edited_time | dateStr}}
    Metaculus Meta

    We now have a few "self-resolving" questions onsite, and seeing how they play out suggests some changes to the method. So I'm proposing a new possible scheme, and looking for feedback.

    First, the rationale for these types of questions:

    • There is a set of propositions of widespread interest whose truth value is in dispute, even though there probably is some fact of the matter.

    • Often (though not always) the truth value becomes widely agreed-upon and manifest at a later time, such a great majority of reasonable people will agree.

    • These issues can be very hard to cleanly operationalize as a usual Metaculus question.

    So the idea is to look for a method where predictors can converge on a resolution as more information comes to light, with a resolution triggered at some point when a threshold certainty is achieved.

    The tricky part is for this to be real confidence in the truth value rather than just a self-reinforcing convergence.

    So here's my new proposal:

    1. The question comes in the form of a proposition that will be judged to be either true or false.

    2. The question opens for some Initial Period. During this time the community prediction is hidden. (I'm on the fence as to whether to keep the scoring value hidden, and prefer not to hide comments.)

    3. Next comes a Convergence Period. During this period the community prediction becomes visible, and new predictions are still allowed.

    4. At the end of the Convergence Period:

    • If the community prediction has converged sufficiently closely (say <= 2% or >= 98%) the question resolves in that manner.

    • Otherwise, the question goes to a 3-person panel of unpredictable nature, whose unanimous ruling resolves the question one direction or another, and non-unanimous ruling renders it ambiguous.

    • However, via a random number generation there is a P ~ 10% chance that even if the question resolves via the first mechanism, it will be turned to the panel anyway. (Optional alternative is for this to be at the question author's discretion according to a method of their own.)