M
Question
  • Facebook
  • Twitter
    •

    • Your submission is now a Draft.

    Once it's ready, please submit your draft for review by our team of Community Moderators. Thank you!

    You have been invited to co-author this question.

    When it is ready, the author will submit it for review by Community Moderators. Thanks for helping!

    Pending

    This question now needs to be reviewed by Community Moderators.

    We have high standards for question quality. We also favor questions on our core topic areas or that we otherwise judge valuable. We may not publish questions that are not a good fit.

    If your question has not received attention within a week, or is otherwise pressing, you may request review by tagging @moderators in a comment.

    You have been invited to co-author this question.

    It now needs to be approved by Community Moderators. Thanks for helping!

    {{qctrl.question.title}}

    {{qctrl.question.predictionCount() | abbrNumber}} predictions
    {{"myPredictionLabel" | translate}}:  
    {{ qctrl.question.resolutionString() }}
    {{qctrl.question.predictionCount() | abbrNumber}} predictions
    My score: {{qctrl.question.player_log_score | logScorePrecision}}
    Created by: MetaculusOutlooks and
    co-authors , {{coauthor.username}}
    Forecasting AI Progress

    Make a Prediction

    Prediction

    This question is part of the Maximum Likelihood Round of the Forecasting AI Progress Tournament. You can view all other questions in this round here.

    SuperGLUE (Wang et al., 2019) is a benchmark for evaluating general-purpose language understanding systems. The set of eight tasks in the benchmark emphasises diverse task formats and low-data training data tasks, with nearly half the tasks having fewer than 1k examples and all but one of the tasks having fewer than 10k examples.

    As of writing this question, the state-of-the-art model for is T5: Text-To-Text Transfer Transformer (Raffel et al., 2019), which achieves an average score 89.3, just below the human baseline of 89.8

    The SuperGLUE leaderboard may be accessed here.

    Categories:
    Series — Forecasting AI Progress
    Maximum Likelihood Round
    on {{qctrl.question.created_time | dateStr}}
    Partnered with The Open Philanthropy Project.