Verify evals on Papers with Code

Open
#36,130 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
45/100
Issue type
Documentation
Clarity
Mostly clear
Activity status
Quiet
Domain
documentation

Research direction

Start with the linked Papers with Code paper and its listed evaluation and benchmark pages. Verify the imported scores, model names, benchmark protocols, and openness metadata against the paper or official release artifacts; done means reporting any corrections needed for the paper page or repository presentation.

Written by the indexing model from the issue text.

Description

documentation

Hi,

Niels here from the open-source team at Hugging Face. Congratulations on your work!

I've made the paper and 7 paper-native evaluations available on Papers with Code.

The paper has results on Coding Agents, Agents, Image Understanding, and 1 additional task page.

The GPT-5.6 Sol results currently rank first on DeepSWE and GPQA Diamond.

The GPT-5.6 Sol (with tools) result currently ranks first on MMMU-Pro.

The GPT-5.6 Sol Ultra result currently ranks first on Terminal Bench 2.1.

Would it be possible to verify these results and let me know if any score, model name, benchmark protocol, or openness metadata should be corrected? The imported rows are tied to the paper or its official release artifacts; comparison-table baselines were not added.

You can also edit the task, methods, project page, and GitHub URL directly from the paper page using your Hugging Face account.

If you'd like to showcase the results in your repository README, you can copy these live leaderboard badges (or use the “Copy PwC badge” button in the Results section):

Papers with Code: SOTA on DeepSWE
Papers with Code: SOTA on GPQA Diamond
Papers with Code: SOTA on MMMU-Pro
Papers with Code: SOTA on Terminal Bench 2.1
Papers with Code: #2 on BrowseComp

Kind regards,

Niels

Dominant language
Rust
Stars
125k
Forks
19.5k
Avg merge
1m
Merged PRs (30d)
1k

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from openai/codex

All issues in openai/codex

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.