PolicyEngine / PolicyEngine/policybench

Keep DeepSeek out of public leaderboard until benchmark requests complete reliably

Open
#4 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2
Forks
3
Avg merge
12h 7m
Merged PRs (30d)
13

Description

DeepSeek is configured only as an experimental/manual model path for now, not as a default benchmark model or public pending leaderboard entry.

Reason for exclusion from the public leaderboard:

  • Credentials work, and a raw DeepSeek Flash toy JSON call succeeds.
  • A real PolicyBench one-household, one-output deepseek-v4-flash smoke completed, but took about 109 seconds and used 8,207 reasoning tokens for one requested output.
  • deepseek-v4-pro did not finish the same one-household, one-output smoke within 150 seconds.
  • Earlier full multi-output PolicyBench attempts timed out or hung.

Current repo policy:

  • DeepSeek remains in EXPERIMENTAL_MODELS so it can be manually probed if the API behavior improves.
  • DeepSeek is excluded from default runs, app model metadata, and public pending-model messaging.

Acceptance criteria before adding it to the public leaderboard:

  • Complete a representative multi-output smoke without manual repair or process intervention.
  • Produce parseable JSON with nonempty explanations.
  • Have latency and token usage that make a 100-household run practical.
  • Pass the same analysis/rebuild path as other public leaderboard models.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating EXPERIMENTAL_MODELS, the default-run configuration, app model metadata, and public pending-model messaging. Run the representative multi-output PolicyBench smoke and inspect the analysis/rebuild path used by public models. Done means DeepSeek remains excluded until it produces parseable, explained outputs with practical latency and token usage without manual intervention.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.