PolicyEngine / PolicyEngine/policybench
Keep DeepSeek out of public leaderboard until benchmark requests complete reliably
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2
- Forks
- 3
- Avg merge
- 12h 7m
- Merged PRs (30d)
- 13
Description
DeepSeek is configured only as an experimental/manual model path for now, not as a default benchmark model or public pending leaderboard entry.
Reason for exclusion from the public leaderboard:
- Credentials work, and a raw DeepSeek Flash toy JSON call succeeds.
- A real PolicyBench one-household, one-output
deepseek-v4-flashsmoke completed, but took about 109 seconds and used 8,207 reasoning tokens for one requested output. deepseek-v4-prodid not finish the same one-household, one-output smoke within 150 seconds.- Earlier full multi-output PolicyBench attempts timed out or hung.
Current repo policy:
- DeepSeek remains in
EXPERIMENTAL_MODELSso it can be manually probed if the API behavior improves. - DeepSeek is excluded from default runs, app model metadata, and public pending-model messaging.
Acceptance criteria before adding it to the public leaderboard:
- Complete a representative multi-output smoke without manual repair or process intervention.
- Produce parseable JSON with nonempty explanations.
- Have latency and token usage that make a 100-household run practical.
- Pass the same analysis/rebuild path as other public leaderboard models.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating EXPERIMENTAL_MODELS, the default-run configuration, app model metadata, and public pending-model messaging. Run the representative multi-output PolicyBench smoke and inspect the analysis/rebuild path used by public models. Done means DeepSeek remains excluded until it produces parseable, explained outputs with practical latency and token usage without manual intervention.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100