deepseek-ai / deepseek-ai/DeepSeek-Math

minif2f-Isabella acc

Open
#30 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.4k
Forks
592
PR merge metrics
No merged PRs in 30d

Description

I use the [docker image](https://github.com/albertqjiang/Portal-to-ISAbelle/blob/main/docker/docker_tutorial.md) from the PISA repository and the prediction file from output.zip of your repository(path/outputs/DeepSeekMath-Base/miniF2F-Isabelle-test/results/cot/predictions.json). But my acc is about 10% compared to the result of 24.6%. I'd like to know what is the reason for this difference.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked docker/docker_tutorial.md in the PISA repository and path/outputs/DeepSeekMath-Base/miniF2F-Isabelle-test/results/cot/predictions.json. Reproduce the reported 10% accuracy using that image and prediction file, then compare the evaluation procedure and inputs with the stated 24.6% result; done means identifying and documenting the source of the discrepancy.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.