facebookresearch / facebookresearch/deepconf

about GPQA-D prompt in Deepseek-8B

Open
#12 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
412
Forks
62
PR merge metrics
No merged PRs in 30d

Description

Regarding the prompt used for evaluating the DeepSeek-8B model on the GPQA-D dataset, I noticed while reading the paper that you did not specify exactly what prompt was used for GPQA-D on DeepSeek-8B. Could you provide the exact prompt? I found that when I use the default prompt to test DeepSeek-8B on GPQA-D, the results differ significantly from the numbers reported in the paper. Interestingly, using the same prompt to test the Qwen3-32B model on GPQA-D does not cause this issue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.