huggingface / huggingface/lighteval
Align GPQA zero-shot / few-shot prompts with paper?
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
GPQA uses a fixed prompt for zero-shot and few-shot evaluation (see Appendix A.3.1 of the [paper](https://arxiv.org/pdf/2311.12022.pdf)). For example, this is the format of the zero-shot prompt:
```
What is the correct answer to this question: {QUESTION}
Choices:
(A) {CHOICE_A}
(B) {CHOICE_B}
(C) {CHOICE_C}
(D) {CHOICE_D}
Format your response as follows: "The correct answer is (insert answer here)".
```
In particular, note the final instruction to format the answer and that they also mention that they use a regex parser to extract the desired answer:
> We extracted answers from the model response using a simple regex matching phrases like ‘answer is’, ‘answer:’ etc.
However, inspecting the details from `lighteval` I see we have the following for zero-shot:
```
Select the correct answer to the following questions.
Question: Identify the final product produced when cyclobutyl(cyclopropyl)methanol reacts with phosphoric acid in water.
A. spiro[3.4]oct-5-ene
B. 1,2-dimethylcyclohexa-1,4-diene
C. 1,2,3,4,5,6-hexahydropentalene
D. [1,1'-bi(cyclobutan)]-1-ene
Answer:
```
The trouble with this format is that it heavily penalises chat models which will typically produce a long-winded explanation and thus fail to produce the expected format (A,B,C,D) that a base model typically will.
Another thing I noticed is that the paper uses a fixed few-shot CoT prompt ([link](https://github.com/idavidrein/gpqa/blob/main/prompts/chain_of_thought.txt)) which can be adapted to pure few-shot by removing the reasoning steps. However, it seems that `lighteval` samples fewshot prompts from the dataset and I wonder if it makes sense to align the evaluation in both cases (zeroshot / fewshot) in line with the paper?
Happy to take a stab at this one if you agree!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.