[llm_judge] key mismatch during match list deduplication when mode=pairwise-baseline
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I'm getting an error during the deduplication of the matches.
> Traceback (most recent call last):
File ".../gen_judgment.py", line 302, in
[
File ".../gen_judgment.py", line 303, in
f"{e['question_id']}{e['model']}{e['judge'][0]}{e['judge'][1]}{e['turn']}"
KeyError: 'model'
It works fine when mode=single but for mode=pairwise-baseline, it has a different set of keys, which leads to a mismatch.
For mode=single:
> dict_keys(['question_id', 'model', 'judge', 'user_prompt', 'judgment', 'score', 'turn', 'tstamp'])
For mode=pairwise-baseline:
> dict_keys(['question_id', 'model_1', 'model_2', 'g1_winner', 'g2_winner', 'judge', 'g1_user_prompt', 'g1_judgment', 'g2_user_prompt', 'g2_judgment', 'turn', 'tstamp'])
Here is a command I used
```
python gen_judgment.py \
--mode pairwise-baseline \
--baseline-model vicuna-7b-v1.3 \
--model-list mpt-7b-chat \
--parallel 2
```
Any pointer would be appreciated!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in gen_judgment.py around lines 302-303 and compare the deduplication key used for mode=single with the keys produced by mode=pairwise-baseline. Reproduce the issue with the command in the report. Done means pairwise-baseline no longer raises KeyError during match-list deduplication while single mode continues to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100