lm-sys / lm-sys/FastChat

[llm_judge] key mismatch during match list deduplication when mode=pairwise-baseline

Open
#2,332 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

Hi, I'm getting an error during the deduplication of the matches.

> Traceback (most recent call last):
File ".../gen_judgment.py", line 302, in
[
File ".../gen_judgment.py", line 303, in
f"{e['question_id']}{e['model']}{e['judge'][0]}{e['judge'][1]}{e['turn']}"
KeyError: 'model'

It works fine when mode=single but for mode=pairwise-baseline, it has a different set of keys, which leads to a mismatch.

For mode=single:
> dict_keys(['question_id', 'model', 'judge', 'user_prompt', 'judgment', 'score', 'turn', 'tstamp'])

For mode=pairwise-baseline:
> dict_keys(['question_id', 'model_1', 'model_2', 'g1_winner', 'g2_winner', 'judge', 'g1_user_prompt', 'g1_judgment', 'g2_user_prompt', 'g2_judgment', 'turn', 'tstamp'])

Here is a command I used
```
python gen_judgment.py \
--mode pairwise-baseline \
--baseline-model vicuna-7b-v1.3 \
--model-list mpt-7b-chat \
--parallel 2
```

Any pointer would be appreciated!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in gen_judgment.py around lines 302-303 and compare the deduplication key used for mode=single with the keys produced by mode=pairwise-baseline. Reproduce the issue with the command in the report. Done means pairwise-baseline no longer raises KeyError during match-list deduplication while single mode continues to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.