lm-sys / lm-sys/FastChat

Support package-data for llm_judge

Open
#3,199 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

It seems better to ensure that data for `llm_judge` is included in the package, even if installed via pip.

Install with pip
```
pip3 install "fschat[model_worker,llm_judge]"
```

Script for using llm_judge as module
```
python3 -m fastchat.llm_judge.gen_model_answer --model-path lmsys/vicuna-7b-v1.5 --model-id vicuna-7b-v1.5
```

Error
```
File "/usr/local/lib/python3.10/dist-packages/fastchat/llm_judge/gen_model_answer.py", line 288, in
run_eval(
File "/usr/local/lib/python3.10/dist-packages/fastchat/llm_judge/gen_model_answer.py", line 36, in run_eval
questions = load_questions(question_file, question_begin, question_end)
File "/usr/local/lib/python3.10/dist-packages/fastchat/llm_judge/common.py", line 91, in load_questions
with open(question_file, "r") as ques_file:
FileNotFoundError: [Errno 2] No such file or directory: 'data/mt_bench/question.jsonl'
```

The `data` directory does not exist.
```
/usr/local/lib/python3.10/dist-packages/fastchat/llm_judge
├── clean_judgment.py
├── common.py
├── compute_agreement.py
├── download_mt_bench_pregenerated.py
├── gen_api_answer.py
├── gen_judgment.py
├── gen_model_answer.py
├── qa_browser.py
└── show_result.py
```

Expected solution
```
# pyproject.toml
...

[tool.setuptools.package-data]
fastchat.llm_judge = ["data/*"]

...
```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with pyproject.toml and inspect the fastchat/llm_judge/data directory and its packaging configuration. Install the project with the llm_judge extra using pip, then run the reported python3 -m fastchat.llm_judge.gen_model_answer command. Done means the installed package contains the data files and no longer raises FileNotFoundError for data/mt_bench/question.jsonl.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, tooling
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.