Support package-data for llm_judge
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
It seems better to ensure that data for `llm_judge` is included in the package, even if installed via pip.
Install with pip
```
pip3 install "fschat[model_worker,llm_judge]"
```
Script for using llm_judge as module
```
python3 -m fastchat.llm_judge.gen_model_answer --model-path lmsys/vicuna-7b-v1.5 --model-id vicuna-7b-v1.5
```
Error
```
File "/usr/local/lib/python3.10/dist-packages/fastchat/llm_judge/gen_model_answer.py", line 288, in
run_eval(
File "/usr/local/lib/python3.10/dist-packages/fastchat/llm_judge/gen_model_answer.py", line 36, in run_eval
questions = load_questions(question_file, question_begin, question_end)
File "/usr/local/lib/python3.10/dist-packages/fastchat/llm_judge/common.py", line 91, in load_questions
with open(question_file, "r") as ques_file:
FileNotFoundError: [Errno 2] No such file or directory: 'data/mt_bench/question.jsonl'
```
The `data` directory does not exist.
```
/usr/local/lib/python3.10/dist-packages/fastchat/llm_judge
├── clean_judgment.py
├── common.py
├── compute_agreement.py
├── download_mt_bench_pregenerated.py
├── gen_api_answer.py
├── gen_judgment.py
├── gen_model_answer.py
├── qa_browser.py
└── show_result.py
```
Expected solution
```
# pyproject.toml
...
[tool.setuptools.package-data]
fastchat.llm_judge = ["data/*"]
...
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with pyproject.toml and inspect the fastchat/llm_judge/data directory and its packaging configuration. Install the project with the llm_judge extra using pip, then run the reported python3 -m fastchat.llm_judge.gen_model_answer command. Done means the installed package contains the data files and no longer raises FileNotFoundError for data/mt_bench/question.jsonl.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, tooling
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100