open-compass / open-compass/opencompass
[Bug] Please make sure `./data/safety.txt` is correct
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.5k
- Forks
- 869
- Avg merge
- 17h 52m
- Merged PRs (30d)
- 13
Description
Prerequisite
- I have searched Issues and Discussions but cannot get the expected help.
- The bug has not been fixed in the latest version.
Type
I'm evaluating with the officially supported tasks/models/datasets.
Environment
Name: opencompass
Version: 0.3.8
Summary: A comprehensive toolkit for large model evaluation
Home-page: https://github.com/open-compass/opencompass
Author: OpenCompass Contributors
Author-email:
License: Apache License 2.0
Location: /work/opencompass
Editable project location: /work/opencompass
Requires: absl-py, accelerate, cpm_kernels, datasets, einops, evaluate, func_timeout, fuzzywuzzy, gradio-client, h5py, httpx, huggingface_hub, immutabledict, importlib-metadata, jieba, json5, jsonlines, mmengine-lite, nltk, numpy, openai, OpenCC, opencv-python-headless, pandas, prettytable, protobuf, pyext, python-Levenshtein, rank_bm25, rapidfuzz, requests, retrying, rich, rouge, rouge_chinese, rouge_score, sacrebleu, scikit_learn, seaborn, sentence_transformers, tabulate, tiktoken, timeout_decorator, tokenizers, torch, tqdm, transformers, tree-sitter, tree_sitter_languages, typer
Required-by:
Reproduces the problem - code/configuration sample
python run.py
--models vllm_qwen2p5_7b_instruct
--datasets safety_gen\
Reproduces the problem - command or script
python run.py
--models vllm_qwen2p5_7b_instruct
--datasets safety_gen\
Reproduces the problem - error message
Traceback (most recent call last):
File "/work/opencompass/run.py", line 4, in
main()
File "/work/opencompass/opencompass/cli/main.py", line 299, in main
tasks = partitioner(cfg)
File "/work/opencompass/opencompass/partitioners/base.py", line 83, in call
tasks = self.partition(**model_and_dataset_args,
File "/work/opencompass/opencompass/partitioners/num_worker.py", line 69, in partition
dataset_size = self.get_size(dataset)
File "/work/opencompass/opencompass/partitioners/num_worker.py", line 141, in get_size
dataset = build_dataset_from_cfg(dataset)
File "/work/opencompass/opencompass/utils/build.py", line 13, in build_dataset_from_cfg
return LOAD_DATASET.build(dataset_cfg)
File "/usr/local/lib/python3.10/dist-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/usr/local/lib/python3.10/dist-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
obj = obj_cls(**args) # type: ignore
File "/work/opencompass/opencompass/datasets/base.py", line 12, in init
self.dataset = self.load(**kwargs)
File "/work/opencompass/opencompass/datasets/safety.py", line 14, in load
path = get_data_path(path, local_mode=True)
File "/work/opencompass/opencompass/utils/datasets.py", line 30, in get_data_path
return download_dataset(local_path, cache_dir)
File "/work/opencompass/opencompass/utils/datasets.py", line 90, in download_dataset
assert dataset_name, f'No valid url for {data_path}!\n' + \
AssertionError: No valid url for ./data/safety.txt!
Please make sure ./data/safety.txt is correct
Other information
What is the dataset corresponding to "safety"? I have already downloaded the relevant zip file as per the documentation.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the safety_gen dataset configuration and trace its loading through opencompass/datasets/safety.py and opencompass/utils/datasets.py, using the reported run.py command to reproduce the failure. Check how ./data/safety.txt is expected to map to a downloadable dataset and compare that with the documentation's zip instructions. Done means the supported safety_gen evaluation resolves the dataset path without the reported assertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100