open-compass / open-compass/opencompass

[Bug] Please make sure `./data/safety.txt` is correct

Open
#1,787 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.5k
Forks
869
Avg merge
17h 52m
Merged PRs (30d)
13

Description

Prerequisite
Type

I'm evaluating with the officially supported tasks/models/datasets.

Environment

Name: opencompass
Version: 0.3.8
Summary: A comprehensive toolkit for large model evaluation
Home-page: https://github.com/open-compass/opencompass
Author: OpenCompass Contributors
Author-email:
License: Apache License 2.0
Location: /work/opencompass
Editable project location: /work/opencompass
Requires: absl-py, accelerate, cpm_kernels, datasets, einops, evaluate, func_timeout, fuzzywuzzy, gradio-client, h5py, httpx, huggingface_hub, immutabledict, importlib-metadata, jieba, json5, jsonlines, mmengine-lite, nltk, numpy, openai, OpenCC, opencv-python-headless, pandas, prettytable, protobuf, pyext, python-Levenshtein, rank_bm25, rapidfuzz, requests, retrying, rich, rouge, rouge_chinese, rouge_score, sacrebleu, scikit_learn, seaborn, sentence_transformers, tabulate, tiktoken, timeout_decorator, tokenizers, torch, tqdm, transformers, tree-sitter, tree_sitter_languages, typer
Required-by:

Reproduces the problem - code/configuration sample

python run.py
--models vllm_qwen2p5_7b_instruct
--datasets safety_gen\

Reproduces the problem - command or script

python run.py
--models vllm_qwen2p5_7b_instruct
--datasets safety_gen\

Reproduces the problem - error message

Traceback (most recent call last):
File "/work/opencompass/run.py", line 4, in
main()
File "/work/opencompass/opencompass/cli/main.py", line 299, in main
tasks = partitioner(cfg)
File "/work/opencompass/opencompass/partitioners/base.py", line 83, in call
tasks = self.partition(**model_and_dataset_args,
File "/work/opencompass/opencompass/partitioners/num_worker.py", line 69, in partition
dataset_size = self.get_size(dataset)
File "/work/opencompass/opencompass/partitioners/num_worker.py", line 141, in get_size
dataset = build_dataset_from_cfg(dataset)
File "/work/opencompass/opencompass/utils/build.py", line 13, in build_dataset_from_cfg
return LOAD_DATASET.build(dataset_cfg)
File "/usr/local/lib/python3.10/dist-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/usr/local/lib/python3.10/dist-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
obj = obj_cls(**args) # type: ignore
File "/work/opencompass/opencompass/datasets/base.py", line 12, in init
self.dataset = self.load(**kwargs)
File "/work/opencompass/opencompass/datasets/safety.py", line 14, in load
path = get_data_path(path, local_mode=True)
File "/work/opencompass/opencompass/utils/datasets.py", line 30, in get_data_path
return download_dataset(local_path, cache_dir)
File "/work/opencompass/opencompass/utils/datasets.py", line 90, in download_dataset
assert dataset_name, f'No valid url for {data_path}!\n' + \
AssertionError: No valid url for ./data/safety.txt!
Please make sure ./data/safety.txt is correct

Other information

What is the dataset corresponding to "safety"? I have already downloaded the relevant zip file as per the documentation.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the safety_gen dataset configuration and trace its loading through opencompass/datasets/safety.py and opencompass/utils/datasets.py, using the reported run.py command to reproduce the failure. Check how ./data/safety.txt is expected to map to a downloadable dataset and compare that with the documentation's zip instructions. Done means the supported safety_gen evaluation resolves the dataset path without the reported assertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.