huggingface / huggingface/lighteval
[BUG] encounter an ArrowInvalid error while saving experiment tracker
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
encounter an ArrowInvalid error while saving experiment tracker.
The most process of evaluation is done, but error occur when saving.
The error info is as follow:
```
[2025-04-06 18:27:14,942] [[32m INFO[0m]: Saving experiment tracker (evaluation_tracker.py:180)[0m
| Task |Version|Metric|Value | |Stderr|
|-------------|------:|------|-----:|---|-----:|
|all | |em |0.0102|± |0.0020|
| | |qem |0.0110|± |0.0021|
| | |pem |0.1925|± |0.0078|
| | |pqem |0.3937|± |0.0097|
| | |acc |0.2578|± |0.0087|
|helm:med_qa:0| 0|em |0.0102|± |0.0020|
| | |qem |0.0110|± |0.0021|
| | |pem |0.1925|± |0.0078|
| | |pqem |0.3937|± |0.0097|
| | |acc |0.2578|± |0.0087|
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/lc/.conda/envs/lighteval/lib/python3.11/site-packages/lighteval/main_v │
│ llm.py:163 in vllm │
│ │
│ 160 │ │
│ 161 │ results = pipeline.get_results() │
│ 162 │ │
│ ❱ 163 │ pipeline.save_and_push_results() │
│ 164 │ │
│ 165 │ return results │
│ 166 │
│ │
│ /home/lc/.conda/envs/lighteval/lib/python3.11/site-packages/lighteval/pipeli │
│ ne.py:536 in save_and_push_results │
│ │
│ 533 │ def save_and_push_results(self): │
│ 534 │ │ logger.info("--- SAVING AND PUSHING RESULTS ---") │
│ 535 │ │ if self.is_main_process(): │
│ ❱ 536 │ │ │ self.evaluation_tracker.save() │
│ 537 │ │
│ 538 │ def _init_final_dict(self): │
│ 539 │ │ if self.is_main_process(): │
│ │
│ /home/lc/.conda/envs/lighteval/lib/python3.11/site-packages/lighteval/loggin │
│ g/evaluation_tracker.py:201 in save │
│ │
│ 198 │ │ details_datasets: dict[str, Dataset] = {} │
│ 199 │ │ for task_name, task_details in self.details_logger.details.ite │
│ 200 │ │ │ # Create a dataset from the dictionary - we force cast to │
│ ❱ 201 │ │ │ dataset = Dataset.from_list([asdict(detail) for detail in │
│ 202 │ │ │ │
│ 203 │ │ │ # We don't keep 'id' around if it's there │
│ 204 │ │ │ column_names = dataset.column_names │
│ │
│ /home/lc/.conda/envs/lighteval/lib/python3.11/site-packages/datasets/arrow_d │
│ ataset.py:986 in from_list │
│ │
│ 983 │ │ """ │
│ 984 │ │ # for simplicity and consistency wrt OptimizedTypedSequence w │
│ 985 │ │ mapping = {k: [r.get(k) for r in mapping] for k in mapping[0] │
│ ❱ 986 │ │ return cls.from_dict(mapping, features, info, split) │
│ 987 │ │
│ 988 │ @staticmethod │
│ 989 │ def from_csv( │
│ │
│ /home/lc/.conda/envs/lighteval/lib/python3.11/site-packages/datasets/arrow_d │
│ ataset.py:940 in from_dict │
│ │
│ 937 │ │ │ │ ) │
│ 938 │ │ │ arrow_typed_mapping[col] = data │
│ 939 │ │ mapping = arrow_typed_mapping │
│ ❱ 940 │ │ pa_table = InMemoryTable.from_pydict(mapping=mapping) │
│ 941 │ │ if info is None: │
│ 942 │ │ │ info = DatasetInfo() │
│ 943 │ │ info.features = features │
│ │
│ /home/lc/.conda/envs/lighteval/lib/python3.11/site-packages/datasets/table.p │
│ y:758 in from_pydict │
│ │
│ 755 │ │ Returns: │
│ 756 │ │ │ `datasets.table.Table` │
│ 757 │ │ """ │
│ ❱ 758 │ │ return cls(pa.Table.from_pydict(*args, **kwargs)) │
│ 759 │ │
│ 760 │ @classmethod │
│ 761 │ def from_pylist(cls, mapping, *args, **kwargs): │
│ │
│ in pyarrow.lib._Tabular.from_pydict:1968 │
│ │
│ in pyarrow.lib._from_pydict:6337 │
│ │
│ in pyarrow.lib.asarray:402 │
│ │
│ in pyarrow.lib.array:252 │
│ │
│ in pyarrow.lib._handle_arrow_array_protocol:114 │
│ │
│ /home/lc/.conda/envs/lighteval/lib/python3.11/site-packages/datasets/arrow_w │
│ riter.py:229 in __arrow_array__ │
│ │
│ 226 │ │ │ │ out = list_of_np_array_to_pyarrow_listarray(data) │
│ 227 │ │ │ else: │
│ 228 │ │ │ │ trying_cast_to_python_objects = True │
│ ❱ 229 │ │ │ │ out = pa.array(cast_to_python_objects(data, only_1d_fo │
│ 230 │ │ │ # use smaller integer precisions if possible │
│ 231 │ │ │ if self.trying_int_optimization: │
│ 232 │ │ │ │ if pa.types.is_int64(out.type): │
│ │
│ in pyarrow.lib.array:372 │
│ │
│ in pyarrow.lib._sequence_to_array:42 │
│ │
│ in pyarrow.lib.pyarrow_internal_check_status:155 │
│ │
│ in pyarrow.lib.check_status:92 │
╰──────────────────────────────────────────────────────────────────────────────╯
ArrowInvalid: cannot mix list and non-list, non-null values
```
## To Reproduce
I executed the following command to eval Qwen2.5-0.5B-Instruct with med_qa benchmark, but got error.
```bash
NAMESPACE=Qwen
MODEL_NAME=Qwen2.5-0.5B-Instruct #
MODEL=$NAMESPACE/$MODEL_NAME
MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,max_model_length=2048,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:2048,temperature:0.6,top_p:0.95}"
OUTPUT_DIR=data/evals/$MODEL
TASK=med_qa
LOG_FILE=logs/evals/${TASK}_${MODEL_NAME}.log
CUDA_VISIBLE_DEVICES=0 nohup lighteval vllm $MODEL_ARGS "helm|$TASK|0|0" \
--use-chat-template \
--output-dir $OUTPUT_DIR \
> ${LOG_FILE} 2>&1 &
```
## Expected behavior
Save results of evaluation successfully.
## Version info
Ubuntu 24.04
lighteval 0.8.1
cuda 12.4
vllm 0.8.3
torch 2.6.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.