huggingface / huggingface/lighteval

[BUG] assertion error `assert text[: len(left)] == left` on MATH wen Qwen-Math-2.5

Open
#345 0 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
```
Greedy generation: 100%|█████████████████████████████████████████████████████████████████████████████████| 1
Splits: 100%|███████████████████████████████████████████████████████████████████████████████████████████████Splits: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [1:01:38<00:00, 3698.19s/it]
WARNING:lighteval.logging.hierarchical_logger: } [1:01:40.194374]
WARNING:lighteval.logging.hierarchical_logger:} [1:02:01.667415]
Traceback (most recent call last):
File "/home//miniconda3/envs/test/bin/lighteval", line 8, in
sys.exit(cli_evaluate())
^^^^^^^^^^^^^^
File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/__main__.py", line 58, in cli_evaluate
main_accelerate(args)
File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/logging/hierarchical_logger.py", line 175, in wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/main_accelerate.py", line 85, in main
pipeline.evaluate()
File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/pipeline.py", line 236, in evaluate
self._compute_metrics(sample_id_to_responses)
File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/pipeline.py", line 288, in _compute_metrics
metrics = compute_metric(results=sample_responses, formatted_doc=doc, metrics=metric_category_metrics)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/metrics/__init__.py", line 126, in apply_generative_metric
outputs.update(metric.compute(golds=golds, predictions=preds, formatted_doc=formatted_doc))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/metrics/utils.py", line 75, in compute
return {self.metric_name: self.sample_level_fn(**kwargs)} # result, formatted_doc,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/metrics/metrics_sample.py", line 881, in compute
all_answers.append(self.get_processed_pred(pred=pred))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/metrics/metrics_sample.py", line 902, in get_processed_pred
pred = self.normalize_pred(pred)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/metrics/normalizations.py", line 269, in math_normalizer
text = _remove_boxed(_last_boxed_only_string(text))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home//miniconda3/envs/test/lib/python3.11/site-packages/lighteval/metrics/normalizations.py", line 108, in _remove_boxed
assert text[: len(left)] == left
^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
```

## To Reproduce
```
lighteval accelerate \
--model_args "pretrained=Qwen/Qwen2.5-Math-7B" \
--tasks "lighteval|math:algebra|0|0" \
--override_batch_size 1 \
--output_dir="./evals/"
```

## Version info
`lighteval==0.5.0`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.