huggingface / huggingface/lighteval
[BUG] Incompatibility with latest version of xxhash
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
It looks like [xxHash](https://pypi.org/project/xxhash/) is installed with lighteval through the dependency on the `datasets` library, however the latest release of xxHash (4.0) is not compatible with the latest release of lighteval (0.13).
Although xxhash is not listed as direct requirement of lighteval, it is directly imported by lighteval.
Lighteval relies on a now-unsupported feature, which is passing string directly for hashing.
On xxHash side, this raises:
`TypeError: Strings must be encoded before hashing`
I would recommend fixing this issue as well as listing xxhash as direct dependency.
## To Reproduce
```python
from lighteval.logging.info_loggers import DetailsLogger
from lighteval.models.model_output import ModelResponse
from lighteval.tasks.requests import Doc
details_logger = DetailsLogger()
details_logger.log(
task_name="reproducer|0",
doc=Doc(query="xxhash compatibility reproducer", choices=[], gold_index=0),
model_response=ModelResponse(input_tokens=[1], output_tokens=[[2]]),
metrics={})
print("LightEval detail logging completed successfully.")
```
If it could help with the fix, this is the traceback of the error when running our test suite:
```
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/brevitas_examples/test_llm.py:146: in wrapper_main
results, model = quantize_llm(args, extra_args=extra_args)
src/brevitas_examples/llm/main.py:747: in quantize_llm
few_shot_eval_results = run_lighteval(
src/brevitas_examples/llm/eval_lighteval.py:277: in run_lighteval
pipeline.evaluate()
.nox/tests_brevitas_examples_llm_lighteval-3-10-jit_disabled-pytorch_2-7-1/lib/python3.10/site-packages/lighteval/pipeline.py:291: in evaluate
self._compute_metrics(outputs)
.nox/tests_brevitas_examples_llm_lighteval-3-10-jit_disabled-pytorch_2-7-1/lib/python3.10/site-packages/lighteval/pipeline.py:399: in _compute_metrics
self.evaluation_tracker.details_logger.log(task_name, doc, response, output)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = DetailsLogger(hashes=defaultdict(, {}), compiled_hashes=defaultdict( None:
"""Stores the relevant information for one sample of one task to the total list of samples stored in the DetailsLogger.
Args:
task_name (str): Name of the current task of interest.
doc (Doc): Current sample that we want to store.
model_response (ModelResponse): Model outputs for the current sample
metrics (dict): Model scores for said sample on the current task's metrics.
"""
detail = self.Detail(doc, model_response, metrics)
self.details[task_name].append(detail)
hash = self.Hash()
> hash.example = xxhash.xxh64(doc.query).hexdigest()
E TypeError: Strings must be encoded before hashing
```
## Expected behavior
Compatibility between lighteval and xxhash or alternative hashing solution.
## Version info
lighteval==0.13
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the DetailsLogger.log method and the hash.example assignment shown in the traceback, then run the reproducer with xxhash 4.0 installed. Confirm the failing hashing path works with the current xxhash release and that xxhash is declared as a direct dependency; the reproduction should complete successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100