huggingface / huggingface/lighteval

[BUG] incorrect type hints such as `callable`

Open
#759 0 comments 0 reactions 0 assignees View on GitHub
bug documentation feature
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug

Reading the code, it's clear this code base can really benefit from type checking, running vanilla `mypy` reveals over 1.2k type errors.

For example, `callable` is not a valid type hint: https://github.com/huggingface/lighteval/blob/v0.9.2/src/lighteval/metrics/utils/metric_utils.py#L62-L63

## To Reproduce

```bash
uv sync --extra dev
uv pip install mypy==0.15.0
mypy src
```

## Expected behavior

Type hints to be correct.

## Version info

`lighteval==0.9.2`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.