huggingface / huggingface/lighteval
[FT] Introduce a static 'lint' command to validate custom tasks before execution
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Issue encountered
When developers create custom evaluation tasks using `LightevalTaskConfig`, there is currently no fast way to perform static structural validation before executing the pipeline. If a developer forgets the `TASKS_TABLE` module export, requests an `evaluation_split` that isn't declared in `hf_avail_splits`, or fails to return a `Doc` object from their `prompt_function`, the error is only caught at runtime (usually triggering a messy traceback when `lighteval tasks inspect` attempts to iterate over `eval_docs()`). This delays developer feedback loops, especially when pushing heavy evaluation jobs to headless compute nodes.
## Solution/Feature
I propose introducing a lightweight static validation utility, exposed via a new `lighteval tasks lint ` CLI command.
This feature would statically validate both the `LightevalTaskConfig` structure and the module export boundary *without* downloading datasets or executing the pipeline.
Specifically, the pure-Python linter would verify:
1. **Module Export Integrity**: Asserts that the script correctly exposes a `list` named `TASKS_TABLE`.
2. **Split Verification**: Cross-references `evaluation_splits` and `few_shots_split` strictly against the declared `hf_avail_splits`.
3. **Metric Conformity**: Validates that `config.metrics` contains populated grading instances.
4. **Prompt Function Signature**: Uses `inspect.signature` to statically ensure the `prompt_function` accepts the required row dictionary and correctly returns the expected `Doc` class.
I have already built the pure-Python validation logic locally, along with a `pytest` suite that uses `types.ModuleType` to mock the namespace boundaries and verify the exports. If this aligns with the core roadmap, I would be more than happy to open a Pull Request!
## Possible alternatives
1. **Relying strictly on `lighteval tasks inspect`**: Leaving the current behavior as-is. The downside is that `inspect` actually instantiates the datasets and loops through data, which is computationally heavier than a fast static check for spelling/typing errors.
2. **Pydantic Validation**: Refactoring `LightevalTaskConfig` to inherit from `pydantic.BaseModel` to catch typing errors on instantiation. However, this introduces a heavier external dependency footprint and still wouldn't solve the `TASKS_TABLE` module-export boundary issue.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.