huggingface / huggingface/lighteval
Fix Doc init and missing metadata in Summarization tasks
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
The `cnn_dm_prompt` and `xsum_prompt` functions in the summarization task fail with a `TypeError` because they do not provide the required `choices` and `gold_index` arguments to the `Doc` constructor. Furthermore, metrics such as extractiveness and faithfulness fail because the source text is not passed in the `Doc.specific` field.
## To Reproduce
```python
from lighteval.tasks.tasks.summarization import cnn_dm_prompt
# Using a standard record from CNN/DM
# doc = cnn_dm_prompt({"article": "...", "highlights": "..."}, "cnn_dm")
# TypeError: Doc.__init__() missing 2 required positional arguments: 'choices' and 'gold_index'
```
## Expected behavior
- The `Doc` object should be initialized with placeholder `choices=[""]` and `gold_index=0` to satisfy the API for generative tasks.
- The `specific` field should include `{"text": line["article"]}` (or document) for metric internal access.
## Version info
- OS: mac
- Lighteval version: main (local development)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.