huggingface / huggingface/lighteval

Fix Doc init and missing metadata in Summarization tasks

Open
#1,134 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
The `cnn_dm_prompt` and `xsum_prompt` functions in the summarization task fail with a `TypeError` because they do not provide the required `choices` and `gold_index` arguments to the `Doc` constructor. Furthermore, metrics such as extractiveness and faithfulness fail because the source text is not passed in the `Doc.specific` field.

## To Reproduce
```python
from lighteval.tasks.tasks.summarization import cnn_dm_prompt
# Using a standard record from CNN/DM
# doc = cnn_dm_prompt({"article": "...", "highlights": "..."}, "cnn_dm")
# TypeError: Doc.__init__() missing 2 required positional arguments: 'choices' and 'gold_index'
```

## Expected behavior
- The `Doc` object should be initialized with placeholder `choices=[""]` and `gold_index=0` to satisfy the API for generative tasks.
- The `specific` field should include `{"text": line["article"]}` (or document) for metric internal access.

## Version info
- OS: mac
- Lighteval version: main (local development)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.