bigscience-workshop / bigscience-workshop/promptsource
Behavior in cases the template is not applicable
- Dominant language
- Python
- Stars
- 3k
- Forks
- 375
- PR merge metrics
- No merged PRs in 30d
Description
If the template is not applicable for a particular example, the `prompt.apply()` method returns only a single-element list with an empty string instead of a two-element list with the prompted input and the reference answer. This contradicts the [docstring](https://github.com/bigscience-workshop/promptsource/blob/f5c3977626c155823468854479605dbafcbc4426/promptsource/templates.py#L163) which says that two elements are returned.
Should we just update the docstring, or handle this case with more care?
**Example:** the [e2e_nlg_cleaned](https://github.com/bigscience-workshop/promptsource/blob/main/promptsource/templates/e2e_nlg_cleaned/templates.yaml) dataset with QA-style templates such as _food_qa_ (1acabbc3-c9b9-4624-a684-29faeccff46f). This manifests in the web interface by displaying "∅∅∅ Blank result" instead of an input-target pair.

Contributor guide
Research direction
Read the prompt.apply() docstring in promptsource/templates.py and reproduce the behavior with the e2e_nlg_cleaned templates.yaml entry for food_qa. Trace how the web interface renders the returned value, then determine whether the applicable behavior should consistently produce an input-target pair and verify that the interface no longer shows a blank result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100