bigscience-workshop / bigscience-workshop/promptsource

Behavior in cases the template is not applicable

Open
#749 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3k
Forks
375
PR merge metrics
No merged PRs in 30d

Description

If the template is not applicable for a particular example, the `prompt.apply()` method returns only a single-element list with an empty string instead of a two-element list with the prompted input and the reference answer. This contradicts the [docstring](https://github.com/bigscience-workshop/promptsource/blob/f5c3977626c155823468854479605dbafcbc4426/promptsource/templates.py#L163) which says that two elements are returned.

Should we just update the docstring, or handle this case with more care?

**Example:** the [e2e_nlg_cleaned](https://github.com/bigscience-workshop/promptsource/blob/main/promptsource/templates/e2e_nlg_cleaned/templates.yaml) dataset with QA-style templates such as _food_qa_ (1acabbc3-c9b9-4624-a684-29faeccff46f). This manifests in the web interface by displaying "∅∅∅ Blank result" instead of an input-target pair.

![screen-2022-04-27-11-30-16](https://user-images.githubusercontent.com/8361334/165488260-325cc328-e757-4e94-8b56-74bfdda123b2.png)

Contributor guide

Open the contributing guide

Research direction

Read the prompt.apply() docstring in promptsource/templates.py and reproduce the behavior with the e2e_nlg_cleaned templates.yaml entry for food_qa. Trace how the web interface renders the returned value, then determine whether the applicable behavior should consistently produce an input-target pair and verify that the interface no longer shows a blank result.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.