CentreForDigitalHumanities / CentreForDigitalHumanities/langpro-annotator
Sentecnes shared by NLI problems
- Dominant language
- TypeScript
- Stars
- 0
- Forks
- 0
- Avg merge
- 5d 11h
- Merged PRs (30d)
- 2
Description
NLI datasets usually contain NLI problems `` that share sentences.
For example, this is the case for the SICK, FraCaS, and SNLI datasets.
```
SICK-56 SICK-58
P: Three kids are sitting in the leaves P: Children in red shirts are playing in the leaves
H: Three kids are jumping in the leaves H: Three kids are sitting in the leaves
```
The question is whether to distinguish or not such identical sentences from each other based on their corresponding NLI problem IDs and the sentence position (e.g., `56P` vs `58H`).
Pros and cons of not distinguishing identical sentences.
➕ annotation economy: no need to annotate (e.g., correct LLFs or parse trees) multi-occurring sentences several times;
➕ complicates the annotation workflow;
➖ not accounting for ambiguity: If a sentence is ambiguous (e.g., three kids and leaves), and different meanings are relevant for different NLI problems.
For simplicity, I suggest not distinguishing sentences based on their involvement in other NLI problems.
The annotation process will show us the trade-off of this decision.
Later, we can always switch to the distinguishing option more smoothly than in the opposite direction.
Contributor guide
Assessment
This issue has not been assessed yet.