CentreForDigitalHumanities / CentreForDigitalHumanities/langpro-annotator

Sentecnes shared by NLI problems

Open
#25 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
0
Forks
0
Avg merge
5d 11h
Merged PRs (30d)
2

Description

NLI datasets usually contain NLI problems `` that share sentences.
For example, this is the case for the SICK, FraCaS, and SNLI datasets.

```
SICK-56 SICK-58
P: Three kids are sitting in the leaves P: Children in red shirts are playing in the leaves
H: Three kids are jumping in the leaves H: Three kids are sitting in the leaves
```

The question is whether to distinguish or not such identical sentences from each other based on their corresponding NLI problem IDs and the sentence position (e.g., `56P` vs `58H`).

Pros and cons of not distinguishing identical sentences.
➕ annotation economy: no need to annotate (e.g., correct LLFs or parse trees) multi-occurring sentences several times;
➕ complicates the annotation workflow;
➖ not accounting for ambiguity: If a sentence is ambiguous (e.g., three kids and leaves), and different meanings are relevant for different NLI problems.

For simplicity, I suggest not distinguishing sentences based on their involvement in other NLI problems.
The annotation process will show us the trade-off of this decision.
Later, we can always switch to the distinguishing option more smoothly than in the opposite direction.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.