globalwordnet / globalwordnet/semcor

Negated "can" splits into "cann't" (extra n) instead of "can't"

Open
#42 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 22h
Merged PRs (30d)
24

Description

Part of #5. Found by `semcor-compare-brown-nolines`'s word-level diff
against `brown_nolines.txt` (see PR for that tool).

Whenever a negated `can` is split into two tokens, this corpus renders it
as `cann't` (double `n`) instead of `can't`. `nltk.corpus.brown` keeps
`can't` as a single token (tag `MD*`), so this split is this corpus's own
convention -- the same one used for `don't` -> `do`/`n't`,
`isn't` -> `is`/`n't`, etc. But `can't` is only 5 characters
(`c`-`a`-`n`-`'`-`t`): the correct split is `ca`/`n't` (2 + 3), the way
Penn-Treebank-style tokenization does it. This corpus instead keeps `can`
(3 chars) as the first token and starts the second at `n't` (3 chars),
duplicating the shared `n` and producing a 6-character `cann't` when the
two token spans are concatenated.

An exhaustive scan of `lemmas`/`tokens` across `data/*.yaml` (matching
`lemmas[i-1] == "can"` immediately followed by `lemmas[i] == "n't"`, with
adjacent token spans) found **111 confirmed instances across 73 files** --
all the same shape:

```
lemmas: [..., "can", "n't", ...]
pos: [..., "MD", "RB", ...]
tokens: [..., [56, 59], [59, 62], ...] # spans are flush, no gap
text: '...he cann''t help it.' # "can" + "n't" concatenated
```

e.g. `data/fiction_science/br-m01.yaml`: `he cann't help it.` (Brown:
`he can't help it.`), `data/fiction_mystery/br-l08.yaml`: `And cann't say
I blame you` (Brown: `And can't say I blame you`).

This affects both `text` (needs the extra `n` removed) and the first
token's own `tokens` span (needs shortening from `can` (3 chars) to `ca`
(2 chars), with the second token's span starting one character earlier
to absorb the freed-up `n` into `n't`). `lemmas`/`pos` themselves
(`"can"`/`MD`, `"n't"`/`RB`) don't need to change, since the lemma is
already correct -- only the surface character span is wrong.

Two adjacent-but-different cases turned up in the same scan and are
**not** included in the 111 above, since they're a different bug:
`data/fiction_adventure/br-n16.yaml` has `lemmas[i-1] == "can"` but the
actual surface token is `cai` (not `can` at all), and
`data/fiction_romance/br-p09.yaml` has surface `could` where the lemma
says `can`. Filed here for visibility but need their own look.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the 111 confirmed cases in data/*.yaml, using the adjacent lemmas "can"/"n't" and flush token spans to locate them. Check the examples in data/fiction_science/br-m01.yaml and data/fiction_mystery/br-l08.yaml, then verify that text and token spans are corrected while lemmas and POS remain unchanged. Keep the cai and could cases separate and confirm all 73 affected files pass the corpus comparison.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.