globalwordnet / globalwordnet/semcor
Negated "can" splits into "cann't" (extra n) instead of "can't"
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 24
Description
Part of #5. Found by `semcor-compare-brown-nolines`'s word-level diff
against `brown_nolines.txt` (see PR for that tool).
Whenever a negated `can` is split into two tokens, this corpus renders it
as `cann't` (double `n`) instead of `can't`. `nltk.corpus.brown` keeps
`can't` as a single token (tag `MD*`), so this split is this corpus's own
convention -- the same one used for `don't` -> `do`/`n't`,
`isn't` -> `is`/`n't`, etc. But `can't` is only 5 characters
(`c`-`a`-`n`-`'`-`t`): the correct split is `ca`/`n't` (2 + 3), the way
Penn-Treebank-style tokenization does it. This corpus instead keeps `can`
(3 chars) as the first token and starts the second at `n't` (3 chars),
duplicating the shared `n` and producing a 6-character `cann't` when the
two token spans are concatenated.
An exhaustive scan of `lemmas`/`tokens` across `data/*.yaml` (matching
`lemmas[i-1] == "can"` immediately followed by `lemmas[i] == "n't"`, with
adjacent token spans) found **111 confirmed instances across 73 files** --
all the same shape:
```
lemmas: [..., "can", "n't", ...]
pos: [..., "MD", "RB", ...]
tokens: [..., [56, 59], [59, 62], ...] # spans are flush, no gap
text: '...he cann''t help it.' # "can" + "n't" concatenated
```
e.g. `data/fiction_science/br-m01.yaml`: `he cann't help it.` (Brown:
`he can't help it.`), `data/fiction_mystery/br-l08.yaml`: `And cann't say
I blame you` (Brown: `And can't say I blame you`).
This affects both `text` (needs the extra `n` removed) and the first
token's own `tokens` span (needs shortening from `can` (3 chars) to `ca`
(2 chars), with the second token's span starting one character earlier
to absorb the freed-up `n` into `n't`). `lemmas`/`pos` themselves
(`"can"`/`MD`, `"n't"`/`RB`) don't need to change, since the lemma is
already correct -- only the surface character span is wrong.
Two adjacent-but-different cases turned up in the same scan and are
**not** included in the 111 above, since they're a different bug:
`data/fiction_adventure/br-n16.yaml` has `lemmas[i-1] == "can"` but the
actual surface token is `cai` (not `can` at all), and
`data/fiction_romance/br-p09.yaml` has surface `could` where the lemma
says `can`. Filed here for visibility but need their own look.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the 111 confirmed cases in data/*.yaml, using the adjacent lemmas "can"/"n't" and flush token spans to locate them. Check the examples in data/fiction_science/br-m01.yaml and data/fiction_mystery/br-l08.yaml, then verify that text and token spans are corrected while lemmas and POS remain unchanged. Keep the cai and could cases separate and confirm all 73 affected files pass the corpus comparison.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100