globalwordnet / globalwordnet/semcor
General sweep: missing space between two already-adjacent (flush) tokens
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 24
Description
Mirror image of #63/#64's general sweep: instead of finding N of our words that should collapse to 1 reference word (an extra separator to remove), look for the reverse -- 1 of our words that should split into N reference words (a separator missing entirely), using the same whole-document \`difflib\` alignment plus an exact-concatenation check.
Examples reported: \`(Kika)de\` for Brown's \`(Kika) de\`, \`23dward\` for \`23d ward\`, \`21stand\` for \`21st and\`.
**205 raw candidates** found. Of these:
- **2 are reference-side typos**, not real bugs: bley-vroman's \`brown_nolines.txt\` itself has \`to d o whatever\` and \`the s ame level\` (confirmed directly in the reference file) where the *real* word is \`do\`/\`same\` -- fixing our correctly-spelled data to match would introduce nonsense. Left alone.
- **180 confirmed and fixed**: in every one of these, our data *already* has the two (or more) words as separate, correctly-tokenized, already-flush tokens -- the bug is purely that the space between them is missing, not a tokenization problem. Applied via a new \`semcor-fix-missing-space-gap\`, a pure single-character insertion (mirror image of \`semcor-fix-genitive-gap\`'s deletion): \`lemmas\`/\`pos\`/every sense-key layer untouched, only \`text\` and the shifted \`tokens\` offsets change.
- **23 left unresolved**, deliberately: these need an actual token *split* (\`U.S.\` -> \`U.\` + \`S.\`, \`wildlife.\` -> \`wild\` + \`life.\`, ...), which raises the same kind of editorial question #43/#46 hit for token *merges* -- what lemma/pos/sense (if any) does each new half get. Left for a follow-up, individual look rather than a blanket rule.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the 23 unresolved flush-token candidates left after the semcor-fix-missing-space-gap sweep, using the whole-document difflib alignment and the reference corpus for comparison. The work is done when each selected split has an explicit editorial decision for its new halves' lemma, POS, and sense data, while the two reference-side typos remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100