globalwordnet / globalwordnet/semcor
General sweep: remaining spurious-gap divergences fixable by deleting one character
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 24
Description
A general, whole-document-alignment scan (align this corpus's reconstructed word list against \`brown-nolines.txt\` per document with \`difflib\`, then look for any \`replace\` block where N (>1) of our words concatenate *exactly* to 1 reference word) finds every remaining case where a single stray character -- always a plain space or an MWE-joining underscore, never anything more complex -- is the only thing standing between our text and Brown's real text at that point.
This generalizes and closes out the deferred remainders from several earlier issues in one sweep:
- #61's abbreviation-period-gap remainder (multi-word abbreviations like \`N._Y.\`/\`p.m.\` needing no space at all between parts, e.g. \`N.Y.\`, \`p.m..\`)
- #62's comma-flush remainder (the embedded number-range hyphens, e.g. \`607-608.\`)
- More instances of #8's own quote-gap pattern beyond what the original context-window manifest could uniquely confirm
- A wide range of previously-uncatalogued cases: ordinal suffixes (\`72nd\`), race/time notation (\`2:36h;\`), citation abbreviations (\`U.S.C.\`), parenthetical/bracket flush-ness (\`(1955).\`, \`[N]o\`), apostrophe-prefixed names (\`B'dikkat\`), and more.
**2353 confirmed instances** across 293 files, all verified via exact string-concatenation equality against the reference (not a guess -- the whole-document alignment plus an exact match requirement rules out coincidence). A further 15 cases are left out because the gap spans a sentence boundary in this corpus's own data (e.g. \`appellant".)\` ending one of our sentences right before a lone \`.\` starts the next) -- a different, more structural issue than a single stray character, worth its own look.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the whole-document alignment scan described in the issue, using difflib to compare the reconstructed word list for each document against brown-nolines.txt. Review the replace blocks where multiple reconstructed words concatenate exactly to one reference word, then verify that the reported divergences are removed while the 15 sentence-boundary cases remain separate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100