globalwordnet / globalwordnet/semcor

General sweep: remaining spurious-gap divergences fixable by deleting one character

Open
#63 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 22h
Merged PRs (30d)
24

Description

A general, whole-document-alignment scan (align this corpus's reconstructed word list against \`brown-nolines.txt\` per document with \`difflib\`, then look for any \`replace\` block where N (>1) of our words concatenate *exactly* to 1 reference word) finds every remaining case where a single stray character -- always a plain space or an MWE-joining underscore, never anything more complex -- is the only thing standing between our text and Brown's real text at that point.

This generalizes and closes out the deferred remainders from several earlier issues in one sweep:
- #61's abbreviation-period-gap remainder (multi-word abbreviations like \`N._Y.\`/\`p.m.\` needing no space at all between parts, e.g. \`N.Y.\`, \`p.m..\`)
- #62's comma-flush remainder (the embedded number-range hyphens, e.g. \`607-608.\`)
- More instances of #8's own quote-gap pattern beyond what the original context-window manifest could uniquely confirm
- A wide range of previously-uncatalogued cases: ordinal suffixes (\`72nd\`), race/time notation (\`2:36h;\`), citation abbreviations (\`U.S.C.\`), parenthetical/bracket flush-ness (\`(1955).\`, \`[N]o\`), apostrophe-prefixed names (\`B'dikkat\`), and more.

**2353 confirmed instances** across 293 files, all verified via exact string-concatenation equality against the reference (not a guess -- the whole-document alignment plus an exact match requirement rules out coincidence). A further 15 cases are left out because the gap spans a sentence boundary in this corpus's own data (e.g. \`appellant".)\` ending one of our sentences right before a lone \`.\` starts the next) -- a different, more structural issue than a single stray character, worth its own look.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the whole-document alignment scan described in the issue, using difflib to compare the reconstructed word list for each document against brown-nolines.txt. Review the replace blocks where multiple reconstructed words concatenate exactly to one reference word, then verify that the reported divergences are removed while the 15 sentence-boundary cases remain separate.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.