globalwordnet / globalwordnet/semcor

General sweep: missing space between two already-adjacent (flush) tokens

Open
#68 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 22h
Merged PRs (30d)
24

Description

Mirror image of #63/#64's general sweep: instead of finding N of our words that should collapse to 1 reference word (an extra separator to remove), look for the reverse -- 1 of our words that should split into N reference words (a separator missing entirely), using the same whole-document \`difflib\` alignment plus an exact-concatenation check.

Examples reported: \`(Kika)de\` for Brown's \`(Kika) de\`, \`23dward\` for \`23d ward\`, \`21stand\` for \`21st and\`.

**205 raw candidates** found. Of these:
- **2 are reference-side typos**, not real bugs: bley-vroman's \`brown_nolines.txt\` itself has \`to d o whatever\` and \`the s ame level\` (confirmed directly in the reference file) where the *real* word is \`do\`/\`same\` -- fixing our correctly-spelled data to match would introduce nonsense. Left alone.
- **180 confirmed and fixed**: in every one of these, our data *already* has the two (or more) words as separate, correctly-tokenized, already-flush tokens -- the bug is purely that the space between them is missing, not a tokenization problem. Applied via a new \`semcor-fix-missing-space-gap\`, a pure single-character insertion (mirror image of \`semcor-fix-genitive-gap\`'s deletion): \`lemmas\`/\`pos\`/every sense-key layer untouched, only \`text\` and the shifted \`tokens\` offsets change.
- **23 left unresolved**, deliberately: these need an actual token *split* (\`U.S.\` -> \`U.\` + \`S.\`, \`wildlife.\` -> \`wild\` + \`life.\`, ...), which raises the same kind of editorial question #43/#46 hit for token *merges* -- what lemma/pos/sense (if any) does each new half get. Left for a follow-up, individual look rather than a blanket rule.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the 23 unresolved flush-token candidates left after the semcor-fix-missing-space-gap sweep, using the whole-document difflib alignment and the reference corpus for comparison. The work is done when each selected split has an explicit editorial decision for its new halves' lemma, POS, and sense data, while the two reference-side typos remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.