globalwordnet / globalwordnet/semcor
Hyphenated compound tokens (80-hp, anti-A) over-split into 3 tokens with spurious spaces
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 24
Description
Part of #5. Found while fixing #9 (the em-dash issue): cross-checking all 2,939 "single hyphen surrounded by spaces" tokens against `nltk.corpus.brown` split them into three groups, not one -- 1,868 genuine em dashes and 17 number-range hyphens, both fixed in #9's PR, and a third group this issue is about.
Brown tokenizes some hyphenated compounds -- a number plus a unit (`80-hp`, `42-inch`), a prefix plus a word (`anti-A`, `anti-B`), a compact number range (confirmed as its own separate case in some contexts, see #9) -- as a **single token**. This corpus has split at least some of these into three separate tokens (`80`, `-`, `hp`) with spurious spaces added around the `-`, the same surface symptom as #9 but a different underlying bug: fixing it means *merging tokens back together*, not just editing whitespace/characters.
Confirmed examples (Brown vs. this corpus):
- `data/skill_and_hobbies/br-e16.yaml`, sentence `v2IR`: Brown has the single token `80-hp` (`['Merc', '800', '.', 'The', '80-hp', 'motor', ...]`); this corpus has three tokens `The 80 - hp motor`.
- `data/learned/br-j09.yaml`, sentences `Tfea`/`6FW3`/`NEbQ`: Brown has `anti-A` and `anti-B` as single tokens; this corpus has `anti - A` / `anti - B`.
### Scope
Of the ~1,054 hyphen instances #9's verification pass couldn't confirm as either an em dash or a number-range hyphen, at least 498 have a digit immediately before the hyphen (`5 - foot`, `42 - inch`, `1 - hp`, `1769 - 1842`, ...) -- a conservative lower bound for this pattern, since it doesn't catch the word-prefix cases (`anti - A`) or anything where the digit isn't the immediately preceding token. The remaining ~556 are a mix of this same compound-merge pattern with a non-numeric prefix, and genuine em dashes that #9's verification simply couldn't confirm (usually because a neighboring word is itself multiword-joined, e.g. `social_welfare`, breaking the exact-match search against Brown's separately-tokenized words) -- these two haven't been cleanly separated from each other yet.
### Suggested approach
This needs actual token merging (combining 3 SemCor tokens into 1, adjusting `pos`/`lemmas`/`tokens` together, and shifting any `oewn_key`/`wn16_key`/`wn30_key` indices at or after the merge point), not a whitespace-only edit like #8/#9/#13 -- a meaningfully different shape of fix, hence its own issue. Whoever picks this up will likely want to redo the Brown cross-check with a proper alignment tool (per #12) rather than the ad hoc context-window search used to find these, since that's also what's needed to cleanly separate the compound-merge cases from the unconfirmed em dashes above.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with data/skill_and_hobbies/br-e16.yaml sentence v2IR and data/learned/br-j09.yaml sentences Tfea, 6FW3, and NEbQ, then inspect the existing #9 verification and token metadata handling. Redo the Brown cross-check with an alignment tool, separate compound merges from unconfirmed em dashes, and verify that tokens, pos, lemmas, and all affected WordNet indices remain aligned.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100