globalwordnet / globalwordnet/semcor

Leftover hyphen-related cases from #24: word-prefix merges, more em dashes, **h-as-hyphen

Open
#38 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 22h
Merged PRs (30d)
24

Description

Part of #5. Split off from #24's fix (PR TBD) -- the same context-window
investigation that found 315 confirmed hyphenated-compound merges (289
fixed there, restricted to a purely numeric left token) also turned up
several other things, not fixed there, tracked here.

### 13 word-prefixed compound merges (needs individual sense review)

Confirmed the same "3 tokens should be 1" shape, but the left token is
a word, not a number: `AFL-CIO` (×2), `radio-TV` (×2), `NBC-TV`,
`Class-D`, `anti-Rh`, `Bar-H`, `mid-1958`, `U.S.-Soviet`,
`U.N.-chartered`, `U-I`, and `O-B` (this last one, from the C&O/B&O
railroad article #17 also touched, may be a false positive from a
messy multi-hyphen sentence and deserves a closer look before trusting
it).

Unlike a number, a word carries a real, specific WordNet sense that a
merge can't safely resolve automatically: `Class-D`'s left token is
sensed `class%1:14:05::`, `NBC-TV`'s `group%1:03:00::`, `radio-TV`'s
`radio%1:10:00::` -- all real content senses, not boilerplate, so
merging needs someone to decide per-instance which sense (if either)
the merged compound should carry, or whether to drop both.

### More confirmed em dashes / number-range hyphens beyond #9

The same context-window search that classified the compound-merge cases
also confirmed a good number of `-` tokens that #9's narrower
neighbour-only search couldn't (`Gap` contains `--` or a bare `-`, e.g.
`"social welfare -- which"`, `"by design -- and"` -- neighbours like
`social_welfare` being multiword-joined broke #9's exact-match search).
Same fix mechanism as `fix_em_dash.py`, just needs its own manifest built
from this session's better search.

### `**h` placeholder rendered as a lone `-` instead of `* * h`

A handful of these `-`-shaped cases turned out to be neither dash nor
merge: `nltk.corpus.brown` drops the original `**h` marker character
entirely at these spots (e.g. `black statuary - six figures` -- Brown
has literally nothing between `statuary` and `six`), but
`sls.hawaii.edu`'s raw dump confirms the real source has `**h` there --
the exact marker #16 already restored elsewhere in the corpus, just
landing as a lone `-` in this handful of spots instead of the `* * h`
shape #16's fix searched for. Needs its own small investigation to find
the full set.

### Unresolved / ambiguous remainder

Of the corpus's 1,002 lone-`-`-surrounded-by-spaces tokens, ~344
couldn't be placed in Brown at all with this session's context-window
search (too little/no surrounding context, or the context itself
differs for unrelated reasons) and 42 matched more than one place in
their file. Left as a known remainder for whoever picks any of the
above up next.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with fix_em_dash.py and the context-window investigation described in the issue. Compare nltk.corpus.brown against the sls.hawaii.edu raw dump, then review the listed word-prefixed cases and build manifests for confirmed dash and **h corrections. Done means the full affected set is identified, ambiguous senses are explicitly resolved, and the corpus corrections are covered by reproducible checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.