globalwordnet / globalwordnet/semcor
Hyphen dropped entirely from compound modifiers (word space word instead of word-word)
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 24
Description
Part of #5. Found by `semcor-compare-brown-nolines`'s word-level diff
against `brown_nolines.txt` (see PR for that tool).
A large number of hyphenated compound modifiers in Brown's real text
(`self-acceptance`, `high-pitched`, `Yankee-hatred`, `anti-American`,
`fifty-odd`, ...) show up in this corpus with the hyphen gone entirely --
not as a spurious separate `-` token (that's #24/#38's shape), but simply
missing, leaving two independent words with an ordinary space between
them.
An exhaustive word-level diff of all 352 documents against
`brown_nolines.txt` found **808 instances across 169 files** where this
corpus has two consecutive words `A`, `B` and the reference has a single
hyphenated word `A-B` (case/spelling otherwise identical on both sides).
These split into two different shapes:
- **702 instances**: `A` and `B` are two completely ordinary, separately
tokenized/tagged words -- e.g. `data/belles_lettres/br-g01.yaml` has
`Yankee hatred` where Brown has `Yankee-hatred`; `br-g17.yaml` has
`mid twentieth` for `mid-twentieth`; `br-g14.yaml` has `fifty odd` for
`fifty-odd`. No trace of the hyphen survives anywhere in `tokens`/`pos`.
- **106 instances**: this corpus *did* recognize the compound as a single
WordNet-sensed lexeme, but joined it with an underscore instead of
preserving the real hyphen -- e.g. `data/belles_lettres/br-g12.yaml` has
`self_acceptance` (one token, sensed `self_acceptance%1:07:00::`) where
Brown has `self-acceptance`. Here the multiword-join *itself* is
correct (matches #16/#18's already-accepted underscore convention for
multiword collocations), it's specifically the choice of `_` over `-`
that loses the real source character.
This is a different, and much larger, class of loss than #24 (which
covers `NUMBER-word` compounds split into 3 tokens with the hyphen
surviving as its own spurious token) and #38's 13 word-prefixed leftovers
(same "hyphen survives as its own token" shape, just a word rather than a
number on the left) -- here the hyphen has been dropped rather than
merely mis-spaced, and the left token is a word, not a number, in nearly
all cases.
Given the scale, this likely needs the same kind of individual
context-window verification against Brown that #9/#24/#38 used before
building a fix manifest (some fraction of the 808 could be
false-positives from unrelated coincidental word pairs, though a spot
check of ~15 samples across both shapes found no false positives). Not
attempting a fix here, just recording the scan's findings.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the semcor-compare-brown-nolines word-level diff and inspect the cited YAML files, including data/belles_lettres/br-g01.yaml, br-g17.yaml, br-g14.yaml, and br-g12.yaml. Compare each suspected pair with the Brown reference in brown_nolines.txt using context windows, separating ordinary word pairs from underscore-joined compounds. Done means producing a verified fix manifest for the confirmed instances.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100