globalwordnet / globalwordnet/semcor

Hyphen dropped entirely from compound modifiers (word space word instead of word-word)

Open
#43 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 22h
Merged PRs (30d)
24

Description

Part of #5. Found by `semcor-compare-brown-nolines`'s word-level diff
against `brown_nolines.txt` (see PR for that tool).

A large number of hyphenated compound modifiers in Brown's real text
(`self-acceptance`, `high-pitched`, `Yankee-hatred`, `anti-American`,
`fifty-odd`, ...) show up in this corpus with the hyphen gone entirely --
not as a spurious separate `-` token (that's #24/#38's shape), but simply
missing, leaving two independent words with an ordinary space between
them.

An exhaustive word-level diff of all 352 documents against
`brown_nolines.txt` found **808 instances across 169 files** where this
corpus has two consecutive words `A`, `B` and the reference has a single
hyphenated word `A-B` (case/spelling otherwise identical on both sides).
These split into two different shapes:

- **702 instances**: `A` and `B` are two completely ordinary, separately
tokenized/tagged words -- e.g. `data/belles_lettres/br-g01.yaml` has
`Yankee hatred` where Brown has `Yankee-hatred`; `br-g17.yaml` has
`mid twentieth` for `mid-twentieth`; `br-g14.yaml` has `fifty odd` for
`fifty-odd`. No trace of the hyphen survives anywhere in `tokens`/`pos`.
- **106 instances**: this corpus *did* recognize the compound as a single
WordNet-sensed lexeme, but joined it with an underscore instead of
preserving the real hyphen -- e.g. `data/belles_lettres/br-g12.yaml` has
`self_acceptance` (one token, sensed `self_acceptance%1:07:00::`) where
Brown has `self-acceptance`. Here the multiword-join *itself* is
correct (matches #16/#18's already-accepted underscore convention for
multiword collocations), it's specifically the choice of `_` over `-`
that loses the real source character.

This is a different, and much larger, class of loss than #24 (which
covers `NUMBER-word` compounds split into 3 tokens with the hyphen
surviving as its own spurious token) and #38's 13 word-prefixed leftovers
(same "hyphen survives as its own token" shape, just a word rather than a
number on the left) -- here the hyphen has been dropped rather than
merely mis-spaced, and the left token is a word, not a number, in nearly
all cases.

Given the scale, this likely needs the same kind of individual
context-window verification against Brown that #9/#24/#38 used before
building a fix manifest (some fraction of the 808 could be
false-positives from unrelated coincidental word pairs, though a spot
check of ~15 samples across both shapes found no false positives). Not
attempting a fix here, just recording the scan's findings.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the semcor-compare-brown-nolines word-level diff and inspect the cited YAML files, including data/belles_lettres/br-g01.yaml, br-g17.yaml, br-g14.yaml, and br-g12.yaml. Compare each suspected pair with the Brown reference in brown_nolines.txt using context windows, separating ordinary word pairs from underscore-joined compounds. Done means producing a verified fix manifest for the confirmed instances.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.