globalwordnet / globalwordnet/semcor
Literal '^' diaeresis escape leaked into 34 tokens, some split across tokens
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 24
Description
While extending #8/#5's `semcor-compare-brown-nolines` to strip
`brown_nolines.txt`'s own typesetting escapes (the '^' diaeresis marker
on e.g. `Hammarskjo^ld`, `nai^ve`), found that this corpus's own
`data/*.yaml` also has 34 leftover literal '^' characters that were never
cleaned up, in two shapes:
- Embedded in one token's surface with no whitespace on either side
(`La^utner`, `Leverku^hn`, `Du^rer`, `Tonio_Kro^ger`, 8 instances, all
in `data/belles_lettres/br-g15.yaml`) -- the token itself just needs
the stray character deleted.
- Split across multiple tokens by a spurious space around the '^'
(`Scho ^ nberg`, `nai ^ ve`, `Bo ^ o ^ k` for the double-diaeresis
`Böök`, and once fused as `^_vdingar` straight onto the next word --
26 instances). These need a token merge, the same shape as
`semcor-fix-hyphen-compound-merge`/`semcor-fix-function-word-merges`.
At most one token in any merge range carries a WordNet sense (confirmed
across all 26 merge cases -- never a real sense-on-both-sides editorial
choice like #43/#46's remainder), so this is a mechanical fix: no
sense-loss decision needed.
Contributor guide
No contributing guide indexed for this repository
Research direction
Inspect the 34 occurrences in data/*.yaml and compare their intended handling with semcor-fix-hyphen-compound-merge and semcor-fix-function-word-merges. Apply the embedded-character deletions and token merges described in the issue, preserving the one WordNet sense in each merge range. Run the semcor-compare-brown-nolines workflow to verify the corpus cleanup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100