globalwordnet / globalwordnet/semcor

Literal '^' diaeresis escape leaked into 34 tokens, some split across tokens

Open
#60 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 22h
Merged PRs (30d)
24

Description

While extending #8/#5's `semcor-compare-brown-nolines` to strip
`brown_nolines.txt`'s own typesetting escapes (the '^' diaeresis marker
on e.g. `Hammarskjo^ld`, `nai^ve`), found that this corpus's own
`data/*.yaml` also has 34 leftover literal '^' characters that were never
cleaned up, in two shapes:

- Embedded in one token's surface with no whitespace on either side
(`La^utner`, `Leverku^hn`, `Du^rer`, `Tonio_Kro^ger`, 8 instances, all
in `data/belles_lettres/br-g15.yaml`) -- the token itself just needs
the stray character deleted.
- Split across multiple tokens by a spurious space around the '^'
(`Scho ^ nberg`, `nai ^ ve`, `Bo ^ o ^ k` for the double-diaeresis
`Böök`, and once fused as `^_vdingar` straight onto the next word --
26 instances). These need a token merge, the same shape as
`semcor-fix-hyphen-compound-merge`/`semcor-fix-function-word-merges`.

At most one token in any merge range carries a WordNet sense (confirmed
across all 26 merge cases -- never a real sense-on-both-sides editorial
choice like #43/#46's remainder), so this is a mechanical fix: no
sense-loss decision needed.

Contributor guide

No contributing guide indexed for this repository

Research direction

Inspect the 34 occurrences in data/*.yaml and compare their intended handling with semcor-fix-hyphen-compound-merge and semcor-fix-function-word-merges. Apply the embedded-character deletions and token merges described in the issue, preserving the one WordNet sense in each merge range. Run the semcor-compare-brown-nolines workflow to verify the corpus cleanup.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.