globalwordnet / globalwordnet/semcor
Em-dash tokens corrupted to a single spaced hyphen
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 24
Description
Part of #5.
The raw Brown corpus tokenizes an em dash as a single two-character token `--` (tag `--`, e.g. `system_NN --_-- a_AT system_NN` in [`browntag_nolines.txt`](http://www.sls.hawaii.edu/bley-vroman/browntag_nolines.txt)), printed with no surrounding spaces (confirmed against [`brown_nolines.txt`](http://www.sls.hawaii.edu/bley-vroman/brown_nolines.txt)).
In this corpus, every em dash has been reduced to a **single hyphen character surrounded by spaces**:
- `data/popular_lore/br-f20.yaml`: `He carried three guns - one in the right trouser pocket`
- `data/popular_lore/br-f20.yaml`: `On Election_Day - November 4 - he energetically marshalled`
This loses information twice over: the character itself is wrong (`-` instead of `--`), and two spaces are added that don't exist in the source.
Found **2,939** single-hyphen tokens across `data/` that follow this pattern (checked by finding tokens whose surface text is exactly `-`); all of them are surrounded by inserted spaces on both sides.
### Suggested fix
Restore the double-hyphen `--` as the token surface text (matching Brown's own token), with no padding spaces, when regenerating `text` from tokens — same underlying detokenizer issue as the quote/colon/period spacing in the sibling issue.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the detokenizer or regeneration entry point that builds text from tokens, then compare its output with data/popular_lore/br-f20.yaml and the Brown references browntag_nolines.txt and brown_nolines.txt. Check that affected single-hyphen tokens become unpadded `--` tokens and verify the 2,939 affected cases are corrected without changing unrelated spacing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100