globalwordnet / globalwordnet/semcor

Em-dash tokens corrupted to a single spaced hyphen

Open
#9 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 22h
Merged PRs (30d)
24

Description

Part of #5.

The raw Brown corpus tokenizes an em dash as a single two-character token `--` (tag `--`, e.g. `system_NN --_-- a_AT system_NN` in [`browntag_nolines.txt`](http://www.sls.hawaii.edu/bley-vroman/browntag_nolines.txt)), printed with no surrounding spaces (confirmed against [`brown_nolines.txt`](http://www.sls.hawaii.edu/bley-vroman/brown_nolines.txt)).

In this corpus, every em dash has been reduced to a **single hyphen character surrounded by spaces**:

- `data/popular_lore/br-f20.yaml`: `He carried three guns - one in the right trouser pocket`
- `data/popular_lore/br-f20.yaml`: `On Election_Day - November 4 - he energetically marshalled`

This loses information twice over: the character itself is wrong (`-` instead of `--`), and two spaces are added that don't exist in the source.

Found **2,939** single-hyphen tokens across `data/` that follow this pattern (checked by finding tokens whose surface text is exactly `-`); all of them are surrounded by inserted spaces on both sides.

### Suggested fix

Restore the double-hyphen `--` as the token surface text (matching Brown's own token), with no padding spaces, when regenerating `text` from tokens — same underlying detokenizer issue as the quote/colon/period spacing in the sibling issue.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the detokenizer or regeneration entry point that builds text from tokens, then compare its output with data/popular_lore/br-f20.yaml and the Brown references browntag_nolines.txt and brown_nolines.txt. Check that affected single-hyphen tokens become unpadded `--` tokens and verify the 2,939 affected cases are corrected without changing unrelated spacing.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.