globalwordnet / globalwordnet/semcor

Reconstructed text inserts spurious whitespace around quotes, colons, and abbreviation periods

Open
#8 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 22h
Merged PRs (30d)
24

Description

Part of #5.

Comparing the corpus's `text` layer against the raw Brown plaintext ([`brown_nolines.txt`](http://www.sls.hawaii.edu/bley-vroman/brown_nolines.txt)) and token boundaries against the tagged Brown corpus ([`browntag_nolines.txt`](http://www.sls.hawaii.edu/bley-vroman/browntag_nolines.txt)), several kinds of punctuation that should sit flush against neighbouring text have a spurious space inserted around them in this corpus's reconstructed `text`.

Verified by walking each sentence's `tokens` spans and checking whether adjacent tokens are actually contiguous in `text` (i.e. whether a gap was inserted), cross-checked against `browntag_nolines.txt` to confirm the true Brown tokenization.

### Quotation marks

Brown tokenizes `` ` ` `` (opening) and `''` (closing) as their own tokens, but they sit directly against the quoted word with no space in the real text. In this corpus a space is very often inserted on the inside of the quote:

- `data/press_reportage/br-a01.yaml`: `" no evidence "` — raw Brown: `"no evidence"`

Measured across all of `data/`:
- **5,819 / 7,592** opening quotes (pos `` ` ` ``) have a spurious space after them.
- **4,649 / 5,043** closing quotes (pos `''`) have a spurious space before them.

### Numeric ratios / times with a colon

Brown tokenizes things like `11:30` or `1:512` as a single `CD` token. This corpus splits them into separate tokens with a space inserted around the colon:

- `data/press_reportage/br-a24.yaml`: `11: 30` — raw Brown: `11:30` (single `CD` token `11:30_CD`)
- `data/learned/br-j09.yaml`: `1: 512`

Measured: **189 / 189** digit-colon-digit sequences found in the corpus have a surrounding gap — i.e. this is essentially universal, not an occasional glitch.

### Periods inside abbreviations

Brown tokenizes abbreviations like `a.m.`, `i.e.`, `mm.` as single tokens (e.g. `a.m._RB`). This corpus splits them, inserting a space after the internal period:

- `data/press_reportage/br-a24.yaml`: `11: 30 a. m. yesterday` — raw Brown: `11:30 a.m. yesterday` (`a.m._RB`)
- `data/belles_lettres/br-g02.yaml`: `(i. e., societal)` — raw Brown: `i.e.` as one token
- `data/learned/br-j06.yaml`: `6 mm. i.d. Pyrex tubing`
- `data/press_editorial/br-b09.yaml`: `70 m. p. h.`

Roughly 80 such occurrences found via a simple pattern scan (`[a-z]\. [a-z]\.`); this likely undercounts abbreviations with only one internal period or capitalized forms (e.g. `U. S.`).

### Suggested fix

These all look like the same root cause: whatever detokenizer produced the `text` layer treats `` ` ` ``, `''`, and the `.`/`:` inside certain Brown single-token abbreviations/numbers as if they were always separate words needing a surrounding space, rather than consulting Brown's actual token boundaries (or a small punctuation-attachment rule set) when rejoining tokens into text.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the detokenizer that reconstructs the text layer, then compare its output with data/press_reportage/br-a01.yaml and the raw Brown files brown_nolines.txt and browntag_nolines.txt. Verify quote marks, digit-colon-digit sequences, and abbreviation periods against Brown token boundaries across data/. Done means reconstructed text no longer inserts gaps where the raw corpus has contiguous text.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.