globalwordnet / globalwordnet/semcor
Reconstructed text inserts spurious whitespace around quotes, colons, and abbreviation periods
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 24
Description
Part of #5.
Comparing the corpus's `text` layer against the raw Brown plaintext ([`brown_nolines.txt`](http://www.sls.hawaii.edu/bley-vroman/brown_nolines.txt)) and token boundaries against the tagged Brown corpus ([`browntag_nolines.txt`](http://www.sls.hawaii.edu/bley-vroman/browntag_nolines.txt)), several kinds of punctuation that should sit flush against neighbouring text have a spurious space inserted around them in this corpus's reconstructed `text`.
Verified by walking each sentence's `tokens` spans and checking whether adjacent tokens are actually contiguous in `text` (i.e. whether a gap was inserted), cross-checked against `browntag_nolines.txt` to confirm the true Brown tokenization.
### Quotation marks
Brown tokenizes `` ` ` `` (opening) and `''` (closing) as their own tokens, but they sit directly against the quoted word with no space in the real text. In this corpus a space is very often inserted on the inside of the quote:
- `data/press_reportage/br-a01.yaml`: `" no evidence "` — raw Brown: `"no evidence"`
Measured across all of `data/`:
- **5,819 / 7,592** opening quotes (pos `` ` ` ``) have a spurious space after them.
- **4,649 / 5,043** closing quotes (pos `''`) have a spurious space before them.
### Numeric ratios / times with a colon
Brown tokenizes things like `11:30` or `1:512` as a single `CD` token. This corpus splits them into separate tokens with a space inserted around the colon:
- `data/press_reportage/br-a24.yaml`: `11: 30` — raw Brown: `11:30` (single `CD` token `11:30_CD`)
- `data/learned/br-j09.yaml`: `1: 512`
Measured: **189 / 189** digit-colon-digit sequences found in the corpus have a surrounding gap — i.e. this is essentially universal, not an occasional glitch.
### Periods inside abbreviations
Brown tokenizes abbreviations like `a.m.`, `i.e.`, `mm.` as single tokens (e.g. `a.m._RB`). This corpus splits them, inserting a space after the internal period:
- `data/press_reportage/br-a24.yaml`: `11: 30 a. m. yesterday` — raw Brown: `11:30 a.m. yesterday` (`a.m._RB`)
- `data/belles_lettres/br-g02.yaml`: `(i. e., societal)` — raw Brown: `i.e.` as one token
- `data/learned/br-j06.yaml`: `6 mm. i.d. Pyrex tubing`
- `data/press_editorial/br-b09.yaml`: `70 m. p. h.`
Roughly 80 such occurrences found via a simple pattern scan (`[a-z]\. [a-z]\.`); this likely undercounts abbreviations with only one internal period or capitalized forms (e.g. `U. S.`).
### Suggested fix
These all look like the same root cause: whatever detokenizer produced the `text` layer treats `` ` ` ``, `''`, and the `.`/`:` inside certain Brown single-token abbreviations/numbers as if they were always separate words needing a surrounding space, rather than consulting Brown's actual token boundaries (or a small punctuation-attachment rule set) when rejoining tokens into text.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the detokenizer that reconstructs the text layer, then compare its output with data/press_reportage/br-a01.yaml and the raw Brown files brown_nolines.txt and browntag_nolines.txt. Verify quote marks, digit-colon-digit sequences, and abbreviation periods against Brown token boundaries across data/. Done means reconstructed text no longer inserts gaps where the raw corpus has contiguous text.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100