globalwordnet / globalwordnet/semcor

Leftover literal '&' characters in text/lemmas

Open
#11 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 22h
Merged PRs (30d)
24

Description

Part of #5.

20 files in `data/` contain literal `&` characters in their `text`/`lemmas` layers, e.g.:

- `data/fiction_adventure/br-n01.yaml`: `"...if we were sleeping together & & & "` (three separate `&` tokens)
- `data/fiction_adventure/br-n09.yaml`: `Nothing unless & & &.`
- `data/fiction_adventure/br-n08.yaml`: `every Spring & & & or as ancient fertility cults demand...`

Full list of affected files:

```
data/fiction_adventure/br-n01.yaml
data/fiction_adventure/br-n08.yaml
data/fiction_adventure/br-n09.yaml
data/fiction_adventure/br-n10.yaml
data/fiction_adventure/br-n15.yaml
data/fiction_adventure/br-n16.yaml
data/fiction_adventure/br-n17.yaml
data/fiction_general/br-k24.yaml
data/fiction_mystery/br-l01.yaml
data/fiction_mystery/br-l07.yaml
data/fiction_mystery/br-l08.yaml
data/fiction_mystery/br-l10.yaml
data/fiction_romance/br-p02.yaml
data/fiction_romance/br-p24.yaml
data/fiction_science/br-m01.yaml
data/fiction_science/br-m02.yaml
data/learned/br-j25.yaml
data/press_editorial/br-b08.yaml
data/press_editorial/br-b27.yaml
data/press_reviews/br-c10.yaml
```

The pattern is consistently three repeated `&` tokens standing in for what looks like an interrupted/trailing-off line of dialogue (an em-dash-style or ellipsis-style device). This is almost certainly a leftover from whatever intermediate format used `&` to mark a non-sentence-final period (the same convention `brown_nolines.txt` uses, e.g. `Mr&` for `Mr.`) — that conversion evidently ran correctly for ordinary abbreviations elsewhere in the corpus (`Ave&` → `Ave.`, `a&m&` → `a. m.`) but never got applied to these triple-`&` runs, leaving raw markup characters in the data instead of real punctuation.

Note: unlike the other sub-issues here, we couldn't confirm the intended original character directly against `browntag_nolines.txt` — the one instance checked (`br-n01`, "sleeping together...") has no corresponding token in the tagged Brown file at all at that position, so the ground truth for what these three tokens should render as needs to come from the original SemCor/Brown source rather than these two comparison files.

### Suggested fix

Track down what these `&` runs represent in whatever upstream source this corpus was built from, and either normalize them to the correct punctuation or, if they're a genuine editorial elision mark, document it explicitly instead of leaving raw `&` characters in `text`/`lemmas`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the listed files under data/, comparing their text and lemmas layers with brown_nolines.txt and browntag_nolines.txt. Trace the affected passages to the original SemCor/Brown source to determine what the repeated \u0026 tokens represent. Done means all 20 files use the correct punctuation or have the editorial elision explicitly documented, with no raw markup left.

Written by the indexing model from the issue text.

Assessment

Tech stack
yaml
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.