globalwordnet / globalwordnet/semcor
Literal '&' corrupted to '+' in some text
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 24
Description
Part of #5. Found by running the #12 verification tool (`semcor-verify-brown`) over the whole corpus.
At least one literal `&` character from the original text has been corrupted into a `+`:
- `data/belles_lettres/br-g17.yaml`, sentence `xN1w`:
- Brown: `... these institutions having previously been A & M colleges ? ...`
- This corpus: `...ns having previously been A_+_M_colleges?...`
- Same file, sentence `a/hf`:
- Brown: `... supposed , perchance , that A & M ( vocational training , that ...`
- This corpus: `...eges?Was it supposed, perchance, that A_+_M (vocational_training, that_is) was qu...`
That's Texas A&M — `A & M` should stay `A & M` (or at least `A&M`), not become `A + M`. **75** occurrences of a bare `&` → `+` substitution total.
This is the mirror image of #11 (leftover un-normalized `&` characters): there, the abbreviation-period `&` markup convention wasn't cleaned up in some places; here, a genuine literal `&` character in the source got mangled into the wrong character entirely, presumably by the same kind of markup-normalization step misfiring in the opposite direction. Worth investigating together with #11 since they likely share a root cause in whatever step processes `&`.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with data/belles_lettres/br-g17.yaml and run the semcor-verify-brown tool, focusing on sentences xN1w and a/hf. Trace the markup-normalization step that handles literal \u0026 characters, and compare it with issue #11. Done means the 75 bare substitutions are corrected without introducing the opposite normalization problem, with verification passing across the corpus.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100