globalwordnet / globalwordnet/semcor

Literal '&' corrupted to '+' in some text

Open
#17 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 22h
Merged PRs (30d)
24

Description

Part of #5. Found by running the #12 verification tool (`semcor-verify-brown`) over the whole corpus.

At least one literal `&` character from the original text has been corrupted into a `+`:

- `data/belles_lettres/br-g17.yaml`, sentence `xN1w`:
- Brown: `... these institutions having previously been A & M colleges ? ...`
- This corpus: `...ns having previously been A_+_M_colleges?...`
- Same file, sentence `a/hf`:
- Brown: `... supposed , perchance , that A & M ( vocational training , that ...`
- This corpus: `...eges?Was it supposed, perchance, that A_+_M (vocational_training, that_is) was qu...`

That's Texas A&M — `A & M` should stay `A & M` (or at least `A&M`), not become `A + M`. **75** occurrences of a bare `&` → `+` substitution total.

This is the mirror image of #11 (leftover un-normalized `&` characters): there, the abbreviation-period `&` markup convention wasn't cleaned up in some places; here, a genuine literal `&` character in the source got mangled into the wrong character entirely, presumably by the same kind of markup-normalization step misfiring in the opposite direction. Worth investigating together with #11 since they likely share a root cause in whatever step processes `&`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with data/belles_lettres/br-g17.yaml and run the semcor-verify-brown tool, focusing on sentences xN1w and a/hf. Trace the markup-normalization step that handles literal \u0026 characters, and compare it with issue #11. Done means the 75 bare substitutions are corrected without introducing the opposite normalization problem, with verification passing across the corpus.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.