globalwordnet / globalwordnet/semcor
Math/science formula placeholders don't match Brown in learned-genre texts
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 24
Description
Part of #5. Found by running the #12 verification tool (`semcor-verify-brown`) over the whole corpus.
Brown Corpus's `learned` genre (and some `belles_lettres` essays) contain math/science formulas that were evidently transcribed using letter-based placeholder codes in the original corpus text (special symbols the original 1961 typesetting/transcription couldn't represent directly). This corpus renders the *same* placeholders differently, so the text doesn't match even after accounting for real notational differences:
- `data/learned/br-j03.yaml`, sentence `Tojk`:
- Brown: `... probability and entropy which states that Af , where **zq is the probability ...`
- This corpus: `...obability and entropy which states that * * f, where \q is the probability that sta...`
- `data/belles_lettres/br-g20.yaml`, sentence `suze`:
- Brown: `... for N people , there are Af lines of communication between the pairs ...`
- This corpus: `...s provide that, for N people, there are * * f lines of communication between the pa...`
Brown's `Af` consistently becomes this corpus's `* * f` (**770** occurrences of the `A` → `**` substring alone), and other formula fragments turn into other ad hoc symbols (`**yc` → `|c`, `**zq` → `\q`, `**yg` → `|g`) — `data/learned/br-j03.yaml` alone accounts for the majority of these.
This is different in character from the other sub-issues here: it isn't obviously "Brown is right and this corpus is wrong" — both sides are using some kind of placeholder for characters that (as far as either plaintext rendering can tell) were never transcribed correctly in the first place. It may not be fixable from these two comparison sources at all; if anyone chases it, they'll likely need to go back to whatever the actual original source document said (or accept these formula-heavy sentences as inherently unverifiable against Brown plaintext).
### Suggested handling
At minimum, worth deciding whether the #12 verification tool should just skip/flag formula-heavy sentences (e.g. ones matching Brown's `Af`-style tokens) rather than reporting them as ordinary text divergences, since "fixing" them to literally match Brown's `Af` placeholder wouldn't actually recover the real formula either.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the semcor-verify-brown verification tool and inspect the formula-heavy examples in data/learned/br-j03.yaml and data/belles_lettres/br-g20.yaml. Determine whether these divergences should be skipped or flagged rather than treated as ordinary mismatches, and define completion as a consistent handling policy for the reported formula cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100