globalwordnet / globalwordnet/semcor

Math/science formula placeholders don't match Brown in learned-genre texts

Open
#16 1 comment 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 22h
Merged PRs (30d)
24

Description

Part of #5. Found by running the #12 verification tool (`semcor-verify-brown`) over the whole corpus.

Brown Corpus's `learned` genre (and some `belles_lettres` essays) contain math/science formulas that were evidently transcribed using letter-based placeholder codes in the original corpus text (special symbols the original 1961 typesetting/transcription couldn't represent directly). This corpus renders the *same* placeholders differently, so the text doesn't match even after accounting for real notational differences:

- `data/learned/br-j03.yaml`, sentence `Tojk`:
- Brown: `... probability and entropy which states that Af , where **zq is the probability ...`
- This corpus: `...obability and entropy which states that * * f, where \q is the probability that sta...`
- `data/belles_lettres/br-g20.yaml`, sentence `suze`:
- Brown: `... for N people , there are Af lines of communication between the pairs ...`
- This corpus: `...s provide that, for N people, there are * * f lines of communication between the pa...`

Brown's `Af` consistently becomes this corpus's `* * f` (**770** occurrences of the `A` → `**` substring alone), and other formula fragments turn into other ad hoc symbols (`**yc` → `|c`, `**zq` → `\q`, `**yg` → `|g`) — `data/learned/br-j03.yaml` alone accounts for the majority of these.

This is different in character from the other sub-issues here: it isn't obviously "Brown is right and this corpus is wrong" — both sides are using some kind of placeholder for characters that (as far as either plaintext rendering can tell) were never transcribed correctly in the first place. It may not be fixable from these two comparison sources at all; if anyone chases it, they'll likely need to go back to whatever the actual original source document said (or accept these formula-heavy sentences as inherently unverifiable against Brown plaintext).

### Suggested handling

At minimum, worth deciding whether the #12 verification tool should just skip/flag formula-heavy sentences (e.g. ones matching Brown's `Af`-style tokens) rather than reporting them as ordinary text divergences, since "fixing" them to literally match Brown's `Af` placeholder wouldn't actually recover the real formula either.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the semcor-verify-brown verification tool and inspect the formula-heavy examples in data/learned/br-j03.yaml and data/belles_lettres/br-g20.yaml. Determine whether these divergences should be skipped or flagged rather than treated as ordinary mismatches, and define completion as a consistent handling policy for the reported formula cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.