globalwordnet / globalwordnet/semcor
Generalize capitalization fix to all single-character case mismatches vs Brown (both directions)
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 24
Description
Part of #5. Generalizes #13.
#13 fixed one narrow trigger for lost capitalization: a `"` followed by `.`/`?`/`!` immediately followed by a lowercase word (152 instances). Re-running the same whole-document alignment `semcor-verify-brown` (#12) does against `nltk.corpus.brown`, filtered to single-character case-only divergences, found the same shape of problem is much bigger and runs in **both directions**: **976** additional instances beyond what #13 already covers.
### 861 cases: this corpus has lost a capital Brown has
Same bug class as #13, just not gated behind the quote+terminal trigger:
- `data/belles_lettres/br-g21.yaml`, `Y2iy`: Brown `Despite the fact ...` vs this corpus `despite the fact...` — an ordinary sentence boundary, no quote involved.
- `Larimer St.` → `Larimer_st.` — capital lost on a proper-noun abbreviation.
- `Jim` → `jim` inside dialogue.
- Large blocks of embedded-title Title Case (`Twilight Of Southern Regionalism` → `Twilight of Southern_Regionalism`) and foreign-name particles (`De Falla` → `de Falla`, `Prix De Rome` → `Prix de Rome`) that this corpus had normalized *away* from Brown's inconsistent title-casing — #13 explicitly called these "intentional and shouldn't be fixed."
- Single-letter math/science variables in the `learned` genre (`at T` → `at t`).
### 115 cases: the reverse — this corpus adds capitalization Brown's plaintext doesn't have
- `data/belles_lettres/br-g21.yaml`, `6siA`: Brown `... the revolt of the moderates ...` (a book title mentioned in running prose, left lowercase) vs this corpus `... The Revolt Of The Moderates ...`.
- `data/fiction_general/br-k07.yaml`, `Gf74`: Brown `... the Warmongering capitalists ...` vs this corpus `... the Warmongering Capitalists ...` — rhetorical capitalization in dialogue that Brown transcribed plain.
- Several more in `br-k07.yaml`/`br-k11.yaml` of the same shape (`on the Ball`, `Wised Up`, `Home Sweet Home`, etc.).
### Decision
There's no clean rule that separates "genuine bug" from "deliberate improvement over Brown" — the title-case/foreign-name normalizations in the first group look like real improvements, and the reverse-direction group is *this corpus* adding style Brown doesn't have. Rather than trying to algorithmically preserve some categories, the decision (discussed on #13's follow-up) is **unconditional alignment with Brown in both directions, no exceptions** — for title-case-in-titles, foreign-name particles, math/science variables, institutional titles, regional adjectives, or the reverse-direction over-capitalization. This supersedes the "probably-fine pattern" note in #13.
### Suggested fix
Same manifest-driven approach as #9/#10/#11/#13: generate a fix list offline against `nltk.corpus.brown` (file, sentence, character offset, target character), apply it mechanically at runtime with no NLTK dependency. Every one of these is a single-character, same-length swap — no token/offset restructuring needed, only `text` (and, where a `lemmas` entry mirrors the same surface case, `lemmas`) changes.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the semcor-verify-brown whole-document alignment against nltk.corpus.brown and review the manifest-driven approaches in #9, #10, #11, and #13. Done means an offline fix list covers the specified single-character mismatches in both directions, while runtime applies only text and matching lemmas changes without an NLTK dependency.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100