globalwordnet / globalwordnet/semcor

Generalize capitalization fix to all single-character case mismatches vs Brown (both directions)

Open
#28 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 22h
Merged PRs (30d)
24

Description

Part of #5. Generalizes #13.

#13 fixed one narrow trigger for lost capitalization: a `"` followed by `.`/`?`/`!` immediately followed by a lowercase word (152 instances). Re-running the same whole-document alignment `semcor-verify-brown` (#12) does against `nltk.corpus.brown`, filtered to single-character case-only divergences, found the same shape of problem is much bigger and runs in **both directions**: **976** additional instances beyond what #13 already covers.

### 861 cases: this corpus has lost a capital Brown has

Same bug class as #13, just not gated behind the quote+terminal trigger:

- `data/belles_lettres/br-g21.yaml`, `Y2iy`: Brown `Despite the fact ...` vs this corpus `despite the fact...` — an ordinary sentence boundary, no quote involved.
- `Larimer St.` → `Larimer_st.` — capital lost on a proper-noun abbreviation.
- `Jim` → `jim` inside dialogue.
- Large blocks of embedded-title Title Case (`Twilight Of Southern Regionalism` → `Twilight of Southern_Regionalism`) and foreign-name particles (`De Falla` → `de Falla`, `Prix De Rome` → `Prix de Rome`) that this corpus had normalized *away* from Brown's inconsistent title-casing — #13 explicitly called these "intentional and shouldn't be fixed."
- Single-letter math/science variables in the `learned` genre (`at T` → `at t`).

### 115 cases: the reverse — this corpus adds capitalization Brown's plaintext doesn't have

- `data/belles_lettres/br-g21.yaml`, `6siA`: Brown `... the revolt of the moderates ...` (a book title mentioned in running prose, left lowercase) vs this corpus `... The Revolt Of The Moderates ...`.
- `data/fiction_general/br-k07.yaml`, `Gf74`: Brown `... the Warmongering capitalists ...` vs this corpus `... the Warmongering Capitalists ...` — rhetorical capitalization in dialogue that Brown transcribed plain.
- Several more in `br-k07.yaml`/`br-k11.yaml` of the same shape (`on the Ball`, `Wised Up`, `Home Sweet Home`, etc.).

### Decision

There's no clean rule that separates "genuine bug" from "deliberate improvement over Brown" — the title-case/foreign-name normalizations in the first group look like real improvements, and the reverse-direction group is *this corpus* adding style Brown doesn't have. Rather than trying to algorithmically preserve some categories, the decision (discussed on #13's follow-up) is **unconditional alignment with Brown in both directions, no exceptions** — for title-case-in-titles, foreign-name particles, math/science variables, institutional titles, regional adjectives, or the reverse-direction over-capitalization. This supersedes the "probably-fine pattern" note in #13.

### Suggested fix

Same manifest-driven approach as #9/#10/#11/#13: generate a fix list offline against `nltk.corpus.brown` (file, sentence, character offset, target character), apply it mechanically at runtime with no NLTK dependency. Every one of these is a single-character, same-length swap — no token/offset restructuring needed, only `text` (and, where a `lemmas` entry mirrors the same surface case, `lemmas`) changes.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the semcor-verify-brown whole-document alignment against nltk.corpus.brown and review the manifest-driven approaches in #9, #10, #11, and #13. Done means an offline fix list covers the specified single-character mismatches in both directions, while runtime applies only text and matching lemmas changes without an NLTK dependency.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.