globalwordnet / globalwordnet/semcor

Sentence-initial capitalization lost after dialogue-closing quote + ?/!

Open
#13 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 22h
Merged PRs (30d)
24

Description

Part of #5. Found by running the #12 verification tool (`semcor-verify-brown`) over the whole corpus.

When a sentence begins immediately after a closing quote followed by `?` or `!`, its first word's capital letter is lost in this corpus's `text`.

Examples (Brown vs. this corpus, via `nltk.corpus.brown`):

- `data/belles_lettres/br-g04.yaml`, sentence `Blp8`:
- Brown: `... mate '' ? The Australian asked . `` They swear ...`
- This corpus: `..." That smell is something, eh, mate "? the Australian asked." They swear...`
- (`The` → `the`, and note the missing space after `asked.` too — likely the same detokenizer issue as #8, but the lowercasing is a separate problem)
- `data/belles_lettres/br-g04.yaml`, sentence `QUNu`:
- Brown: `... tabac '' ? He said sharply . Idje still stared ...`
- This corpus: `..."Here, Idje, you fella like tabac "? he said sharply.Idje still stared...`
- (`He` → `he`)
- `data/belles_lettres/br-g08.yaml`, sentence `4Z8F` *(genre: same pattern, different file)*:
- Brown: `... his `` English Notebooks '' . And one finds it again in Thomas ...`
- This corpus: `...records it in his " English_Notebooks ".and one finds it again in Thomas_Nelson_P...`
- (`And` → `and`)

### Note on a related but probably-fine pattern in the same data

The verification tool's aggregate pattern counts (single capital letter → the same letter lowercased) also catch a lot of instances that look like a *deliberate and reasonable* normalization, not a bug: Brown renders embedded titles in Title Case (`Twilight Of Southern Regionalism`, `Waiting For Godot`, `Advice To My Country`) and this corpus correctly lowercases the minor words (`Twilight of Southern_Regionalism`, `Waiting_for_Godot`, `Advice to My Country`), and likewise lowercases the preposition in foreign names (`De Falla` → `de Falla`, `Prix De Rome` → `Prix de Rome`). Those look intentional and shouldn't be "fixed" to match Brown's all-caps-style titling. Whoever picks this up should only be chasing the genuine sentence-initial case above, not this one.

### Suggested fix

Look at whatever step decides where a new sentence starts and whether to capitalize its first letter — it's evidently keying off the wrong signal when the previous token is a closing quote immediately followed by `?`/`!`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the semcor-verify-brown verification tool over the corpus and trace the step that determines sentence starts and capitalization. Inspect the examples in data/belles_lettres/br-g04.yaml and br-g08.yaml, then verify that sentence-initial words after a closing quote followed by ? or ! retain capitalization without changing intentional title normalization.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.