globalwordnet / globalwordnet/semcor

Underscore/hash-wrapped section headers dropped entirely (recoverable from brown_nolines.txt)

Open
#51 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 22h
Merged PRs (30d)
24

Description

Part of #5. Found while investigating what to fix after #42.

`brown_nolines.txt` wraps short structural headings in markup characters
that this corpus's own text drops entirely, while the surrounding prose
matches fine on both sides -- this isn't a #32-style permanent loss,
it's recoverable directly from `brown_nolines.txt`, no NLTK needed.

Two wrapper conventions, same underlying phenomenon:

### `_..._` -- legal/academic section markers

Legal-document subsection labels and academic-paper section headings,
e.g.:

- `data/miscellaneous/br-h12.yaml`: brown_nolines.txt has `...covered
into the Treasury to the credit of miscellaneous receipts. _(G)_ The
Attorney General shall assign...` -- this corpus has nothing between
`receipts.` and `The Attorney General`.
- `data/learned/br-j09.yaml`: brown_nolines.txt has `...#MATERIALS AND
METHODS# _SAMPLES._ Serum samples were obtained...` -- this corpus's
`text` has `Serum samples were obtained...` with the `_SAMPLES._`
heading (and the `#MATERIALS AND METHODS#` one right before it --
see below) missing entirely, even though the ordinary word `samples`
appears normally two words later in the real sentence.

A quick scan (word-per-line diff against `brown_nolines.txt`, same
method `semcor-compare-brown-nolines` uses, filtered to reference words
starting and ending with `_`) finds at least 308 such missing
word-instances across 51+ files, concentrated in `miscellaneous`
(government/legal documents) and `learned` (academic papers), with a
handful in `press_editorial`/`popular_lore`/`religion`/`skill_and_hobbies`.
This is a lower bound from a quick single-token check -- multi-word
underscore-wrapped spans would need the same kind of context-aware
scan #16/#43 used to get an exhaustive count.

### `#...#` -- this is #32's own dateline/subheadline content

`grep -o "#[A-Z][A-Z ]*#" src/semcor/brown-nolines.txt` turns up
`#MERGER PROPOSED#`, `#WARDS PROTECTED#`, `#ASK JAIL DEPUTIES#` -- #32's
own flagship examples. **#32 concluded `brown_nolines.txt` was missing
this content too** ("grep for \`Ask jail deputies\` finds nothing
there either"), but that grep was case-sensitive and missed Brown's
own `#ALL-CAPS#` heading convention -- the content is actually there,
recoverable, same as the `_..._` case above. Commenting on #32
separately with this correction.

### Suggested handling

Same manifest-driven approach as #32's own extraction (PR #49), but
sourced from `brown_nolines.txt` directly instead of
`nltk.corpus.brown` -- no NLTK dependency needed here, since the
content is confirmed present in the reference this repo already trusts
(#16/#18/#19). Whether this becomes new sentences merged into the
corpus (needs sense-tagging, a content-expansion project like #32) or
something smaller depends on how these headings should be represented
-- worth a maintainer decision before implementing.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with src/semcor/brown-nolines.txt and the affected data/miscellaneous and data/learned YAML files. Review the semcor-compare-brown-nolines word-per-line comparison and the manifest-driven extraction in PR #49, then check how #32 handles related headings. Done requires a maintainer decision on heading representation and an agreed scope for recovering the missing content.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
content
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.