globalwordnet / globalwordnet/semcor
Underscore/hash-wrapped section headers dropped entirely (recoverable from brown_nolines.txt)
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 24
Description
Part of #5. Found while investigating what to fix after #42.
`brown_nolines.txt` wraps short structural headings in markup characters
that this corpus's own text drops entirely, while the surrounding prose
matches fine on both sides -- this isn't a #32-style permanent loss,
it's recoverable directly from `brown_nolines.txt`, no NLTK needed.
Two wrapper conventions, same underlying phenomenon:
### `_..._` -- legal/academic section markers
Legal-document subsection labels and academic-paper section headings,
e.g.:
- `data/miscellaneous/br-h12.yaml`: brown_nolines.txt has `...covered
into the Treasury to the credit of miscellaneous receipts. _(G)_ The
Attorney General shall assign...` -- this corpus has nothing between
`receipts.` and `The Attorney General`.
- `data/learned/br-j09.yaml`: brown_nolines.txt has `...#MATERIALS AND
METHODS# _SAMPLES._ Serum samples were obtained...` -- this corpus's
`text` has `Serum samples were obtained...` with the `_SAMPLES._`
heading (and the `#MATERIALS AND METHODS#` one right before it --
see below) missing entirely, even though the ordinary word `samples`
appears normally two words later in the real sentence.
A quick scan (word-per-line diff against `brown_nolines.txt`, same
method `semcor-compare-brown-nolines` uses, filtered to reference words
starting and ending with `_`) finds at least 308 such missing
word-instances across 51+ files, concentrated in `miscellaneous`
(government/legal documents) and `learned` (academic papers), with a
handful in `press_editorial`/`popular_lore`/`religion`/`skill_and_hobbies`.
This is a lower bound from a quick single-token check -- multi-word
underscore-wrapped spans would need the same kind of context-aware
scan #16/#43 used to get an exhaustive count.
### `#...#` -- this is #32's own dateline/subheadline content
`grep -o "#[A-Z][A-Z ]*#" src/semcor/brown-nolines.txt` turns up
`#MERGER PROPOSED#`, `#WARDS PROTECTED#`, `#ASK JAIL DEPUTIES#` -- #32's
own flagship examples. **#32 concluded `brown_nolines.txt` was missing
this content too** ("grep for \`Ask jail deputies\` finds nothing
there either"), but that grep was case-sensitive and missed Brown's
own `#ALL-CAPS#` heading convention -- the content is actually there,
recoverable, same as the `_..._` case above. Commenting on #32
separately with this correction.
### Suggested handling
Same manifest-driven approach as #32's own extraction (PR #49), but
sourced from `brown_nolines.txt` directly instead of
`nltk.corpus.brown` -- no NLTK dependency needed here, since the
content is confirmed present in the reference this repo already trusts
(#16/#18/#19). Whether this becomes new sentences merged into the
corpus (needs sense-tagging, a content-expansion project like #32) or
something smaller depends on how these headings should be represented
-- worth a maintainer decision before implementing.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with src/semcor/brown-nolines.txt and the affected data/miscellaneous and data/learned YAML files. Review the semcor-compare-brown-nolines word-per-line comparison and the manifest-driven extraction in PR #49, then check how #32 handles related headings. Done requires a maintainer decision on heading representation and an agreed scope for recovering the missing content.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- content
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100