globalwordnet / globalwordnet/semcor
Build an automated Brown-corpus verification tool covering the whole corpus
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 24
Description
Part of #5.
The other sub-issues here (#8 #9 #10 #11) were each found by hand-comparing a sample of files against the raw ([`brown_nolines.txt`](http://www.sls.hawaii.edu/bley-vroman/brown_nolines.txt)) and tagged ([`browntag_nolines.txt`](http://www.sls.hawaii.edu/bley-vroman/browntag_nolines.txt)) Brown corpus dumps. That was enough to find several systematic bugs, but it was manual and only sampled a handful of files out of 352 — there are almost certainly more (including genuine word-level/lexical divergences we haven't gone looking for yet, as opposed to the systematic whitespace/tokenization ones already filed).
This issue is to build the actual automated verification tool the original issue asked for, so the whole corpus (not a sample) gets checked, and stays checked as the corpus changes.
### What it needs to handle
The main obstacle is that neither reference source marks document boundaries: `brown_nolines.txt` is Brown's full ~500 files concatenated with no file IDs (one paragraph per line), and `browntag_nolines.txt` likewise (one sentence per line). A workable approach (used for the manual investigation):
1. Load a reference token stream with (word, Brown POS tag) pairs, in corpus order — either from `browntag_nolines.txt`, or via NLTK's `nltk.corpus.brown` (per the original issue), which does expose per-file access (`brown.words(fileids=...)` / `brown.tagged_words(...)`) and would sidestep the boundary problem entirely. Worth checking whether NLTK's copy and the Hawaii dump actually agree (a spot check while doing this investigation found at least one place where a sentence in one didn't appear in the other, so they may not be identical editions).
2. For each `data//br-*.yaml` file, in Brown's canonical genre/file order, locate its span in the reference stream by anchor-matching its first/last few tokens (normalized: lowercased, underscores expanded), then diff the file's tokens against that slice.
3. Report, per file, divergences bucketed by type rather than a flat list — at minimum:
- text/whitespace differences (excluding the underscore-vs-space multiword-join convention, which is intentional)
- token count mismatches (splits/merges vs. Brown's tokenization)
- tag differences on aligned tokens
- word/lemma differences on aligned tokens (genuine lexical divergences, e.g. OCR fixes or typos that don't match Brown)
4. Emit a Markdown report (per the original issue), plus ideally machine-readable output (counts per bucket) so it can be re-run and diffed over time to check whether fixes for #8–#11 actually reduced the divergence counts.
### Non-goals
This issue is just the tool. Don't use it as an excuse to also fix everything it finds — file new sub-issues for whatever categories of divergence it surfaces that aren't already covered by #8–#11, the same way those were filed.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by inspecting the data//br-*.yaml files and the existing Brown-corpus handling, then compare nltk.corpus.brown with brown_nolines.txt and browntag_nolines.txt for boundary and edition differences. Design the verifier around corpus-order alignment and the four requested divergence buckets, with a Markdown report and machine-readable counts; keep fixes for findings in separate issues such as #8–#11.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- markdown, python, yaml
- Domain
- data, testing, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100