globalwordnet / globalwordnet/semcor

Build an automated Brown-corpus verification tool covering the whole corpus

Open
#12 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 22h
Merged PRs (30d)
24

Description

Part of #5.

The other sub-issues here (#8 #9 #10 #11) were each found by hand-comparing a sample of files against the raw ([`brown_nolines.txt`](http://www.sls.hawaii.edu/bley-vroman/brown_nolines.txt)) and tagged ([`browntag_nolines.txt`](http://www.sls.hawaii.edu/bley-vroman/browntag_nolines.txt)) Brown corpus dumps. That was enough to find several systematic bugs, but it was manual and only sampled a handful of files out of 352 — there are almost certainly more (including genuine word-level/lexical divergences we haven't gone looking for yet, as opposed to the systematic whitespace/tokenization ones already filed).

This issue is to build the actual automated verification tool the original issue asked for, so the whole corpus (not a sample) gets checked, and stays checked as the corpus changes.

### What it needs to handle

The main obstacle is that neither reference source marks document boundaries: `brown_nolines.txt` is Brown's full ~500 files concatenated with no file IDs (one paragraph per line), and `browntag_nolines.txt` likewise (one sentence per line). A workable approach (used for the manual investigation):

1. Load a reference token stream with (word, Brown POS tag) pairs, in corpus order — either from `browntag_nolines.txt`, or via NLTK's `nltk.corpus.brown` (per the original issue), which does expose per-file access (`brown.words(fileids=...)` / `brown.tagged_words(...)`) and would sidestep the boundary problem entirely. Worth checking whether NLTK's copy and the Hawaii dump actually agree (a spot check while doing this investigation found at least one place where a sentence in one didn't appear in the other, so they may not be identical editions).
2. For each `data//br-*.yaml` file, in Brown's canonical genre/file order, locate its span in the reference stream by anchor-matching its first/last few tokens (normalized: lowercased, underscores expanded), then diff the file's tokens against that slice.
3. Report, per file, divergences bucketed by type rather than a flat list — at minimum:
- text/whitespace differences (excluding the underscore-vs-space multiword-join convention, which is intentional)
- token count mismatches (splits/merges vs. Brown's tokenization)
- tag differences on aligned tokens
- word/lemma differences on aligned tokens (genuine lexical divergences, e.g. OCR fixes or typos that don't match Brown)
4. Emit a Markdown report (per the original issue), plus ideally machine-readable output (counts per bucket) so it can be re-run and diffed over time to check whether fixes for #8–#11 actually reduced the divergence counts.

### Non-goals

This issue is just the tool. Don't use it as an excuse to also fix everything it finds — file new sub-issues for whatever categories of divergence it surfaces that aren't already covered by #8–#11, the same way those were filed.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by inspecting the data//br-*.yaml files and the existing Brown-corpus handling, then compare nltk.corpus.brown with brown_nolines.txt and browntag_nolines.txt for boundary and edition differences. Design the verifier around corpus-order alignment and the four requested divergence buckets, with a Markdown report and machine-readable counts; keep fixes for findings in separate issues such as #8–#11.

Written by the indexing model from the issue text.

Assessment

Tech stack
markdown, python, yaml
Domain
data, testing, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.