globalwordnet / globalwordnet/semcor

Document-boundary anchor search misattributes a file's own leading markup to the previous file

Open
#65 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 22h
Merged PRs (30d)
24

Description

Spotted via a stray standalone \`_\` word at the very end of \`br-a01\`'s
section of \`brown-nolines.diff\`, right before the \`=== br-a02 ===\`
header -- the underscore of \`_AUSTIN, TEXAS_- Committee approval...\`
(br-a02's own dateline) was being attributed to the end of br-a01
instead of the start of br-a02.

Root cause: \`locate_file_boundaries\`'s anchor search matches at the
first *word* of a file's first NLTK-tokenized sentence -- NLTK's own
corpus reader already strips markup, so the match itself can never see
it -- but a document's real opening is often a dateline's leading \`_\`,
a subheadline's \`#\`, or an opening quote/paren/brace that belongs to
*that* file, not a trailing leftover of the previous one. Left
stranded with no matching close within either file's own span,
\`decode_reference_text\`'s markup-stripping regexes (\`_UNDERSCORE_SPAN_RE\`
and friends) can't drop it, so it survives as a spurious extra word
exactly at the boundary.

Fixed by having the anchor search back up over any immediately
preceding whitespace-delimited token(s) containing no alphanumeric
character at all, stopping at a paragraph break so it never crosses
into content that genuinely belongs to the previous file (e.g. a
trailing \`**f\` formula placeholder right before a new file's own
\`#\`-prefixed dateline -- confirmed this exact case during testing).
**88 of the 352 file boundaries** needed the adjustment (almost
entirely single characters: \`_\`, \`#\`, \`{\`, \`<\`, \`"\`, \`(\`).
Regenerated \`brown-nolines-offsets.yaml\` via the existing
\`--regenerate-offsets\`; divergent word-line count drops by 109 with no
\`data/*.yaml\` changes at all.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at locate_file_boundaries and inspect how its anchor search handles the first token of each file, then review decode_reference_text and _UNDERSCORE_SPAN_RE for the related cleanup behavior. Run the existing --regenerate-offsets workflow for brown-nolines-offsets.yaml; the boundary attribution should be corrected, divergent word-line count should drop by 109, and data/*.yaml should remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.