globalwordnet / globalwordnet/semcor
Document-boundary anchor search misattributes a file's own leading markup to the previous file
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 24
Description
Spotted via a stray standalone \`_\` word at the very end of \`br-a01\`'s
section of \`brown-nolines.diff\`, right before the \`=== br-a02 ===\`
header -- the underscore of \`_AUSTIN, TEXAS_- Committee approval...\`
(br-a02's own dateline) was being attributed to the end of br-a01
instead of the start of br-a02.
Root cause: \`locate_file_boundaries\`'s anchor search matches at the
first *word* of a file's first NLTK-tokenized sentence -- NLTK's own
corpus reader already strips markup, so the match itself can never see
it -- but a document's real opening is often a dateline's leading \`_\`,
a subheadline's \`#\`, or an opening quote/paren/brace that belongs to
*that* file, not a trailing leftover of the previous one. Left
stranded with no matching close within either file's own span,
\`decode_reference_text\`'s markup-stripping regexes (\`_UNDERSCORE_SPAN_RE\`
and friends) can't drop it, so it survives as a spurious extra word
exactly at the boundary.
Fixed by having the anchor search back up over any immediately
preceding whitespace-delimited token(s) containing no alphanumeric
character at all, stopping at a paragraph break so it never crosses
into content that genuinely belongs to the previous file (e.g. a
trailing \`**f\` formula placeholder right before a new file's own
\`#\`-prefixed dateline -- confirmed this exact case during testing).
**88 of the 352 file boundaries** needed the adjustment (almost
entirely single characters: \`_\`, \`#\`, \`{\`, \`<\`, \`"\`, \`(\`).
Regenerated \`brown-nolines-offsets.yaml\` via the existing
\`--regenerate-offsets\`; divergent word-line count drops by 109 with no
\`data/*.yaml\` changes at all.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at locate_file_boundaries and inspect how its anchor search handles the first token of each file, then review decode_reference_text and _UNDERSCORE_SPAN_RE for the related cleanup behavior. Run the existing --regenerate-offsets workflow for brown-nolines-offsets.yaml; the boundary attribution should be corrected, divergent word-line count should drop by 109, and data/*.yaml should remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100