globalwordnet / globalwordnet/semcor

4 excluded formula-marker anomalies from #16 (word-fusion, footnote, data loss)

Open
#34 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 22h
Merged PRs (30d)
24

Description

Part of #5. Split off from #16's fix (PR TBD) — the same investigation
that found 602 clean `**f`/`**h` placeholder-marker merges (verified
against http://www.sls.hawaii.edu/bley-vroman/brown_nolines.txt) also
turned up 4 differently-shaped occurrences deliberately left unfixed
there, since none of them share that fix's shape.

### `f_Numbers`: an independent word-fusion bug (2 instances)

`data/skill_and_hobbies/br-e08.yaml`, sentences `Jhh2` and `rnio`:
`* * f_Numbers ...`. The source has `**f Numbers ...` — a formula
placeholder followed by a separate word, `Numbers`, starting the next
clause. This corpus's own tokenizer *also* fused `f` and `Numbers`
together into one token (`f_Numbers`), the same shape of bug #10 fixes
for function words, but for a content word. Fixing this needs a
merge (the two `*` tokens into `f`) *and* a split (`f_Numbers` into
`f` + `Numbers`) in the same place — too rare (2 instances) to build
combined merge+split logic for as part of #16.

### Unrelated footnote-style lone `*` (1 instance)

`data/press_reportage/br-a33.yaml`, sentence `mEZA`: `... the * White_House
library ...` — a single, undoubled `*`, not the `**` pattern at
all. Across the whole corpus this is the *only* `*` character not part
of a `**` pair, so whatever it originally marked (a footnote reference?)
is a one-off, unrelated to the formula/paragraph-marker fix.

### Dangling end-of-document marker, letter missing entirely (1 instance)

`data/skill_and_hobbies/br-e08.yaml`, sentence `Qh0L` ends with `...the
equation becomes: * *` and nothing after — no third token to merge into
at all. This is the corpus's very last sentence in that file, so the
placeholder's own code letter isn't just un-glued, it's missing
entirely: real data loss, not a spacing bug. Nothing to mechanically
recover it with.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the four named sentences in data/skill_and_hobbies/br-e08.yaml and data/press_reportage/br-a33.yaml, then read the fixes for #16 and #10 to understand the existing marker handling. Determine an explicit treatment for the two fused tokens, the lone asterisk, and the missing end marker. Done means all four anomalies have a documented, tested treatment without being incorrectly handled by the 602-case fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.