globalwordnet / globalwordnet/semcor
4 excluded formula-marker anomalies from #16 (word-fusion, footnote, data loss)
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 24
Description
Part of #5. Split off from #16's fix (PR TBD) — the same investigation
that found 602 clean `**f`/`**h` placeholder-marker merges (verified
against http://www.sls.hawaii.edu/bley-vroman/brown_nolines.txt) also
turned up 4 differently-shaped occurrences deliberately left unfixed
there, since none of them share that fix's shape.
### `f_Numbers`: an independent word-fusion bug (2 instances)
`data/skill_and_hobbies/br-e08.yaml`, sentences `Jhh2` and `rnio`:
`* * f_Numbers ...`. The source has `**f Numbers ...` — a formula
placeholder followed by a separate word, `Numbers`, starting the next
clause. This corpus's own tokenizer *also* fused `f` and `Numbers`
together into one token (`f_Numbers`), the same shape of bug #10 fixes
for function words, but for a content word. Fixing this needs a
merge (the two `*` tokens into `f`) *and* a split (`f_Numbers` into
`f` + `Numbers`) in the same place — too rare (2 instances) to build
combined merge+split logic for as part of #16.
### Unrelated footnote-style lone `*` (1 instance)
`data/press_reportage/br-a33.yaml`, sentence `mEZA`: `... the * White_House
library ...` — a single, undoubled `*`, not the `**` pattern at
all. Across the whole corpus this is the *only* `*` character not part
of a `**` pair, so whatever it originally marked (a footnote reference?)
is a one-off, unrelated to the formula/paragraph-marker fix.
### Dangling end-of-document marker, letter missing entirely (1 instance)
`data/skill_and_hobbies/br-e08.yaml`, sentence `Qh0L` ends with `...the
equation becomes: * *` and nothing after — no third token to merge into
at all. This is the corpus's very last sentence in that file, so the
placeholder's own code letter isn't just un-glued, it's missing
entirely: real data loss, not a spacing bug. Nothing to mechanically
recover it with.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the four named sentences in data/skill_and_hobbies/br-e08.yaml and data/press_reportage/br-a33.yaml, then read the fixes for #16 and #10 to understand the existing marker handling. Determine an explicit treatment for the two fused tokens, the lone asterisk, and the missing end marker. Done means all four anomalies have a documented, tested treatment without being incorrectly handled by the 602-case fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100