globalwordnet / globalwordnet/semcor
Adjacent function words spuriously merged into a single mistagged token
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 24
Description
Part of #5.
A large number of tokens in this corpus are two ordinary function words joined with an underscore and given a single, uniform POS tag — almost always `RB` — where the raw Brown corpus ([`browntag_nolines.txt`](http://www.sls.hawaii.edu/bley-vroman/browntag_nolines.txt)) has them as two separate tokens with their own (context-dependent) tags. These don't look like genuine WordNet multiword expressions (unlike legitimate joins such as `take_place` or `in_order_to`); they look like an over-eager collocation/merging step that swallowed ordinary syntactic sequences.
Examples (SemCor form → tag, vs. the actual Brown tokenization):
| This corpus | Brown corpus |
|---|---|
| `in_which` → `RB` | `in_IN which_WDT` |
| `of_this` → `RB` | `of_IN this_DT` |
| `so_that` → `RB` | `so_CS that_CS` (occasionally `so_RB that_CS`) |
| `and_then` → `RB` | `and_CC then_RB` |
| `that_is` → `RB` | `that_DT/WPS is_BEZ` |
| `collided_with` (lemma `collide_with`) → `VB` | `collided_VBD with_IN` |
Concrete instance: `data/press_reportage/br-a24.yaml`, sentence `qQZX`, has `..."an accident in_which a car he was driving collided_with a car driven by..."` — both `in_which` and `collided_with` are single tokens each carrying one tag, where Brown has four separate, correctly tagged tokens (`in_IN`, `which_WDT`, `collided_VBD`, `with_IN`).
### Scope
A scan for just two-word merges where *both* halves are common function words (a narrow, conservative pattern — it won't catch three-word merges or content-word + function-word merges) already found **1,227 occurrences across 34 distinct word pairs**, e.g.:
`of_this` (213), `in_this` (192), `in_which` (126), `so_that` (98), `of_it` (76), `and_then` (72), `that_is` (71), `of_that` (47), `in_that` (46), `of_which` (44), `on_which` (28), `to_it` (25), `if_not` (22), `with_it` (21), `on_it` (21), `to_which` (21), `or_so` (17), `to_that` (13), `from_that` (12), `from_it` (12), plus 14 more distinct pairs.
This is a correctness issue beyond just tagging: because tokens are merged, downstream `wn16_key`/`wn30_key`/`oewn_key` sense annotations index into a shorter, differently-aligned token list than Brown's, and any tool trying to align this corpus to Brown/NLTK by token position will desync at every one of these merges.
### Suggested fix
Identify which merged tokens are genuine multiword lexical items (should stay merged) vs. accidental function-word merges (should be split back into Brown's original tokens with their original tags restored). The pattern above (`{prep} {this/that/it/which}`, `so_that`, `and_then`, `V-ed_with`) suggests this is systematic rather than one-off, so it's likely traceable to a specific step in whatever produced this data.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with data/press_reportage/br-a24.yaml and sentence qQZX, then compare its merged tokens and tags with the linked Brown corpus tokenization. Trace the data-generation step responsible for the 1,227 function-word merges and identify which joins are genuine multiword items. Done means accidental merges are split with Brown's original tags while legitimate lexical items and downstream sense-key alignment remain intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100