globalwordnet / globalwordnet/semcor

Adjacent function words spuriously merged into a single mistagged token

Open
#10 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 22h
Merged PRs (30d)
24

Description

Part of #5.

A large number of tokens in this corpus are two ordinary function words joined with an underscore and given a single, uniform POS tag — almost always `RB` — where the raw Brown corpus ([`browntag_nolines.txt`](http://www.sls.hawaii.edu/bley-vroman/browntag_nolines.txt)) has them as two separate tokens with their own (context-dependent) tags. These don't look like genuine WordNet multiword expressions (unlike legitimate joins such as `take_place` or `in_order_to`); they look like an over-eager collocation/merging step that swallowed ordinary syntactic sequences.

Examples (SemCor form → tag, vs. the actual Brown tokenization):

| This corpus | Brown corpus |
|---|---|
| `in_which` → `RB` | `in_IN which_WDT` |
| `of_this` → `RB` | `of_IN this_DT` |
| `so_that` → `RB` | `so_CS that_CS` (occasionally `so_RB that_CS`) |
| `and_then` → `RB` | `and_CC then_RB` |
| `that_is` → `RB` | `that_DT/WPS is_BEZ` |
| `collided_with` (lemma `collide_with`) → `VB` | `collided_VBD with_IN` |

Concrete instance: `data/press_reportage/br-a24.yaml`, sentence `qQZX`, has `..."an accident in_which a car he was driving collided_with a car driven by..."` — both `in_which` and `collided_with` are single tokens each carrying one tag, where Brown has four separate, correctly tagged tokens (`in_IN`, `which_WDT`, `collided_VBD`, `with_IN`).

### Scope

A scan for just two-word merges where *both* halves are common function words (a narrow, conservative pattern — it won't catch three-word merges or content-word + function-word merges) already found **1,227 occurrences across 34 distinct word pairs**, e.g.:

`of_this` (213), `in_this` (192), `in_which` (126), `so_that` (98), `of_it` (76), `and_then` (72), `that_is` (71), `of_that` (47), `in_that` (46), `of_which` (44), `on_which` (28), `to_it` (25), `if_not` (22), `with_it` (21), `on_it` (21), `to_which` (21), `or_so` (17), `to_that` (13), `from_that` (12), `from_it` (12), plus 14 more distinct pairs.

This is a correctness issue beyond just tagging: because tokens are merged, downstream `wn16_key`/`wn30_key`/`oewn_key` sense annotations index into a shorter, differently-aligned token list than Brown's, and any tool trying to align this corpus to Brown/NLTK by token position will desync at every one of these merges.

### Suggested fix

Identify which merged tokens are genuine multiword lexical items (should stay merged) vs. accidental function-word merges (should be split back into Brown's original tokens with their original tags restored). The pattern above (`{prep} {this/that/it/which}`, `so_that`, `and_then`, `V-ed_with`) suggests this is systematic rather than one-off, so it's likely traceable to a specific step in whatever produced this data.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with data/press_reportage/br-a24.yaml and sentence qQZX, then compare its merged tokens and tags with the linked Brown corpus tokenization. Trace the data-generation step responsible for the 1,227 function-word merges and identify which joins are genuine multiword items. Done means accidental merges are split with Brown's original tags while legitimate lexical items and downstream sense-key alignment remain intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.