Automattic / Automattic/harper
Tagger assigns tags the word's own dictionary entry does not license
- Dominant language
- Rust
- Stars
- 15.4k
- Forks
- 627
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 106
Description
Opening this at @hippietrail's suggestion in #4089, as a place to collect cases where the POS tagger assigns a tag the word's own dictionary entry does not license, so they can be fixed in batches rather than one at a time as linters trip over them.
## The canonical one
`mat`, from the "the cat sat on the mat" test:
```
word The cat sat on the {}. The cat sat on the {} A dog chewed the {}. I bought a {} yesterday. The {} was expensive.
mat DET DET DET DET PRON
rug NOUN NOUN NOUN NOUN NOUN
```
`harper-cli metadata mat` reports `NVJ` with `determiner: null`, so `DET` is not a reading the dictionary offers. It never comes back `NOUN` in any frame I tried, which makes it look lexical rather than contextual.
## Collecting these in bulk
Tagger against dictionary is checkable without human judgement, so the list can start with real entries. I sampled 2,037 lowercase dictionary words, put each in the frame `The cat sat on the ___.`, and compared `pos_tag` against the categories the word's own entry licenses:
```
2,037 sampled
620 came back with no pos_tag at all
1,417 checked
67 disagreements (62 NOUN, 3 ADV, 2 ADJ)
```
## The disagreements point in two directions
In a `the ___` slot `NOUN` is the syntactically expected tag, so I think most of the 62 `NOUN` rows mean the dictionary is missing a noun sense rather than the tagger being wrong:
```
represents tagger=NOUN dictionary: verb
identify tagger=NOUN dictionary: verb
went tagger=NOUN dictionary: verb
institutions tagger=NOUN dictionary: none
riders tagger=NOUN dictionary: none
```
Some of the non-`NOUN` rows look like dictionary gaps too, and they hold their tag across every frame:
```
indoor tagger=ADJ dictionary: noun+verb ADJ in all 5 frames
unmediated tagger=ADJ dictionary: verb ADJ in all 5 frames
```
Both of those are ordinary adjectives, so the entry looks incomplete rather than the tag being wrong.
That leaves the rows where the tag is implausible in every frame, which is the group I would call tagger errors. `mat` is the clearest. `inland`, `afresh` and `ling` come back `ADV` in four frames and `ADJ` in the fifth, against `noun+adjective`, `none` and `noun+verb+adjective` respectively.
So the same sweep produces two different work items, dictionary entries to complete and tagger behaviour to look at, and they need separating before either is actioned.
## Why it reaches further than POS
`np_member` is computed downstream of the tags, in `Document::parse`:
```rust
let token_tags = tagger.tag_sentence(&token_strings);
let np_flags = chunker.chunk_sentence(&token_strings, &token_tags);
```
`burn_chunker()` is a separate model from `BrillTagger`, but it reads the tags, so a bad tag propagates. On the same sentence:
```
the cat sat on the mat the=true cat=true sat=false on=false the=true mat=false
the cat sat on the rug the=true cat=true sat=false on=false the=true rug=true
```
`the mat` comes out as a determiner with nothing attached to it. Anything relying on `is_np_member` inherits that.
## Offer
The sweep is about 60 lines as a `harper-core` example. Happy to contribute it as one, or as a `just` recipe, if a repeatable version is worth having. Otherwise I can keep re-running it and adding batches here.
Contributor guide
Research direction
Start with Document::parse and trace how BrillTagger tags flow into burn_chunker; reproduce the `mat` case with `harper-cli metadata mat` and the documented sentence frames. Review the proposed harper-core example or just recipe, and clarify whether the result should separate dictionary gaps from tagger errors before defining completion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- devtools
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100