Automattic / Automattic/harper
The order of words in the dictionary can affect spell checking
- Dominant language
- Rust
- Stars
- 15.4k
- Forks
- 627
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 102
Description
In my new dictionary curation I sorted the dictionary and that resulted in the order of `Tex` vs `TeX` being reversed, which caused #1873 to show up again.
It was very hard to track this down as there were no code changes in the PR.
The dictionary is case-folded when it's compiled so that `Tex` and `TeX` are the same entry with the same metadata including orthography info.
Case folding should be commutative and not matter what the order is.
I'm not familiar with the spellcheck code. There is some concept of canonical spelling involved. Case folded entries don't have a canonical spelling. Or rather they have multiple canonical spellings but they can't necessarily be retrieved. It might depend on the heuristics though.
I suspect a bug in how the metadata for the first and second word getting case folded is done but it could be a bug in how the canonical spelling is decided.
Here's some example tests that can be added to `harper-core/src/spell/rune/mod.rs` to show the effect:
```rs
#[test]
fn case_folded_tex_before_TeX() {
// Test: Tex comes before TeX in the dictionary
let words = parse_word_list("2\nTex/O\nTeX/O").unwrap();
let attributes = AttributeList::parse(
&json!({
"affixes": {},
"properties": {
"O": {
"#": "proper noun",
"metadata": {
"noun": {
"is_proper": true
}
}
}
}
})
.to_string(),
)
.unwrap();
let mut expanded = WordMap::default();
attributes.expand_annotated_words(words, &mut expanded);
// Both should be in the map
assert!(expanded.get_with_str("Tex").is_some());
assert!(expanded.get_with_str("TeX").is_some());
// The canonical spelling should be TeX (last one wins)
let entry = expanded.get_with_str("tex").unwrap();
let canonical: String = entry.canonical_spelling.iter().collect();
assert_eq!(canonical, "TeX");
}
#[test]
fn case_folded_TeX_before_Tex() {
// Test: TeX comes before Tex in the dictionary
let words = parse_word_list("2\nTeX/O\nTex/O").unwrap();
let attributes = AttributeList::parse(
&json!({
"affixes": {},
"properties": {
"O": {
"#": "proper noun",
"metadata": {
"noun": {
"is_proper": true
}
}
}
}
})
.to_string(),
)
.unwrap();
let mut expanded = WordMap::default();
attributes.expand_annotated_words(words, &mut expanded);
// Both should be in the map
assert!(expanded.get_with_str("Tex").is_some());
assert!(expanded.get_with_str("TeX").is_some());
// The canonical spelling should be Tex (last one wins)
let entry = expanded.get_with_str("tex").unwrap();
let canonical: String = entry.canonical_spelling.iter().collect();
assert_eq!(canonical, "Tex");
}
```
Contributor guide
Research direction
Start with the case-folding tests in harper-core/src/spell/rune/mod.rs and run the existing spell/rune test suite. Trace AttributeList.expand_annotated_words, WordMap, and canonical_spelling for both dictionary orders; done means both entries remain available and canonical spelling behavior no longer depends on input order.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- devtools
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100