Automattic / Automattic/harper
`FstDictionary::curated()` keeps a second copy of the whole dictionary in memory
- Dominant language
- Rust
- Stars
- 15.4k
- Forks
- 627
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 106
Description
> **Disclaimer:** I used an LLM agent (Claude Code) to run these measurements and to write this
> report. The numbers come from builds on my own machine and I have checked them; the patch at the
> bottom is agent-written. Flagging this per `AGENT_POLICY.md`.
Following up on #3725 / #4030: after that fix landed I re-measured the dictionary's memory
footprint, because I maintain the Slovak dictionary data for #3658 and RAM is the main thing
standing between it and a shippable artifact. There is a second copy of the dictionary still
resident, and this one is larger than the `Vec` that #4030 removed.
## What happens
`FstDictionary::curated()` clones the curated `MutableDictionary` and then rebuilds another one
inside `FstDictionary::new()`:
```rust
// harper-core/src/spell/fst_dictionary.rs
static DICT: LazyLock> =
LazyLock::new(|| Arc::new((*MutableDictionary::curated()).clone().into()));
```
`MutableDictionary::curated()` materialises the static `DICT` in `mutable_dictionary.rs`, which
then stays alive in its `LazyLock` forever. The clone is consumed by
`impl From for FstDictionary` into a `Vec<(CharString, DictWordMetadata)>`, and
`FstDictionary::new()` calls `mutable_dict.extend_words(...)` to build a *third* word map, which is
the one the returned `FstDictionary` keeps. Steady state is therefore two complete word maps: the
static one nobody can reach any more, and the one inside the `FstDictionary`.
## Measurements
`VmRSS` read from `/proc/self/status` right after `FstDictionary::curated()` returns; release
builds, rustc 1.97.1, Linux, measured at `52c6aac6`.
| Dictionary | Word forms | `VmRSS` today | With the patch below |
|---|---|---|---|
| English (curated) | 134,729 | 99.9 MB | **56.2 MB** (−44%) |
| German (from the #3402 branch) | 655,674 | 368.4 MB | **192.2 MB** (−48%) |
| Slovak (my WIP for #3658) | 2,233,558 | 1,305.8 MB | **698.0 MB** (−46%) |
Peak RSS drops as well — for Slovak `VmHWM` goes from 1,885 MB to 918 MB — and building the FST
gets faster (2.20 s → 1.23 s), since the word map is no longer cloned and rebuilt.
The split for Slovak, to show where the memory sits: `MutableDictionary::curated()` on its own is
616 MB, and `FstDictionary::curated()` takes it to 1,306 MB. The FST itself is a small part of
that difference; most of it is the duplicate word map.
## Patch
Build the FST over the shared `Arc` instead of cloning:
```rust
static DICT: LazyLock> =
LazyLock::new(|| Arc::new(FstDictionary::from_shared(MutableDictionary::curated())));
pub fn from_shared(mutable_dict: Arc) -> Self {
let mut words: Vec = mutable_dict
.words_iter()
.map(|word| word.iter().copied().collect())
.collect();
words.sort_unstable();
words.dedup();
let mut builder = fst::MapBuilder::memory();
for word_chars in words.iter() {
let word = word_chars.iter().collect::();
builder
.insert(word, WordId::from_word_chars(word_chars).into())
.expect("Insertion not in lexicographical order!");
}
let fst_bytes = builder.into_inner().unwrap();
let word_map = FstMap::new(fst_bytes).expect("Unable to build FST map.");
FstDictionary { mutable_dict, word_map }
}
```
`FstDictionary::new()` is left as it is for callers that own their word list.
`cargo test --release -p harper-core` passes, and `harper-cli lint --only SpellCheck --format json`
over a file of misspellings produces byte-identical output before and after, so the suggestions do
not change.
Happy to open a pull request with this if it is useful — or, given the agent policy, you may prefer
to write the change yourself; the finding is the part I wanted to get to you either way.
Contributor guide
Research direction
Review harper-core/src/spell/fst_dictionary.rs and mutable_dictionary.rs, starting with the curated dictionary initialization and the existing FstDictionary::new() path. Run cargo test --release -p harper-core, then use harper-cli lint --only SpellCheck --format json on misspellings. Done means the duplicate dictionary storage is gone, tests pass, and lint output remains byte-identical.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 62/100