opensanctions / opensanctions/rigour
Serialize the name-tagger automaton to cut first-use latency (~1.5s build -> ~7ms load)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 67
- Forks
- 18
- Avg merge
- 13h 7m
- Merged PRs (30d)
- 15
Description
Summary
The name tagger's Aho-Corasick automaton is rebuilt from its full needle set on the first analyze_names(..., symbols=true) call of every process. That build is ~1.5s and is the dominant cost of first use / cold start. It is cached for the process lifetime afterward (the Tier 1 LazyLock<AhoCorasick>), so it is purely a startup cost.
plans/arch-rust-core.md already calls this out and names the fix:
aho-corasick v1 exposes no serde or raw-byte serialisation, so the build is from-source. If startup cost ever matters,
daachorse(double-array AC with explicit serialisation) is the escape hatch.
This issue is that escape hatch, with measurements showing startup cost does matter (it dominates cold start in serverless / short-lived-process deployments) and that the win is large.
Prototype and numbers
Swapping aho-corasick for daachorse (double-array Aho-Corasick, supports zero-copy de/serialization) on the person tagger's needle set, release build:
| step | time |
|---|---|
| aho-corasick build (current) | 1553 ms |
| daachorse build (one-time, build side) | 658 ms |
| daachorse serialize | 55 ms, 25.7 MB |
| daachorse deserialize (runtime) | ~7 ms |
Overlapping-search results are byte-identical to the current aho-corasick output across a set of probe haystacks. Net: the ~1.5s runtime build becomes a ~7ms load (plus faulting/decompressing the embedded artifact), with output unchanged.
A note on size: the deduped needle count actually fed to the automaton measured ~470k (470,083 unique forms from 564,529 aliases). arch-rust-core.md estimates Tier 1 at ~150k patterns, which appears to count corpus lines/groups rather than the expanded, deduped per-form needle set.
On the rejected-formats list
arch-rust-core.md rejects bincode / postcard / rkyv with "zero-copy gains are irrelevant after init." That reasoning targets the small Tier-2/3 data and the steady state. The daachorse gain here is different: it removes the ~1.5s automaton build, which is paid at init. That is the "if startup cost ever matters" case the same doc explicitly carves out for daachorse, not the post-init zero-copy case that was rejected.
Scope and open questions (would want maintainer input before a PR)
- Engine swap:
aho-corasick->daachorseinnames::matcher::Needles, or a serialized fast path alongside it. daachorse is case-sensitive; rigour already pre-normalizes (CASEFOLD) both needles and haystack, so the currentascii_case_insensitive(true)is belt-and-suspenders, but full-suite parity needs verifying. - Where to serialize:
build.rscannot call the crate's own builder, so it would need a small gen step / workspace member (orgenscripts/), or committing the serialized artifact. - Flag-keyed cache: precompute only for the default
Normalizeflags; other flag combinations fall back to a runtime build. - Artifact size: ~25.7 MB raw (zstd-compressible like the existing corpus), embedded in the wheel.
Happy to implement this as a PR if the approach sounds reasonable. Prototype bench is available.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with plans/arch-rust-core.md and the names::matcher::Needles implementation, then inspect build.rs and the available prototype benchmark. Work out how the default Normalize configuration can load a serialized daachorse artifact while other flag combinations retain a runtime build. Done means the startup path is reduced toward the reported ~7 ms and overlapping-search output remains byte-identical across the full suite.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100