opensanctions / opensanctions/rigour

Serialize the name-tagger automaton to cut first-use latency (~1.5s build -> ~7ms load)

Open
#217 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
67
Forks
18
Avg merge
13h 7m
Merged PRs (30d)
15

Description

Summary

The name tagger's Aho-Corasick automaton is rebuilt from its full needle set on the first analyze_names(..., symbols=true) call of every process. That build is ~1.5s and is the dominant cost of first use / cold start. It is cached for the process lifetime afterward (the Tier 1 LazyLock<AhoCorasick>), so it is purely a startup cost.

plans/arch-rust-core.md already calls this out and names the fix:

aho-corasick v1 exposes no serde or raw-byte serialisation, so the build is from-source. If startup cost ever matters, daachorse (double-array AC with explicit serialisation) is the escape hatch.

This issue is that escape hatch, with measurements showing startup cost does matter (it dominates cold start in serverless / short-lived-process deployments) and that the win is large.

Prototype and numbers

Swapping aho-corasick for daachorse (double-array Aho-Corasick, supports zero-copy de/serialization) on the person tagger's needle set, release build:

step time
aho-corasick build (current) 1553 ms
daachorse build (one-time, build side) 658 ms
daachorse serialize 55 ms, 25.7 MB
daachorse deserialize (runtime) ~7 ms

Overlapping-search results are byte-identical to the current aho-corasick output across a set of probe haystacks. Net: the ~1.5s runtime build becomes a ~7ms load (plus faulting/decompressing the embedded artifact), with output unchanged.

A note on size: the deduped needle count actually fed to the automaton measured ~470k (470,083 unique forms from 564,529 aliases). arch-rust-core.md estimates Tier 1 at ~150k patterns, which appears to count corpus lines/groups rather than the expanded, deduped per-form needle set.

On the rejected-formats list

arch-rust-core.md rejects bincode / postcard / rkyv with "zero-copy gains are irrelevant after init." That reasoning targets the small Tier-2/3 data and the steady state. The daachorse gain here is different: it removes the ~1.5s automaton build, which is paid at init. That is the "if startup cost ever matters" case the same doc explicitly carves out for daachorse, not the post-init zero-copy case that was rejected.

Scope and open questions (would want maintainer input before a PR)

  • Engine swap: aho-corasick -> daachorse in names::matcher::Needles, or a serialized fast path alongside it. daachorse is case-sensitive; rigour already pre-normalizes (CASEFOLD) both needles and haystack, so the current ascii_case_insensitive(true) is belt-and-suspenders, but full-suite parity needs verifying.
  • Where to serialize: build.rs cannot call the crate's own builder, so it would need a small gen step / workspace member (or genscripts/), or committing the serialized artifact.
  • Flag-keyed cache: precompute only for the default Normalize flags; other flag combinations fall back to a runtime build.
  • Artifact size: ~25.7 MB raw (zstd-compressible like the existing corpus), embedded in the wheel.

Happy to implement this as a PR if the approach sounds reasonable. Prototype bench is available.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with plans/arch-rust-core.md and the names::matcher::Needles implementation, then inspect build.rs and the available prototype benchmark. Work out how the default Normalize configuration can load a serialized daachorse artifact while other flag combinations retain a runtime build. Done means the startup path is reduced toward the reported ~7 ms and overlapping-search output remains byte-identical across the full suite.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.