Improve PII generator: more realistic fake data and lookup existing mappings first
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 432
- Forks
- 51
- PR merge metrics
- No merged PRs in 30d
Description
Summary
Once entity persistence is restored (see #464), the next gap is the generator itself:
- The per-label generators use small inline arrays of fake values, so collisions and obviously-fake outputs are common — we should expand the pools and produce more realistic data (e.g., richer name pools, locale-aware addresses, realistic-looking but invalid credit card / SSN / phone formats).
- The masking pipeline regenerates a fresh dummy for every detected entity on every request. We should look up the persisted mapping first so the same original PII always maps to the same dummy, avoiding duplicates and making conversations consistent across turns.
Code path to review
Generator dispatch and per-type generators:
src/backend/pii/generator_service.go:54—GenerateReplacement(label, originalText)and the label → generator routing table at:60src/backend/pii/generators/pii_generators.go— all per-type generators (~500 lines). Example:EmailGeneratorat:29uses ~50 first names and ~40 last names;PhoneGeneratorat:73. Expand the inline pools and/or load from data files.
Masking pipeline that currently bypasses the existing mapping:
src/backend/pii/masking_service.go:89-94— the loop callss.generator.GenerateReplacement(...)unconditionally for every detected entity. It does not check the existing store before generating, so each request produces a new dummy and over-writes the previous mapping viaStoreMapping's upsert.
Mapping lookup that already exists and should be wired in:
src/backend/pii/mapper.go:106—PIIMapping.GetDummy(original)— cache-first, falls through to SQLite. This is exactly the dedupe check we need before calling the generator.src/backend/pii/database.go:185—StoreMappingupserts; once dedupe is in place, repeated entities should hit the cache/DB and skip the generator entirely.
Suggested next steps
- In
masking_service.go:89, before callingGenerateReplacement, pass thePIIMapping(or a small lookup interface) intoMaskingServiceand checkmapper.GetDummy(originalText)first. Only generate if there's no existing mapping. ThenAddMappingthe new one so future requests are consistent. - Audit each generator in
pii_generators.gofor pool size and realism. Decide whether to keep them inline (and just expand) or move to embedded data files (embed.FS) so we can ship larger, locale-aware pools without bloating the source. - Add tests in
src/backend/pii/generators/pii_generators_test.gocovering: (a) the generator never returns the original input, (b) repeated calls for the same(label, original)return the persisted dummy rather than a new one (this test will fail until step 1 lands).
Depends on
- #464 (entity persistence regression) — the dedupe behavior in step 1 only matters once
StoreMappingis actually persisting again.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read #464 first, then inspect GenerateReplacement and routing in src/backend/pii/generator_service.go, the per-type implementations in src/backend/pii/generators/pii_generators.go, and the masking loop in src/backend/pii/masking_service.go. Check PIIMapping.GetDummy in mapper.go and StoreMapping in database.go before running src/backend/pii/generators/pii_generators_test.go. Done means existing mappings are reused and generator tests cover non-original, realistic replacement behavior and repeated-entity consistency.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sqlite
- Domain
- backend, databases, security
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100