dataiku / dataiku/kiji-proxy

Improve PII generator: more realistic fake data and lookup existing mappings first

Open
#465 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
432
Forks
51
PR merge metrics
No merged PRs in 30d

Description

Summary

Once entity persistence is restored (see #464), the next gap is the generator itself:

  1. The per-label generators use small inline arrays of fake values, so collisions and obviously-fake outputs are common — we should expand the pools and produce more realistic data (e.g., richer name pools, locale-aware addresses, realistic-looking but invalid credit card / SSN / phone formats).
  2. The masking pipeline regenerates a fresh dummy for every detected entity on every request. We should look up the persisted mapping first so the same original PII always maps to the same dummy, avoiding duplicates and making conversations consistent across turns.

Code path to review

Generator dispatch and per-type generators:

  • src/backend/pii/generator_service.go:54GenerateReplacement(label, originalText) and the label → generator routing table at :60
  • src/backend/pii/generators/pii_generators.go — all per-type generators (~500 lines). Example: EmailGenerator at :29 uses ~50 first names and ~40 last names; PhoneGenerator at :73. Expand the inline pools and/or load from data files.

Masking pipeline that currently bypasses the existing mapping:

  • src/backend/pii/masking_service.go:89-94 — the loop calls s.generator.GenerateReplacement(...) unconditionally for every detected entity. It does not check the existing store before generating, so each request produces a new dummy and over-writes the previous mapping via StoreMapping's upsert.

Mapping lookup that already exists and should be wired in:

  • src/backend/pii/mapper.go:106PIIMapping.GetDummy(original) — cache-first, falls through to SQLite. This is exactly the dedupe check we need before calling the generator.
  • src/backend/pii/database.go:185StoreMapping upserts; once dedupe is in place, repeated entities should hit the cache/DB and skip the generator entirely.

Suggested next steps

  1. In masking_service.go:89, before calling GenerateReplacement, pass the PIIMapping (or a small lookup interface) into MaskingService and check mapper.GetDummy(originalText) first. Only generate if there's no existing mapping. Then AddMapping the new one so future requests are consistent.
  2. Audit each generator in pii_generators.go for pool size and realism. Decide whether to keep them inline (and just expand) or move to embedded data files (embed.FS) so we can ship larger, locale-aware pools without bloating the source.
  3. Add tests in src/backend/pii/generators/pii_generators_test.go covering: (a) the generator never returns the original input, (b) repeated calls for the same (label, original) return the persisted dummy rather than a new one (this test will fail until step 1 lands).

Depends on

  • #464 (entity persistence regression) — the dedupe behavior in step 1 only matters once StoreMapping is actually persisting again.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read #464 first, then inspect GenerateReplacement and routing in src/backend/pii/generator_service.go, the per-type implementations in src/backend/pii/generators/pii_generators.go, and the masking loop in src/backend/pii/masking_service.go. Check PIIMapping.GetDummy in mapper.go and StoreMapping in database.go before running src/backend/pii/generators/pii_generators_test.go. Done means existing mappings are reused and generator tests cover non-original, realistic replacement behavior and repeated-entity consistency.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sqlite
Domain
backend, databases, security
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.