Purge retention targets through the v2 batch purge specs
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
Retention purges rows through a bespoke primitive (RetentionDrain / RetentionWriteOps, added in BA-7460) instead of the v2 batch purge specs. That primitive was a workaround for two gaps in the v2 purge surface, and it hides a defect.
**Defect it hides**
Several retention targets are entities: endpoints, session_groups, vfolder_invitations (BA-7459 settled the last one as an EntityBatchPurger). Retention hard-deletes their rows and leaves what they put in the RBAC graph – virtual_scopes, entity_memberships, scope_bindings and the permissions granted on them. Those orphans accumulate on every sweep and nothing else clears them. The legacy batch_purge behaved the same way, so this predates BA-7460.
**Gap 1 – no unowned batch purge for field and sidecar rows**
The public field path is batch_purge_field_entities(owner_id, purger), and models/specs/AGENTS.md fixes the axis: an entity batch is bounded by the scopes the call names, a field batch by the owner it is given. Retention is bounded by neither – it deletes on a time boundary. The entity half is fine on batch_purge_entities_in_global (retention is global by nature, so no scope is involved), but the field and sidecar halves have nowhere to go. models/specs already names the sidecar row kind and has DanglingFieldCreator for creates; the purge counterpart is missing.
**Gap 2 – batch purge cannot report a count, and cannot be paced**
_batch_purge_returning accumulates every deleted row's data across all chunks, and _batch_purge_entities additionally accumulates every entity id and then calls _teardown_entity per id, which is four DELETE statements each. A category draining millions of rows would hold an unbounded list and issue 4xN statements. Neither public method exposes batch_size, so retention cannot pass its configured value. Entity teardown does need the rows, but per chunk – not held to the end.
**Scope**
- Add the missing purge root for owner-free batch deletes to models/specs/purger.py, plus its ops method.
- Give the batch purge paths a batch_size argument and a chunk-streaming form that answers with a count instead of the rows; make the entity teardown set-based per chunk rather than one call per row.
- Split the retention catalog by row kind and declare each target as an EntityBatchPurger or the new root, next to the row it removes.
- Delete repositories/ops/v2/retention/ once nothing needs it.
**Verification**
The entity targets must leave no virtual_scopes, entity_memberships, scope_bindings or permissions rows behind after a sweep – that assertion is what proves the defect fixed. Existing retention tests cover the row counts already.
JIRA Issue: BA-7479
Contributor guide
Research direction
Start with models/specs/AGENTS.md and models/specs/purger.py, then trace the existing batch purge ops and the retention catalog under repositories/ops/v2/retention/. Run the existing retention tests and inspect their row-count assertions. Done means retention uses the v2 batch purge roots with configurable chunking, and entity sweeps leave no virtual_scopes, entity_memberships, scope_bindings, or permission rows behind.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- authorization, backend, database
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100