Retention sweep broken under default seed: batch-purger table resolution, clear-history + oneshot model registration
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
BEP-1063 DB record retention does not work under the shipped default seed (all 8 categories enabled). Verified on a live local stack. Three code-path defects, all needed:
(1) [server, most severe] The background LeaderCron sweep fails every tick with AttributeError: 'ColumnSet' object has no attribute 'columns'. execute_batch_purger (repositories/base/purger.py) derives the target table from build_subquery().froms[0], assuming a sa.Table. DeploymentRevisionRow uses lazy="joined", so the compiled FROM is an _ORMJoin whose primary_key is a ColumnSet. Because the sweep is atomic per tick, this one broken category rolls back the entire tick, so nothing is ever purged under the default seed. Fix: resolve the table from the mapped entity (column_descriptions -> inspect().local_table) instead of the FROM clause.
(2) [CLI] mgr clear-history crashes on the forced sweep with InvalidRequestError: mapper KernelRow ... 'ImageRow' failed to locate a name. The standalone CLI process issues an ORM query before all models are imported. Redis stat cleanup runs but the sweep and VACUUM never execute. Fix: call ensure_all_tables_registered() before the sweep in _force_retention_sweep (cli/__main__.py).
(3) [setup] mgr schema oneshot does not create retention_policies. models/retention/__init__.py is empty (per the new no-reexport rule), so ensure_all_tables_registered() never registers the Row. The alembic upgrade path is fine. Fix: have ensure_all_tables_registered() also import each subpackage's .row module (models/base.py).
Additional hardening (approved): isolate each category in the sweep with a savepoint + try/except so a single category's failure no longer aborts the whole tick, mirroring the existing bulk_create_partial pattern.
Found via live-test of Epic BA-6924 / BEP-1063.
JIRA Issue: BA-6952
Contributor guide
Research direction
Start with repositories/base/purger.py and inspect how execute_batch_purger resolves the mapped entity, then read cli/__main__.py and models/base.py with models/retention/.row. Verify the default-seed retention sweep handles joined mappings, the forced CLI sweep registers all models, and the schema oneshot creates retention_policies. The sweep should isolate category failures so one error does not abort the tick.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sqlalchemy
- Domain
- backend, cli, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 50/100