lablup / lablup/backend.ai

Retention sweep broken under default seed: batch-purger table resolution, clear-history + oneshot model registration

Open
#12,993 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

BEP-1063 DB record retention does not work under the shipped default seed (all 8 categories enabled). Verified on a live local stack. Three code-path defects, all needed:

(1) [server, most severe] The background LeaderCron sweep fails every tick with AttributeError: 'ColumnSet' object has no attribute 'columns'. execute_batch_purger (repositories/base/purger.py) derives the target table from build_subquery().froms[0], assuming a sa.Table. DeploymentRevisionRow uses lazy="joined", so the compiled FROM is an _ORMJoin whose primary_key is a ColumnSet. Because the sweep is atomic per tick, this one broken category rolls back the entire tick, so nothing is ever purged under the default seed. Fix: resolve the table from the mapped entity (column_descriptions -> inspect().local_table) instead of the FROM clause.

(2) [CLI] mgr clear-history crashes on the forced sweep with InvalidRequestError: mapper KernelRow ... 'ImageRow' failed to locate a name. The standalone CLI process issues an ORM query before all models are imported. Redis stat cleanup runs but the sweep and VACUUM never execute. Fix: call ensure_all_tables_registered() before the sweep in _force_retention_sweep (cli/__main__.py).

(3) [setup] mgr schema oneshot does not create retention_policies. models/retention/__init__.py is empty (per the new no-reexport rule), so ensure_all_tables_registered() never registers the Row. The alembic upgrade path is fine. Fix: have ensure_all_tables_registered() also import each subpackage's .row module (models/base.py).

Additional hardening (approved): isolate each category in the sweep with a savepoint + try/except so a single category's failure no longer aborts the whole tick, mirroring the existing bulk_create_partial pattern.

Found via live-test of Epic BA-6924 / BEP-1063.

JIRA Issue: BA-6952

Contributor guide

Open the contributing guide

Research direction

Start with repositories/base/purger.py and inspect how execute_batch_purger resolves the mapped entity, then read cli/__main__.py and models/base.py with models/retention/.row. Verify the default-seed retention sweep handles joined mappings, the forced CLI sweep registers all models, and the schema oneshot creates retention_policies. The sweep should isolate category failures so one error does not abort the tick.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sqlalchemy
Domain
backend, cli, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.