Implement retention policy for artifact storage to auto-delete expired artifacts
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Motivation
Backend.AI's artifact storage manages models, datasets, and other large artifacts imported from registries such as HuggingFace and Reservoir. Over time, the volume of stored artifacts grows continuously as new model revisions are imported, while older or unused versions remain indefinitely. Currently, cleanup is entirely manual — administrators or users must explicitly trigger `cleanupArtifactRevisions` via the GraphQL API or REST endpoint (`POST /revisions/cleanup`) to remove artifact files and free storage space.
This lack of automated lifecycle management creates several problems:
- **Uncontrolled storage growth**: Large model files (often multi-GB) accumulate without bound, leading to storage exhaustion.
- **Operational burden**: Administrators must manually identify and clean up stale artifacts, which is error-prone and time-consuming.
- **Cost inefficiency**: In cloud/object storage environments, retaining unused artifacts results in unnecessary storage costs.
- **No policy enforcement**: There is no mechanism to enforce organization-wide data retention compliance or storage governance.
A retention policy feature would allow administrators to define time-based rules for automatically deleting or cleaning up artifacts whose retention period has expired, reducing operational overhead and ensuring predictable storage usage.
----
## Required Features
- **Retention policy configuration**: Allow administrators to define retention policies at multiple granularity levels (global, per-project, per-artifact-type). Each policy should specify a retention period (e.g., 30 days, 90 days, 1 year) after which artifact revisions become eligible for automatic deletion.
- **Policy assignment**: Support assigning retention policies to artifact registries, individual artifacts, or artifact revisions. More specific policies should override broader ones (revision > artifact > project > global).
- **Retention period tracking**: Leverage existing `artifact_revisions.created_at` and `artifact_revisions.updated_at` timestamps to determine artifact age. Consider whether "last accessed" time should also factor into retention decisions.
- **Automated cleanup scheduler**: Implement a periodic background task (e.g., via the manager's event dispatcher or a dedicated scheduler) that scans for expired artifact revisions and triggers cleanup. This should reuse the existing `CleanupArtifactRevisionAction` pipeline where possible.
- **Retention exemptions**: Provide a mechanism to exempt specific artifact revisions from retention policies (e.g., pinned/starred models, or revisions in PULLING/VERIFYING intermediate states).
- **Dry-run / preview mode**: Before actual deletion, allow administrators to preview which artifacts would be affected by a retention policy.
- **Audit logging**: Record all retention-based deletions in the audit log, including which policy triggered the deletion, the artifact details, and the timestamp.
- **API and UI support**: Expose retention policy CRUD operations via both REST and GraphQL APIs. Provide GraphQL queries to view policy assignments and upcoming expirations.
- **Notification support**: Optionally notify artifact owners before their artifacts are scheduled for deletion (e.g., 7 days before expiration).
----
## Impact
- **Database schema** (`models/artifact/`, `models/artifact_revision/`): New table(s) for retention policies and potentially new columns on existing artifact tables (e.g., `retention_policy_id`, `expires_at`, `pinned`).
- **Service layer** (`services/artifact/`, `services/artifact_revision/`): New retention policy service and modifications to the existing cleanup service to support policy-driven deletion.
- **Repository layer** (`repositories/artifact/`): New queries for finding expired artifacts, retention policy CRUD, and batch cleanup operations.
- **API layer** (`api/artifact.py`, `api/gql/artifact/`): New REST and GraphQL endpoints for retention policy management.
- **Storage proxy** (`storage/services/artifacts/`): Indirectly affected — cleanup of expired artifacts triggers file deletion on storage backends (object storage, VFS).
- **Scheduler / background tasks**: New periodic task for scanning and enforcing retention policies.
- **Audit log** (`actions/monitors/audit_log.py`): New audit events for retention-based deletions.
- **Alembic migrations**: New migration(s) for retention policy tables and schema changes.
----
## Testing Scenarios
- **Policy CRUD**: Verify that retention policies can be created, read, updated, and deleted via both REST and GraphQL APIs.
- **Policy assignment**: Test assigning policies at global, project, and artifact levels, and verify that more specific policies correctly override broader ones.
- **Expiration detection**: Create artifact revisions with various ages and verify the scheduler correctly identifies those past the retention period.
- **Automated cleanup execution**: Confirm that expired artifact revisions are cleaned up automatically, including file deletion from storage backends and status reset to SCANNED.
- **Exemption handling**: Verify that pinned/starred artifacts and those in intermediate states (PULLING, VERIFYING) are not deleted even if past retention period.
- **Dry-run mode**: Test that preview mode correctly lists artifacts that would be deleted without actually removing them.
- **Audit trail**: Confirm that all retention-based deletions are properly recorded in the audit log with policy reference.
- **Edge cases**: Test behavior when retention policy is updated or deleted while artifacts are pending cleanup; verify no race conditions with concurrent manual cleanup and scheduled cleanup.
- **Notification**: Verify that owners receive notifications before artifact expiration when notification is enabled.
- **Storage quota interaction**: Ensure retention cleanup correctly updates storage usage metrics after artifact deletion.
----
## Expected Sub-Issues
- **DB schema design**: Add retention policy table and migration (alembic), add `expires_at` / `pinned` columns to artifact revision table.
- **Retention policy service layer**: Implement CRUD actions and processors for retention policies following the existing Action/Processor pattern.
- **Retention policy repository layer**: Implement queries for policy CRUD and expired artifact lookup.
- **REST and GraphQL API**: Expose retention policy endpoints and queries, including dry-run preview.
- **Background scheduler**: Implement periodic task that scans for expired revisions and triggers `CleanupArtifactRevisionAction`.
- **Exemption and pinning support**: Add pin/unpin functionality to protect specific revisions from auto-deletion.
- **Audit logging integration**: Extend audit log to capture retention-driven deletions.
- **Notification system integration**: Implement pre-expiration notifications for artifact owners.
- **Unit and integration tests**: Comprehensive test coverage for all new components.
JIRA Issue: BA-4383
Contributor guide
Assessment
This issue has not been assessed yet.