Basekick-Labs / Basekick-Labs/arc
Retention policies should apply to cold tier storage
- Dominant language
- Go
- Stars
- 677
- Forks
- 53
- Avg merge
- 9h 14m
- Merged PRs (30d)
- 164
Description
## Summary
Retention policies currently only delete files from the hot tier storage backend. When tiered storage is enabled, files that have been migrated to cold tier are never deleted by retention policies, potentially causing unbounded storage growth in cold storage.
## Current Behavior
- `RetentionHandler` only has a single `storage.Backend` (hot tier)
- When retention executes, it only scans and deletes from hot tier
- Files in cold tier are never affected by retention policies
## Expected Behavior
When tiered storage is enabled, retention policies should:
1. Delete expired files from hot tier (current behavior)
2. Also delete expired files from cold tier
3. Update tiering metadata when cold files are deleted
## Implementation Notes
1. Add `tieringManager` field to `RetentionHandler`
2. Wire tiering manager in `cmd/arc/main.go` via `SetTieringManager()`
3. When executing retention:
- Query tiering metadata for cold tier files older than retention threshold
- Delete those files from cold storage backend
- Remove entries from `tier_files` table
### Considerations
- **Cost**: Deleting from Glacier/Archive is cheap (no retrieval needed)
- **Validation**: Ensure `retention_days > tiering_threshold_days` to avoid deleting data before it migrates
- **Metrics**: Track cold tier deletions separately in retention stats
## Files to Modify
- `internal/api/retention.go` - Add tiering manager, extend execution logic
- `cmd/arc/main.go` - Wire tiering manager to retention handler
- `internal/tiering/metadata.go` - May need `GetFilesOlderThan()` filtered by tier
## Related
- Part of tiered storage feature (Enterprise)
- Related to #166 (tiered storage query routing)
Contributor guide
Assessment
This issue has not been assessed yet.