Basekick-Labs / Basekick-Labs/arc

Retention policies should apply to cold tier storage

Open
#168 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
677
Forks
53
Avg merge
9h 14m
Merged PRs (30d)
164

Description

## Summary

Retention policies currently only delete files from the hot tier storage backend. When tiered storage is enabled, files that have been migrated to cold tier are never deleted by retention policies, potentially causing unbounded storage growth in cold storage.

## Current Behavior

- `RetentionHandler` only has a single `storage.Backend` (hot tier)
- When retention executes, it only scans and deletes from hot tier
- Files in cold tier are never affected by retention policies

## Expected Behavior

When tiered storage is enabled, retention policies should:
1. Delete expired files from hot tier (current behavior)
2. Also delete expired files from cold tier
3. Update tiering metadata when cold files are deleted

## Implementation Notes

1. Add `tieringManager` field to `RetentionHandler`
2. Wire tiering manager in `cmd/arc/main.go` via `SetTieringManager()`
3. When executing retention:
- Query tiering metadata for cold tier files older than retention threshold
- Delete those files from cold storage backend
- Remove entries from `tier_files` table

### Considerations

- **Cost**: Deleting from Glacier/Archive is cheap (no retrieval needed)
- **Validation**: Ensure `retention_days > tiering_threshold_days` to avoid deleting data before it migrates
- **Metrics**: Track cold tier deletions separately in retention stats

## Files to Modify

- `internal/api/retention.go` - Add tiering manager, extend execution logic
- `cmd/arc/main.go` - Wire tiering manager to retention handler
- `internal/tiering/metadata.go` - May need `GetFilesOlderThan()` filtered by tier

## Related

- Part of tiered storage feature (Enterprise)
- Related to #166 (tiered storage query routing)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.