Basekick-Labs / Basekick-Labs/arc
WORM storage: S3 Object Lock / Azure immutability support
- Dominant language
- Go
- Stars
- 677
- Forks
- 53
- Avg merge
- 9h 14m
- Merged PRs (30d)
- 164
Description
## Context
Regulated-industry evaluations of Arc (aerospace / AMS2750, heat treat / CQI-9, pharma / 21 CFR Part 11) consistently land on the same requirement: the stored record must be **provably unalterable**, not merely "we don't expose an UPDATE endpoint."
Arc's current position is decent but incomplete:
- Parquet files are immutable — altering a record means rewriting the file
- There is no UPDATE path in the write API
- Deletes are off by default and must be explicitly enabled
What's missing is **storage-level enforcement**. Today, an operator with credentials to the bucket can delete or replace an Arc Parquet file and Arc has no way to prevent it or notice. S3 Object Lock in Compliance Mode closes this at the storage layer: not even the account root can delete a locked object before its retention expires.
This is worth building specifically because it is the mitigation regulated buyers *already ask for by name*. Arc is object-storage-native, so it should be a configuration surface rather than an engineering project — and being able to say "bucket policy, not custom code" is a strong differentiator against appliance-based competitors.
`grep -rniE "objectlock|object_lock|WORM|retain_until|legal.?hold" internal/` currently returns nothing.
## Goal
Let operators run Arc against a WORM-enabled bucket such that written Parquet files cannot be deleted or overwritten for a configured retention period, with Arc's own lifecycle operations behaving correctly against that constraint.
## Scope
**1. Write path**
Set Object Lock retention headers on PUT for data objects. New config, all needing `v.SetDefault()`:
```
storage.object_lock.enabled (bool, default false)
storage.object_lock.mode ("governance" | "compliance")
storage.object_lock.retain_days (int)
```
Understand the difference before implementing: **Governance** mode can be bypassed by a user with `s3:BypassGovernanceRetention`; **Compliance** mode cannot be bypassed by anyone, including the root account, until expiry. Only Compliance mode satisfies the regulatory ask. Governance is the safer default for operators experimenting — document the distinction prominently, because an operator who thinks they have Compliance and has Governance has no protection against exactly the insider threat they bought this for.
**2. Lifecycle operations must handle undeletable objects — this is the real work**
Arc's compaction, retention, and tiering paths currently assume deletes succeed. Against a locked bucket they will not. Every one of these needs auditing:
- `internal/compaction/` — compaction writes merged files and deletes sources. Against a locked bucket the delete fails and the source files remain. **What is the correct behavior?** Options: refuse to compact locked-and-unexpired files at all; compact but leave sources (accepting storage amplification); or make compaction WORM-aware and skip files still within retention. This needs a decision, not a guess.
- `internal/tiering/` — hot→cold migration deletes from the hot tier after copying to cold. Same question.
- `internal/retention/` — deleting data past its retention window will fail if lock retention outlives the data retention policy. Arc should detect this misconfiguration and refuse to start (or warn loudly) rather than failing every night at 2am.
- Raft manifest consistency — per the Cluster Operations Checklist in `.claude/CLAUDE.md`, manifest updates happen *before* storage deletes. If the storage delete then fails due to a lock, the manifest says the file is gone but it is still there. Trace this carefully.
**Failures must be loud.** A silently-failing delete that leaves orphans is worse than a refused operation. Decide on the semantics and document them.
**3. Azure equivalent**
Azure Blob immutability policies (time-based retention + legal hold) are the counterpart. Arc supports Azure storage, so this should not be S3-only. Reasonable to land S3 first and Azure as a follow-up — say which in the design.
**4. Legal hold (nice-to-have, propose separately if it grows)**
An indefinite hold on a measurement or time range, independent of retention expiry — for data under litigation or active investigation. Genuinely useful for the compliance story but a larger surface; feel free to scope it out of a first PR.
## Design questions to resolve before coding
- What does compaction do when its source files are locked? (the central question)
- How does Arc detect at startup that lock retention outlives data retention policy?
- Are checkpoints from the hash-chained-audit issue written to the same locked bucket? (probably yes, and that's the point)
- Does this need license gating as an enterprise feature, or is it OSS-appropriate?
- Migration/compatibility: what happens when an existing non-locked bucket has locking turned on mid-life?
## Acceptance criteria
- [ ] Design written up in `docs/progress/` and agreed **before** implementation
- [ ] Object Lock headers set on write when enabled
- [ ] Compaction, tiering, and retention behave correctly and observably against a locked bucket
- [ ] Manifest/storage consistency preserved when a storage delete is refused
- [ ] Startup validation for retention-vs-lock misconfiguration
- [ ] Every new config key has `v.SetDefault()` in `internal/config/config.go`
- [ ] Docs cover Governance vs Compliance mode explicitly, with the bypass caveat
- [ ] Release notes updated
## Notes for contributors
Read `.claude/CLAUDE.md` before starting — especially the **configuration matrix** section of the review process and the Cluster Operations Checklist.
This change is a textbook case for the matrix: it introduces new config keys, and the interesting bugs live in cells like *"cluster + compaction + object_lock.enabled=true"* where an independently-enabled subsystem meets an undeletable object. Please build that matrix as part of the PR — a run with `object_lock.enabled` left at its default proves nothing here.
Testing note: MinIO supports Object Lock and is suitable for integration testing without an AWS bill.
**Please discuss the design in this issue before writing code** — particularly the compaction-against-locked-files question, which changes the shape of the whole PR.
Related: companion issue on tamper-evident hash-chained audit log.
Contributor guide
Research direction
Read .claude/CLAUDE.md first, especially the configuration matrix and Cluster Operations Checklist, then inspect internal/config/config.go and the internal/compaction, internal/tiering, and internal/retention paths. Trace manifest updates around storage deletes and use MinIO for Object Lock experiments; done begins with an agreed design in docs/progress covering lifecycle semantics, validation, observability, and the stated acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, azure, go
- Domain
- backend, cloud, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100