Ease persisted Raft data schema modification
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
When a struct is persisted in Raft, we must support its schema "forever". Consider `Allocation.TaskResources` field migration to `Allocation.AllocatedResources.Task`, we effectively cannot drop `Allocation.TaskResources` ever from the allocation struct. If an allocation is created in a Nomad 0.8 (pre AllocatedResources intro) cluster, and the cluster is upgraded in rapid secession to Nomad 0.9 then 0.10, ... then 0.N; at version 0.N we aren't assured that the persisted Raft representation `AllocatedResources` is ever populated, and thus always need to fallback to reading TaskResources fields.
We can guarantee that a field is no longer needed, if we guarantee that Raft Snapshot and Client Store data have been migrate to latest schema version
Next Steps:
* Document the backward compatibility requirement and approaches for persisted data structures for engineers
* Consider some approaches to force a snapshot during an upgrade process. Some ideas:
* force taking snapshot after an upgrade; add an Force Snapshot api/cli endpoint to be called during an upgrade
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the persisted Allocation schema, including the migration from Allocation.TaskResources to Allocation.AllocatedResources.Task, and the Raft Snapshot and Client Store data paths. Evaluate the proposed upgrade-time snapshot and Force Snapshot API/CLI approaches. Done means the backward-compatibility requirement is documented and a clear approach for migrating persisted data or forcing snapshots is agreed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100