CSI volumes can fail for `poststop` tasks for a stopped job after client restart
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
In https://github.com/hashicorp/nomad/pull/16722 we've fixed a crash described in detail in https://github.com/hashicorp/nomad/issues/16623#issuecomment-1492415089, but the crash fix still leaves poststop tasks in a failed state under the following circumstances:
- a job has a
poststoptask that wants a CSI volume - the client stops
- the server stops the job
- the client starts again
- the client restores the allocrunner but the client receives the stop from
GetClientAllocsbefore the allocrunner hits thePrerunalloc hooks. - this will result in
shouldRunreturning false, which prevents the CSI hook from running. - but the
poststoptask is still run, which means its volume hook doesn't have access to any info from the CSI hook, so thepoststoptask fails.
It would be nice if we could serialize client startup such that GetClientAllocs happens before we restore the state, because that would at least enforce a reasonable workflow, but that would also cause the bug to always happen instead of just some of the time. Do we even want to guarantee that poststop tasks to survive across client restarts in this way? Are there other lurking cases of this scenario in other hooks that we just haven't found yet?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with client/allocrunner/alloc_runner.go, especially shouldRun and the Prerun alloc hooks, then trace the CSI hook and poststop task handling described in the issue. Reproduce the client-restart sequence if possible and establish the intended behavior for poststop tasks and CSI volumes before defining tests and completion criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100