hashicorp / hashicorp/nomad

CSI volumes can fail for `poststop` tasks for a stopped job after client restart

Open
#16,746 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stage/needs-discussion theme/storage theme/task lifecycle type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

In https://github.com/hashicorp/nomad/pull/16722 we've fixed a crash described in detail in https://github.com/hashicorp/nomad/issues/16623#issuecomment-1492415089, but the crash fix still leaves poststop tasks in a failed state under the following circumstances:

  • a job has a poststop task that wants a CSI volume
  • the client stops
  • the server stops the job
  • the client starts again
  • the client restores the allocrunner but the client receives the stop from GetClientAllocs before the allocrunner hits the Prerun alloc hooks.
  • this will result in shouldRun returning false, which prevents the CSI hook from running.
  • but the poststop task is still run, which means its volume hook doesn't have access to any info from the CSI hook, so the poststop task fails.

It would be nice if we could serialize client startup such that GetClientAllocs happens before we restore the state, because that would at least enforce a reasonable workflow, but that would also cause the bug to always happen instead of just some of the time. Do we even want to guarantee that poststop tasks to survive across client restarts in this way? Are there other lurking cases of this scenario in other hooks that we just haven't found yet?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with client/allocrunner/alloc_runner.go, especially shouldRun and the Prerun alloc hooks, then trace the CSI hook and poststop task handling described in the issue. Reproduce the client-restart sequence if possible and establish the intended behavior for poststop tasks and CSI volumes before defining tests and completion criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.