task lifecycle: long running poststop task with dynamic template data can hang
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
1.3.3
### Operating system and Environment details
Ubuntu 20.04
3 Servers
3 Clients
Single Region
### Issue
A long running `poststop` task that references dynamic data can cause an allocation to never stop if the dynamic data keeps changing before the long-running `poststop` task can complete.
### Reproduction steps
First, create some sort of dynamic data that can be referenced by a job template. This example will use a Vault PKI secret. Notice the `max_ttl=15s` arg passed for `my-path/roles/nomad`.
```hcl
vault secrets enable -path=my-path pki
vault write my-path/root/generate/internal common_name=service.consul ttl=1h
# Notice the max_ttl below
vault write my-path/roles/nomad \
allowed_domains=service.consul \
allow_subdomains=true \
generate_lease=true \
max_ttl=15s
vault secrets tune -max-lease-ttl=1m my-path
# create secrets policy
vault policy write policy-my-path -<> "${NOMAD_ALLOC_DIR}/lifecycle"
while true; do
sleep 1
done
EOF
destination = "local/script.sh"
}
resources {
cpu = 50
memory = 50
}
}
task "poststop" {
driver = "raw_exec"
lifecycle {
hook = "poststop"
}
config {
command = "/bin/bash"
args = ["local/script.sh"]
}
template {
data = <> "${NOMAD_ALLOC_DIR}/lifecycle"
sleep 100
EOF
destination = "local/script.sh"
}
resources {
cpu = 50
memory = 50
}
}
}
}
```
Now run `job stop -purge` on the job, and watch the `poststop` task from the UI.
#### Expected Result
This works as designed, but it illustrates the need for a way to configure the task so that the allocation does not get hung trying to stop.
#### Actual Result
The task will continually restart every `15s` due to the `max_ttl` on the Vault secret causing the template to re-render which triggers a task restart. The `poststop` task will never finish, and the allocation will continue to run.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the issue with the provided Vault PKI setup, jobspec, and `job stop -purge`, then trace the poststop task lifecycle when dynamic template data triggers restarts. Done means a configurable behavior prevents the poststop task from continually restarting and allows the allocation to stop while preserving the intended template handling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100