hashicorp / hashicorp/nomad

task lifecycle: long running poststop task with dynamic template data can hang

Open
#14,299 0 comments 0 reactions 0 assignees View on GitHub
theme/task lifecycle type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version

1.3.3

### Operating system and Environment details

Ubuntu 20.04
3 Servers
3 Clients
Single Region

### Issue

A long running `poststop` task that references dynamic data can cause an allocation to never stop if the dynamic data keeps changing before the long-running `poststop` task can complete.

### Reproduction steps

First, create some sort of dynamic data that can be referenced by a job template. This example will use a Vault PKI secret. Notice the `max_ttl=15s` arg passed for `my-path/roles/nomad`.

```hcl
vault secrets enable -path=my-path pki

vault write my-path/root/generate/internal common_name=service.consul ttl=1h

# Notice the max_ttl below
vault write my-path/roles/nomad \
allowed_domains=service.consul \
allow_subdomains=true \
generate_lease=true \
max_ttl=15s

vault secrets tune -max-lease-ttl=1m my-path

# create secrets policy
vault policy write policy-my-path -<> "${NOMAD_ALLOC_DIR}/lifecycle"
while true; do
sleep 1
done
EOF
destination = "local/script.sh"
}

resources {
cpu = 50
memory = 50
}
}

task "poststop" {
driver = "raw_exec"

lifecycle {
hook = "poststop"
}

config {
command = "/bin/bash"
args = ["local/script.sh"]
}

template {
data = <> "${NOMAD_ALLOC_DIR}/lifecycle"
sleep 100
EOF
destination = "local/script.sh"
}

resources {
cpu = 50
memory = 50
}
}
}
}
```

Now run `job stop -purge` on the job, and watch the `poststop` task from the UI.

#### Expected Result

This works as designed, but it illustrates the need for a way to configure the task so that the allocation does not get hung trying to stop.

#### Actual Result

The task will continually restart every `15s` due to the `max_ttl` on the Vault secret causing the template to re-render which triggers a task restart. The `poststop` task will never finish, and the allocation will continue to run.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the issue with the provided Vault PKI setup, jobspec, and `job stop -purge`, then trace the poststop task lifecycle when dynamic template data triggers restarts. Done means a configurable behavior prevents the poststop task from continually restarting and allows the allocation to stop while preserving the intended template handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devops, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.