"check_restart.grace" is firing early for "group.service.check" that have a task defined
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
When using a `service` inside a `group` stanza, the `check.check_restart.grace` timer starts as soon as the allocation is created, while the workloads are still in the "starting" phase. This behavior works as intended, but there are some inconsistencies when a task is provided for that service
Users can define a `task` in either the [service](https://developer.hashicorp.com/nomad/docs/job-specification/service#task-1) or [check](https://developer.hashicorp.com/nomad/docs/job-specification/check#task) stanza (by default, the `check` inherits the value from `service.task` if defined). The problem is that if a task benefits from a sidecar or an init container, the grace period starts for the overall allocation rather than waiting for the specific task to start
**What is happening:**
- The grace timer starts at allocation creation, not after the task that the check is designed for
- For tasks that use a sidecar or an init container, this means the grace timer is already counting down even before the main task is still waiting for its lifecycle phase
**Expected Behavior:**
If the `service` or `check` stanza defines a `task` attribute, Nomad should wait for the "task started" event from that task before starting the grace timer
**Additional Notes:**
- The issue occurs with both Nomad and Consul service providers
- When the service is defined inside a `task` stanza, the grace period works as intended, starting after the "task started" event
- For services using the Consul provider with "consul-connect", the sidecar container may have a slow start/warmup, resulting in inconsistent grace periods across replicas and occasional premature restarts
**Steps to Reproduce:**
1. Deploy the job definition below
2. By using the UI, observe that Nomad fires a restart event for the `service.task` before the task has started, as it waits for the init container lifecycle to complete
### Job file
```hcl
# sandbox.nomad.hcl
job "sandbox" {
datacenters = ["dc1"]
group "hashicorp" {
network {
mode = "bridge"
port "api-http" {
to = 5678
}
}
service {
provider = "nomad"
port = "api-http"
check {
task = "echo"
type = "http"
path = "/health"
port = "api-http"
interval = "2s"
timeout = "3s"
check_restart {
limit = 5
grace = "10s"
}
}
}
task "sleep" {
driver = "docker"
lifecycle {
hook = "prestart"
}
config {
image = "docker.io/library/alpine:latest"
command = "sleep"
args = ["30"]
}
}
task "echo" {
driver = "docker"
config {
image = "docker.io/hashicorp/http-echo:latest"
args = ["-text=☕"]
}
}
}
}
```
---
### Environment information
Nomad version (client)
Nomad v1.9.6
BuildDate 2025-02-11T18:55:10Z
Revision https://github.com/hashicorp/nomad/commit/7f8b44963d36d025520348d7f24735774d26f13b+CHANGES
Nomad version (server)
Nomad v1.9.5
BuildDate 2025-01-14T18:35:12Z
Revision https://github.com/hashicorp/nomad/commit/0b7bb8b60758981dae2a78a0946742e09f8316f5+CHANGES
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the behavior with the provided sandbox.nomad.hcl job and observing the task-started event and restart timing for the group service. Trace the handling of service and check task attributes for both Nomad and Consul providers. Done means the grace timer waits for the configured task to start, including when sidecars or init containers delay that task.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100