hashicorp / hashicorp/nomad

"check_restart.grace" is firing early for "group.service.check" that have a task defined

Open
#25,318 1 comment 0 reactions 0 assignees View on GitHub
hcc/jira stage/accepted theme/client theme/service-discovery type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

When using a `service` inside a `group` stanza, the `check.check_restart.grace` timer starts as soon as the allocation is created, while the workloads are still in the "starting" phase. This behavior works as intended, but there are some inconsistencies when a task is provided for that service

Users can define a `task` in either the [service](https://developer.hashicorp.com/nomad/docs/job-specification/service#task-1) or [check](https://developer.hashicorp.com/nomad/docs/job-specification/check#task) stanza (by default, the `check` inherits the value from `service.task` if defined). The problem is that if a task benefits from a sidecar or an init container, the grace period starts for the overall allocation rather than waiting for the specific task to start

**What is happening:**
- The grace timer starts at allocation creation, not after the task that the check is designed for
- For tasks that use a sidecar or an init container, this means the grace timer is already counting down even before the main task is still waiting for its lifecycle phase

**Expected Behavior:**
If the `service` or `check` stanza defines a `task` attribute, Nomad should wait for the "task started" event from that task before starting the grace timer

**Additional Notes:**
- The issue occurs with both Nomad and Consul service providers
- When the service is defined inside a `task` stanza, the grace period works as intended, starting after the "task started" event
- For services using the Consul provider with "consul-connect", the sidecar container may have a slow start/warmup, resulting in inconsistent grace periods across replicas and occasional premature restarts

**Steps to Reproduce:**
1. Deploy the job definition below
2. By using the UI, observe that Nomad fires a restart event for the `service.task` before the task has started, as it waits for the init container lifecycle to complete

### Job file
```hcl
# sandbox.nomad.hcl

job "sandbox" {
datacenters = ["dc1"]

group "hashicorp" {
network {
mode = "bridge"

port "api-http" {
to = 5678
}
}

service {
provider = "nomad"
port = "api-http"

check {
task = "echo"
type = "http"
path = "/health"
port = "api-http"
interval = "2s"
timeout = "3s"

check_restart {
limit = 5
grace = "10s"
}
}
}

task "sleep" {
driver = "docker"

lifecycle {
hook = "prestart"
}

config {
image = "docker.io/library/alpine:latest"
command = "sleep"
args = ["30"]
}
}

task "echo" {
driver = "docker"

config {
image = "docker.io/hashicorp/http-echo:latest"
args = ["-text=☕"]
}
}
}
}
```

---

### Environment information
Nomad version (client)
Nomad v1.9.6
BuildDate 2025-02-11T18:55:10Z
Revision https://github.com/hashicorp/nomad/commit/7f8b44963d36d025520348d7f24735774d26f13b+CHANGES

Nomad version (server)
Nomad v1.9.5
BuildDate 2025-01-14T18:35:12Z
Revision https://github.com/hashicorp/nomad/commit/0b7bb8b60758981dae2a78a0946742e09f8316f5+CHANGES

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the behavior with the provided sandbox.nomad.hcl job and observing the task-started event and restart timing for the group service. Trace the handling of service and check task attributes for both Nomad and Consul providers. Done means the grace timer waits for the configured task to start, including when sidecars or init containers delay that task.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
devops, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.