hashicorp / hashicorp/nomad

Old script health checks come back when upgrading a cluster

Open
#6,815 10 comments 0 reactions 0 assignees View on GitHub
stage/needs-discussion theme/consul theme/service-discovery/consul type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

When upgrading a Nomad client in a cluster from 0.9.4 to 0.10.1, we see allocations end up with two script checks registered in Consul against them - the new one which passes, and an old one which fails. Allocations running or started on the 0.9.4 nodes in the cluster continue to function normally.

### Nomad version
All server nodes: Nomad v0.10.1 (829f9af35c77d564b3dab74454eeba9bf25e2df8)
Some client nodes: Nomad v0.10.1 (829f9af35c77d564b3dab74454eeba9bf25e2df8)
Remaining client nodes: Nomad v0.9.4 (a81aa846a45fb8248551b12616287cb57c418cd6)

### Operating system and Environment details
Debian Jessie

### Issue

Old script-based health checks come back, a problem similar to https://github.com/hashicorp/nomad/issues/5824. It seems to only happen with script checks.

### Reproduction steps

Start with a cluster running 0.9.4, with a number of jobs with health check scripts.
Upgrade the server nodes to 0.10.1
Drain a client node
Upgrade nomad on the drained client node to 0.10.1
Restart the nomad job to allow it to run on the newly upgraded node
Wait a few minutes and watch the old script health checks get re-registered against the new allocations.

I can't replicate this in development mode with a single combined server/client, but it reliably happens on our clusters.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the stated upgrade sequence from Nomad 0.9.4 to 0.10.1, including draining and restarting a client, while observing script-check registration in Consul. Compare the handling of allocations and checks across old and new clients; done means old script checks are not re-registered against new allocations after the upgrade.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devops, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.