Old script health checks come back when upgrading a cluster
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
When upgrading a Nomad client in a cluster from 0.9.4 to 0.10.1, we see allocations end up with two script checks registered in Consul against them - the new one which passes, and an old one which fails. Allocations running or started on the 0.9.4 nodes in the cluster continue to function normally.
### Nomad version
All server nodes: Nomad v0.10.1 (829f9af35c77d564b3dab74454eeba9bf25e2df8)
Some client nodes: Nomad v0.10.1 (829f9af35c77d564b3dab74454eeba9bf25e2df8)
Remaining client nodes: Nomad v0.9.4 (a81aa846a45fb8248551b12616287cb57c418cd6)
### Operating system and Environment details
Debian Jessie
### Issue
Old script-based health checks come back, a problem similar to https://github.com/hashicorp/nomad/issues/5824. It seems to only happen with script checks.
### Reproduction steps
Start with a cluster running 0.9.4, with a number of jobs with health check scripts.
Upgrade the server nodes to 0.10.1
Drain a client node
Upgrade nomad on the drained client node to 0.10.1
Restart the nomad job to allow it to run on the newly upgraded node
Wait a few minutes and watch the old script health checks get re-registered against the new allocations.
I can't replicate this in development mode with a single combined server/client, but it reliably happens on our clusters.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the stated upgrade sequence from Nomad 0.9.4 to 0.10.1, including draining and restarting a client, while observing script-check registration in Consul. Compare the handling of allocations and checks across old and new clients; done means old script checks are not re-registered against new allocations after the upgrade.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100