Client status pending on down node.
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v0.5.6
### Operating system and Environment details
3-node server cluster:
```
$ uname -a
Linux nomad-server-01 4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"
```
### Issue
I've shut down about 20 nomad client nodes. Most of them were cleaned up by the garbage collector, but a couple of them are not going away. Tried to force a GC with `/v1/system/gc` but it still won't go away.
Client nodes are/were Windows 2012 R2 Datacenter Edition on same nomad version.
### Reproduction steps
This is how it happened for me, but may not happen every time:
1. Create a large-ish cluster (30 nodes?)
2. Have a bunch of tasks running on the cluster - mostly allocated. (>60%)
3. Have [Hashi-UI](https://github.com/jippi/hashi-ui) running against the cluster (unsure if important)
4. Start to drain tasks off 1/2 of them
5. Shut the drained nodes down
6. Force a GC with `/v1/system/gc` (Optional)
### Nomad Server logs (if appropriate)
[nomad.log.zip](https://github.com/hashicorp/nomad/files/940086/nomad.log.zip)
Contributor guide
No contributing guide indexed for this repository
Research direction
Begin with the /v1/system/gc endpoint and the attached nomad.log.zip, then reproduce the reported sequence of draining and shutting down nodes in a large cluster. No source files or tests are named; done means identifying why some down clients remain pending and ensuring the reported stale clients are removed by the expected cleanup path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100