hashicorp / hashicorp/nomad

Client status pending on down node.

Open
#2,576 6 comments 0 reactions 0 assignees View on GitHub
theme/core type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v0.5.6

### Operating system and Environment details

3-node server cluster:

```
$ uname -a
Linux nomad-server-01 4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"
```

### Issue

I've shut down about 20 nomad client nodes. Most of them were cleaned up by the garbage collector, but a couple of them are not going away. Tried to force a GC with `/v1/system/gc` but it still won't go away.

Client nodes are/were Windows 2012 R2 Datacenter Edition on same nomad version.

### Reproduction steps

This is how it happened for me, but may not happen every time:

1. Create a large-ish cluster (30 nodes?)
2. Have a bunch of tasks running on the cluster - mostly allocated. (>60%)
3. Have [Hashi-UI](https://github.com/jippi/hashi-ui) running against the cluster (unsure if important)
4. Start to drain tasks off 1/2 of them
5. Shut the drained nodes down
6. Force a GC with `/v1/system/gc` (Optional)

### Nomad Server logs (if appropriate)

[nomad.log.zip](https://github.com/hashicorp/nomad/files/940086/nomad.log.zip)

Contributor guide

No contributing guide indexed for this repository

Research direction

Begin with the /v1/system/gc endpoint and the attached nomad.log.zip, then reproduce the reported sequence of draining and shutting down nodes in a large cluster. No source files or tests are named; done means identifying why some down clients remain pending and ensuring the reported stale clients are removed by the expected cleanup path.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.