hashicorp / hashicorp/consul

Zombie agent trapped in catalog

Open
#10,848 4 comments 0 reactions 0 assignees View on GitHub
theme/operator-usability type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.

#### Overview of the Issue

I am seeing a lot of errors like this in my server logs:

2021-08-13T10:43:35.033Z [INFO] agent.server.snapshot: creating new snapshot: path=data\raft\snapshots\101973-54331185-1628851415033.tmp
2021-08-13T10:43:35.050Z [INFO] agent.server.snapshot: reaping snapshot: path=data\raft\snapshots\101973-54331185-1628851394441
2021-08-13T10:43:35.051Z [INFO] agent.server.raft: copied to local snapshot: bytes=579416
2021-08-13T10:43:35.055Z [ERROR] agent.server.raft: failed to restore snapshot: error="failed to restore snapshot 101973-54331185-1628851415033: check node "EW1A-TST-MPC" does not match node "ew1a-tst-mpc""
2021-08-13T10:43:35.862Z [ERROR] agent.anti_entropy: failed to sync remote state: error="ACL not found"

and on others:

Aug 13 10:08:05 ip-10-142-20-109 consul[1944]: 2021-08-13T10:08:05.944Z [WARN] agent.fsm: DeleteNode failed: error="failed generating events from changes: Missing node registration"

On agents I see:

2021-08-13T10:43:35.862Z [ERROR] agent.anti_entropy: failed to sync remote state: error="ACL not found"

The errors and warnings just repeat. I have tried force-leave and -prune but they report the node doesnt exist.

#### Reproduction Steps

I believe what has happened is:

- a node registered itself EW1A-TST-MPC
- this should be lower case so it was re-registered via config with node-name ew1a-tst-mpc
- that instance was then destroyed
- in the period before consul would have cleaned it up, its agent token was removed
- when consul went to remove the node it wasnt able to complete the process

Now its left in this "zombie" state where its too dead for force-leave to work but not dead enough to get cleaned up

### Consul info for both Client and Server

1.9,8 windows and linux servers

Contributor guide

Open the contributing guide

Research direction

No source file or test is named. Start by examining the Consul agent logs around snapshot restoration, anti-entropy ACL sync, and FSM DeleteNode failures, then reproduce the sequence involving case-changed node names and force-leave or -prune. Done means the zombie node is cleaned up and the repeated errors stop.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.