Zombie agent trapped in catalog
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.
#### Overview of the Issue
I am seeing a lot of errors like this in my server logs:
2021-08-13T10:43:35.033Z [INFO] agent.server.snapshot: creating new snapshot: path=data\raft\snapshots\101973-54331185-1628851415033.tmp
2021-08-13T10:43:35.050Z [INFO] agent.server.snapshot: reaping snapshot: path=data\raft\snapshots\101973-54331185-1628851394441
2021-08-13T10:43:35.051Z [INFO] agent.server.raft: copied to local snapshot: bytes=579416
2021-08-13T10:43:35.055Z [ERROR] agent.server.raft: failed to restore snapshot: error="failed to restore snapshot 101973-54331185-1628851415033: check node "EW1A-TST-MPC" does not match node "ew1a-tst-mpc""
2021-08-13T10:43:35.862Z [ERROR] agent.anti_entropy: failed to sync remote state: error="ACL not found"
and on others:
Aug 13 10:08:05 ip-10-142-20-109 consul[1944]: 2021-08-13T10:08:05.944Z [WARN] agent.fsm: DeleteNode failed: error="failed generating events from changes: Missing node registration"
On agents I see:
2021-08-13T10:43:35.862Z [ERROR] agent.anti_entropy: failed to sync remote state: error="ACL not found"
The errors and warnings just repeat. I have tried force-leave and -prune but they report the node doesnt exist.
#### Reproduction Steps
I believe what has happened is:
- a node registered itself EW1A-TST-MPC
- this should be lower case so it was re-registered via config with node-name ew1a-tst-mpc
- that instance was then destroyed
- in the period before consul would have cleaned it up, its agent token was removed
- when consul went to remove the node it wasnt able to complete the process
Now its left in this "zombie" state where its too dead for force-leave to work but not dead enough to get cleaned up
### Consul info for both Client and Server
1.9,8 windows and linux servers
Contributor guide
Research direction
No source file or test is named. Start by examining the Consul agent logs around snapshot restoration, anti-entropy ACL sync, and FSM DeleteNode failures, then reproduce the sequence involving case-changed node names and force-leave or -prune. Done means the zombie node is cleaned up and the repeated errors stop.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100