hashicorp / hashicorp/consul

Consul catalog services disappears when underlying node goes away where client agent pod was running as daemonset

Open
#10,057 9 comments 0 reactions 0 assignees View on GitHub
theme/catalog theme/kubernetes type/docs type/question
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

we have external services that we bootstrap during the initial deployment. we have consul agents running on spot instances as a daemonset. so it's expected to see nodes going out time to time.

**consul version** 1.8

Steps to reproduce this issue, eg:

1. Register a service.
```
/# jq . -C abc.json
{
"services": [
{
"id": "storagecentral-msk-0",
"name": "storagecentral-msk",
"tags": [
"default"
],
"address": "XXX",
"port": 9094
},
{
"id": "storagecentral-msk-1",
"name": "storagecentral-msk",
"tags": [
"default"
],
"address": "XXXX",
"port": 9094
},
{
"id": "storagecentral-msk-2",
"name": "storagecentral-msk",
"tags": [
"default"
],
"address": "XXXXX",
"port": 9094
}
]
}
/ #
/ # consul services register abc.json
Registered service: storagecentral-msk
Registered service: storagecentral-msk
Registered service: storagecentral-msk
/ # consul reload
Configuration reload triggered
/ #
```
2. Check the Node which hosted these services , checked from UI under "services"
3. Remove the node from the cluster.

Result:
Not able to see those consul services anymore.

**Wanted to know**

1. Is this expected?
2. we want to persist custom bootstrap. how to do that ?

How can we troubleshoot further.

Contributor guide

Open the contributing guide

Research direction

Start with the registration payload in abc.json and the consul services register and consul reload commands, then observe the Consul UI after removing the hosting node. Check whether the disappearing services are tied to the daemonset agent and determine the persistence behavior for bootstrap registrations; done means explaining whether this is expected and how to retain them.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
devops, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.