Active Vault DNS is incorrect after the active Vault died
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Overview of the Issue
Vault 1.0.1
Consul 1.4.2
After the active Vault died, Consul DNS resolves active.vault.service.consulto 2 entries: one points to the died vault instance, the other points to the new active vault instance.
Expected behaviour: Consul DNS should only resolve active.vault.service.consul to only one entry, which is the current active vault instance.
#### Reproduction Steps
1. Create a Consul cluster with 3 server nodes. Setup a Vault cluster in these nodes that have Consul backend storage.
Assume these 3 nodes IP are 10.35.1.10 (1), 10.35.2.10 (2), 10.35.3.10 (3); (1) is the active Vault node.
2. Stimulate a failure scenario in the active Vault node (1) by blocking all ports for Consul and Vault
> iptables -A INPUT -p tcp --destination-port 8300 -j DROP
> iptables -A OUTPUT -p tcp --destination-port 8300 -j DROP
> iptables -A INPUT -p tcp --destination-port 8301 -j DROP
> iptables -A OUTPUT -p tcp --destination-port 8301 -j DROP
> iptables -A INPUT -p tcp --destination-port 8302 -j DROP
> iptables -A OUTPUT -p tcp --destination-port 8302 -j DROP
> iptables -A INPUT -p tcp --destination-port 8600 -j DROP
> iptables -A OUTPUT -p tcp --destination-port 8600 -j DROP
> iptables -A INPUT -p tcp --destination-port 53 -j DROP
> iptables -A OUTPUT -p tcp --destination-port 53 -j DROP
> iptables -A INPUT -p tcp --destination-port 8500 -j DROP
> iptables -A OUTPUT -p tcp --destination-port 8500 -j DROP
> iptables -A INPUT -p tcp --destination-port 8200 -j DROP
> iptables -A OUTPUT -p tcp --destination-port 8200 -j DROP
> iptables -A INPUT -p tcp --destination-port 8201 -j DROP
> iptables -A OUTPUT -p tcp --destination-port 8201 -j DROP
> iptables-save
3. Wait for a minute for the new active Vault node to be elected. Assume this node is (2)
4. Try DNS lookup active.vault.service.consul, you will see it resolves to both (1) and (2)
`dig @10.35.2.10 8600 active.vault.service.consul`
### More information
1. When checking Consul catalog service for Vault, there are 2 nodes which have ServiceTags is "active"
2. When trying DNS lookup for active.vault.service.consul in node (1) `dig @10.35.1.10 8600 active.vault.service.consul`, it resolves to node (1) only
3. I tried to "fix" the DNS issue by removing the faulty vault node from Consul by using `consul force-leave` and `consul operator raft remove-peer`, neither of them removed that node from Consul nodes or Raft peers. However, shutting down that node helped to "fix" it.
Contributor guide
Assessment
This issue has not been assessed yet.