Session with empty `NodeChecks` created from Consul Server is immediately lost if Consul Server loss network
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
### Overview of the Issue
If session is created from Consul Server with empty `NodeChecks`, and this Consul Server loss network, this session is immediately deleted. There is no such issue if session is created from Consul Client.
### Details
[Session documentation](https://developer.hashicorp.com/consul/docs/dynamic-app-config/sessions):
```
The contract that Consul provides is that under any of the following situations,
the session will be invalidated:
— Node is deregistered
— Any of the health checks are deregistered
— Any of the health checks go to the critical state
— Session is explicitly destroyed
— TTL expires, if applicable
```
None of this applies: node is not deregistered yet (just short network issues for ~5 s) and health checks are not failed (default `serfHealth` is not applied).
Also, this works correctly as documented if test is done from Consul Client, and session is not lost.
This case is important if we build pattern [Application leader election with sessions](https://developer.hashicorp.com/consul/tutorials/developer-configuration/application-leader-elections) on 3 servers. For example, place Consul Server and [Patroni](https://github.com/zalando/patroni) on each of 3 servers. It is expected that if connection is lost, the session will not be deleted immediately, but only after TTL is expired.
Why behavior differs if session is created from Consul Server?
How to improve this behavior to respect specified session TTL?
---
### Reproduction Steps
1. Create a cluster with 3 Consul Servers.
2. Create a session from Consul Server 1:
```
curl -X PUT -d '{"Name": "testlock", "NodeChecks": [], "TTL": "5m"}' http://localhost:8500/v1/session/create
{"ID":"db870c93-b8d8-688a-a763-e259f0409be3"}
```
3. Check session:
```
Thu 1 Feb 00:20:57 2024
curl -s http://localhost:8500/v1/session/info/db870c93-b8d8-688a-a763-e259f0409be3
[{"ID":"db870c93-b8d8-688a-a763-e259f0409be3","Name":"testlock","Node":"db4",
"LockDelay":15000000000,"Behavior":"release","TTL":"5m",
"NodeChecks":[],"ServiceChecks":null,"CreateIndex":11465749,
"ModifyIndex":11465749}]
```
4. Disconnect [network](https://developer.hashicorp.com/consul/docs/install/ports) on Consul Server 1:
```
Thu 1 Feb 00:21:04 2024
PORTS="8300,8301,8302,8500,8503,8600" &&
iptables-legacy -A INPUT -p tcp -m multiport --dports $PORTS -j DROP &&
iptables-legacy -A INPUT -p udp -m multiport --dports $PORTS -j DROP &&
iptables-legacy -A OUTPUT -p tcp -m multiport --dports $PORTS -j DROP &&
iptables-legacy -A OUTPUT -p udp -m multiport --dports $PORTS -j DROP
```
5. On Consul Server 2 check session — session is lost immediately:
```
Thu 1 Feb 00:21:22 2024
curl -s http://localhost:8500/v1/session/info/db870c93-b8d8-688a-a763-e259f0409be3
[]
```
### Consul info for both Client and Server
Server info
consul info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 7736539d
version = 1.17.2
version_metadata =
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = xxx:8300
server = true
raft:
applied_index = 11466634
commit_index = 11466634
fsm_pending = 0
last_contact = 29.711728ms
last_log_index = 11466634
last_log_term = 5883
last_snapshot_index = 11464290
last_snapshot_term = 5836
latest_configuration = [{Suffrage:Voter ID:eb490abe-4e5b-b953-154e-399bbd643be8 Address:xxxx:8300} {Suffrage:Voter ID:21e95163-2318-e3db-c9da-b0a621a1b3de Address:xxxx:8300} {Suffrage:Voter ID:913ffc1e-68c1-481f-ca7e-bd55d90ed79b Address:xxxx:8300} {Suffrage:Voter ID:54b5021d-2f7c-cd07-7da8-f4ad2dbb406a Address:xxxx:8300}]
latest_configuration_index = 0
num_peers = 3
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 5883
runtime:
arch = amd64
cpu_count = 4
goroutines = 161
max_procs = 4
os = linux
version = go1.21.6
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 227
failed = 0
health_score = 0
intent_queue = 0
left = 1
member_time = 21614
members = 6
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 1
member_time = 7598
members = 5
query_queue = 0
query_time = 1
```
Client agent HCL config:
```
data_dir = "/var/lib/consul"
server = true
bootstrap_expect = 3
bind_addr = "xxx"
retry_join = [
"xxx",
"xxx",
"xxx",
]
encrypt = "xxx"
```
### Operating system and Environment details
Ubuntu 22 LTS
### Log Fragments
```
Feb 01 00:21:05 db3 consul[3199803]: 2024-02-01T00:21:05.978+0300 [WARN] agent: error getting server health from server: server=db4 error="context deadline exceeded"
Feb 01 00:21:07 db3 consul[3199803]: 2024-02-01T00:21:07.978+0300 [WARN] agent: error getting server health from server: server=db4 error="context deadline exceeded"
Feb 01 00:21:09 db3 consul[3199803]: 2024-02-01T00:21:09.978+0300 [WARN] agent: error getting server health from server: server=db4 error="context deadline exceeded"
Feb 01 00:21:10 db3 consul[3199803]: 2024-02-01T00:21:10.116+0300 [INFO] agent.server.memberlist.lan: memberlist: Marking db4 as failed, suspect timeout reached (2 peer confirmations)
Feb 01 00:21:10 db3 consul[3199803]: 2024-02-01T00:21:10.117+0300 [INFO] agent.server.serf.lan: serf: EventMemberFailed: db4 xxx
Feb 01 00:21:10 db3 consul[3199803]: 2024-02-01T00:21:10.117+0300 [INFO] agent.server: Removing LAN server: server="db4 (Addr: tcp/xxx:8300) (DC: dc1)"
Feb 01 00:21:11 db3 consul[3199803]: 2024-02-01T00:21:11.978+0300 [WARN] agent: error getting server health from server: server=db4 error="context deadline exceeded"
Feb 01 00:21:13 db3 consul[3199803]: 2024-02-01T00:21:13.978+0300 [WARN] agent: error getting server health from server: server=db4 error="context deadline exceeded"
Feb 01 00:21:15 db3 consul[3199803]: 2024-02-01T00:21:15.219+0300 [INFO] agent.server.serf.lan: serf: EventMemberLeave (forced): db4 xxx
Feb 01 00:21:15 db3 consul[3199803]: 2024-02-01T00:21:15.219+0300 [INFO] agent.server: Removing LAN server: server="db4 (Addr: tcp/xxx:8300) (DC: dc1)"
Feb 01 00:21:15 db3 consul[3199803]: 2024-02-01T00:21:15.979+0300 [WARN] agent: error getting server health from server: server=db4 error="context deadline exceeded"
```
### Another discussions of this issue
- [failover not paying attention ttl · Issue #2868 · zalando/patroni](https://github.com/zalando/patroni/issues/2868#issuecomment-1722926442)
- [Consul: lock is lost on 7-10 s network issues · Issue #522 · zalando/patroni](https://github.com/zalando/patroni/issues/522)
- [Increase timeout before node is deleted and its locks are lost](https://groups.google.com/g/consul-tool/c/YMa4J40qgM0)
Contributor guide
Research direction
Reproduce the behavior through the /v1/session/create and /v1/session/info endpoints using an empty NodeChecks list, then inspect the server membership and session handling indicated by the supplied logs. Compare the server-created case with the client-created case; done means a brief network interruption does not remove the session before its five-minute TTL expires.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100