hashicorp / hashicorp/consul

consul lock does not enforce SIGTERM to child process within 30s

Open
#9,788 2 comments 0 reactions 0 assignees View on GitHub
needs-investigation theme/internals
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Overview of the Issue

Consul lock does not kill child process until +90s after locking timeout.

If node1 and node2 use a consul lock (n=1) to coordinate active/passive states on the application and the active node gets fenced off from all communication with the consul leader, the consul lock command on the fenced off node does not kill the app strictly after the timeout 15s+15s, but after 15s+15s+90s, causing both apps to be active at the same time because node2 is granted the lock after 15s+15s of the disappearance of node1.

#### Reproduction Steps

1. on node1 launch a process protected by a n=1 lock (using localhost connection to consul agent running locally on localhost:8500). node1 is immediately granted the lock and myapp starts.
```
consul lock --name lock-node1 --verbose kv/myapp/.lock myapp
```
2. on node2 launch a process attempting to acquire the same lock as node1 (using localhost connection to consul agent running locally on localhost:8500). node2 does not get the lock and therefore stays waiting.
```
consul lock --name lock-node1 --verbose kv/myapp/.lock myapp
```
3. sessions seem ok when we inspect our sessions with curl:

```
[
{
"ID": "0cb7dd51-c32a-ef90-cc44-79542792a5b2",
"Name": "lock-node1",
"Node": "node1",
"Checks": [
"serfHealth"
],
"LockDelay": 15000000000,
"Behavior": "release",
"TTL": "15s",
"CreateIndex": 3137459821,
"ModifyIndex": 3137459821
},
```
4. use iptables on node1 to firewall it completely off all access to consul
```
iptables -I OUTPUT -p tcp -m multiport --dport 443,8300,8500 -j DROP
```
5. watch consul on both nodes and notice the standard 15s TTL timeout + 15s of lock grace time elapse, when that elapses node2's consul lock process executes myapp on node2.
6. on node1 myapp running under the protection of the consul lock continues to run, so myapp is running in both places at the same time, which goes against the expected behaviour.
9. wait further 90s and notice node1 finally kill the app with SIGTERM
```
Lock lost, killing child
Terminating child pid 5504
Error running handler: signal: terminated
Child terminated
Lock release failed: failed to release lock: Put "http://127.0.0.1:8500/v1/kv/kv/myapp/.lock myapp?flags=3304740253564472344&release=9713c575-6b28-029a-b3ca-78eb4beb3519": dial tcp 127.0.0.1:8500: i/o timeout
```
### Consul info for both Client and Server

Note that nodes are running consul 1.9.2 and the consul masters themselves are running 1.6.1 all on linux Centos7. All consul binaries have been download directly from hashicorp and not built from source.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.