hashicorp / hashicorp/consul

Inconsistent behavior invoking a script watch handler

Open
#6,636 2 comments 0 reactions 0 assignees View on GitHub
theme/operator-usability type/docs
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.

#### Overview of the Issue

Watch handlers are not invoked for as many number of changes under a key prefix. In the following scenario, "hello" is printed only 2 times even though I am creating 10 new KV under the watched prefix.

consul watch CLI does not inhibit the same issue and works as expected.

#### Reproduction Steps

Scenario
--------

1. I have a simple watch configure in consul that looks for updates under "foo/" KV prefix.
```
{
"watches": [
{
"prefix": "foo/",
"args": [
"my-handler.sh"
],
"type": "keyprefix"
}
]
}
```

2. The my-handler.sh needs to call another python program that runs the business logic. However, to simulate the problem, I have it like below.

```
#!/bin/bash

(sleep 1; echo "hello" >> /tmp/out)&
```

3. A simple script to load 10 keys under the prefix.
```
#!/bin/bash

for i in {1..10}; do
echo "Adding key foo/key$i to consul kv..";
consul kv put foo/key$i $i;
done
```

### Consul info for both Client and Server

agent:
check_monitors = 5
check_ttls = 0
checks = 19
services = 15
build:
prerelease =
revision = a42ded47
version = 1.5.3
consul:
acl = disabled
bootstrap = false
known_datacenters = 2
leader = false
leader_addr = 192.168.1.13:8300
server = true
raft:
applied_index = 46376
commit_index = 46376
fsm_pending = 0
last_contact = 45.895642ms
last_log_index = 46376
last_log_term = 344
last_snapshot_index = 32811
last_snapshot_term = 344
latest_configuration = [{Suffrage:Voter ID:91c0e708-1522-9d56-88a9-da32681a177a Address:192.168.1.11:8300} {Suffrage:Voter ID:fd9951a3-c80d-bca8-effe-d70a6030768b Address:192.168.1.13:8300} {Suffrage:Voter ID:3695e4d6-f34d-2065-611c-eece244f09ff Address:192.168.1.12:8300}]
latest_configuration_index = 1
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 344
runtime:
arch = amd64
cpu_count = 4
goroutines = 153
max_procs = 4
os = linux
version = go1.12.1
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 26
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 30
members = 3
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 39
members = 6
query_queue = 0
query_time = 1

### Operating system and Environment details

Consul: v1.5.3
OS: CentOS 7
x86_64

### Questions
1. Can I have long running scripts as watch handlers (non-HTTP) ?
2. Can I spawn processes from within my script watch handler ?
3. Any recommended solution for the problem abve ?

NOTE: I have tried 'trap "echo signal; kill -TERM $child;" SIGTERM' and 'wait' for children in my watch handler, but no luck.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the keyprefix watch scenario with the consul watch CLI, the shown my-handler.sh, and the loop that creates foo/key1 through foo/key10. Read the watch-handler path and compare it with the CLI behavior; done means the invocation behavior is consistent for all relevant prefix changes or the limitation is clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, shell
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.