Inconsistent behavior invoking a script watch handler
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.
#### Overview of the Issue
Watch handlers are not invoked for as many number of changes under a key prefix. In the following scenario, "hello" is printed only 2 times even though I am creating 10 new KV under the watched prefix.
consul watch CLI does not inhibit the same issue and works as expected.
#### Reproduction Steps
Scenario
--------
1. I have a simple watch configure in consul that looks for updates under "foo/" KV prefix.
```
{
"watches": [
{
"prefix": "foo/",
"args": [
"my-handler.sh"
],
"type": "keyprefix"
}
]
}
```
2. The my-handler.sh needs to call another python program that runs the business logic. However, to simulate the problem, I have it like below.
```
#!/bin/bash
(sleep 1; echo "hello" >> /tmp/out)&
```
3. A simple script to load 10 keys under the prefix.
```
#!/bin/bash
for i in {1..10}; do
echo "Adding key foo/key$i to consul kv..";
consul kv put foo/key$i $i;
done
```
### Consul info for both Client and Server
agent:
check_monitors = 5
check_ttls = 0
checks = 19
services = 15
build:
prerelease =
revision = a42ded47
version = 1.5.3
consul:
acl = disabled
bootstrap = false
known_datacenters = 2
leader = false
leader_addr = 192.168.1.13:8300
server = true
raft:
applied_index = 46376
commit_index = 46376
fsm_pending = 0
last_contact = 45.895642ms
last_log_index = 46376
last_log_term = 344
last_snapshot_index = 32811
last_snapshot_term = 344
latest_configuration = [{Suffrage:Voter ID:91c0e708-1522-9d56-88a9-da32681a177a Address:192.168.1.11:8300} {Suffrage:Voter ID:fd9951a3-c80d-bca8-effe-d70a6030768b Address:192.168.1.13:8300} {Suffrage:Voter ID:3695e4d6-f34d-2065-611c-eece244f09ff Address:192.168.1.12:8300}]
latest_configuration_index = 1
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 344
runtime:
arch = amd64
cpu_count = 4
goroutines = 153
max_procs = 4
os = linux
version = go1.12.1
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 26
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 30
members = 3
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 39
members = 6
query_queue = 0
query_time = 1
### Operating system and Environment details
Consul: v1.5.3
OS: CentOS 7
x86_64
### Questions
1. Can I have long running scripts as watch handlers (non-HTTP) ?
2. Can I spawn processes from within my script watch handler ?
3. Any recommended solution for the problem abve ?
NOTE: I have tried 'trap "echo signal; kill -TERM $child;" SIGTERM' and 'wait' for children in my watch handler, but no luck.
Contributor guide
Research direction
Start by reproducing the keyprefix watch scenario with the consul watch CLI, the shown my-handler.sh, and the loop that creates foo/key1 through foo/key10. Read the watch-handler path and compare it with the CLI behavior; done means the invocation behavior is consistent for all relevant prefix changes or the limitation is clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, shell
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100