influxdata / influxdata/influxdata-docker

[BUG] kapacitor alerts are not running as child processes

Open
#480 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
364
Forks
255
Avg merge
3h 18m
Merged PRs (30d)
7

Description

When using the `exec` handler of kapacitor, the script triggered by kapacitor is not running as a child process of `kapacitord`, which itself has pid1.

In my use case, I have scripts that handle a longer chain in pushover (send alert, wait for acknowledgement, send to another user...)

If the docker container is stopped within that runtime of the script, my script is silently and ruggedly killed - as there is no TERM signal I can handle with.

SOLUTION: All the alerts triggered by kapacitor should run as child processes of kapacitord or another process with pid1 handling all the subprocesses.

Showing the processlist with an alert running `sleep 30`, just to show the missing process tree:

```
root 1 4.9 0.1 2706372 161568 ? Ssl 17:43 0:44 kapacitord
root 19575 0.0 0.0 17976 2824 ? S 17:58 0:00 /bin/bash /usr/local/bin/pushover-alert-chain.sh
root 19576 0.0 0.0 4184 612 ? S 17:58 0:00 \_ sleep 30
```

Image:` kapacitor:1.5.4`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by examining Kapacitor's exec handler and how kapacitord launches alert scripts in the kapacitor:1.5.4 Docker image. Reproduce the process tree with a script that runs sleep 30, then observe shutdown behavior and signal delivery. Done means an exec alert remains a child process of the service and can handle TERM when the container stops.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, shell
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.