gliderlabs / gliderlabs/docker-consul

defunct/zombie healthcheck processes

Open
#45 24 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
1.1k
Forks
270
PR merge metrics
No merged PRs in 30d

Description

We're running AWS EC2 instances within a VPC, with many Docker containers running under [mainly] Docker 1.3.3. We're finding that the health-checks within the Consul container are not terminating properly, and leaving behind defunct/zombie processes. In some containers, there are only a few zombies, on other machines, it can run into the 1000s, probably because "interval" is 5 seconds; this is causing the process list to fill over few days, which in turn locks the Docker host machine. If the process list hasn't completely filled, killing the Consul container terminates all these zombies and frees the resources, but obviously this is not an optimal solution. We're running the latest Consul 0.4.1 in the progrium/consul image from Dockerhub. Consul has been amazing so far, and this is the only "glaring" issue we've run into. Anyone have any ideas? Here is an example healthcheck .json file, similar to one we're using:

``` json
{
"id": "fooid",
"name": "fooname",
"tags": [
"footag"
],
"port": 12345
,
"check": {
"script": "/opt/consul/foo_health.sh 12.34.56.78 12345",
"interval": "5s",
"note": ""
}
}
```

The bash script we're using looks like:

``` bash
#!/bin/bash

IP=$1
PORT=$2

OUT=`curl -s --data-binary '{"jsonrpc": "1.0", "id":"healthcheck", "method": "getwork", "params": [] }' -H 'content-type: text/plain;' http://ziftr:abc123@$IP:$PORT/`

echo $OUT | grep '"error":null' > /dev/null

if [ $? -gt 0 ]; then
echo $OUT | sed -n -e 's/.*\"error":{\([^}]\+\).*/\1/p' | sed -n -e 's/.*\"message":"\([^"]\+\).*/\1/p'

echo && echo 'Raw message:' $OUT
exit 2
else
echo OK
exit 0
fi
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.