gliderlabs / gliderlabs/docker-consul
defunct/zombie healthcheck processes
- Dominant language
- Shell
- Stars
- 1.1k
- Forks
- 270
- PR merge metrics
- No merged PRs in 30d
Description
We're running AWS EC2 instances within a VPC, with many Docker containers running under [mainly] Docker 1.3.3. We're finding that the health-checks within the Consul container are not terminating properly, and leaving behind defunct/zombie processes. In some containers, there are only a few zombies, on other machines, it can run into the 1000s, probably because "interval" is 5 seconds; this is causing the process list to fill over few days, which in turn locks the Docker host machine. If the process list hasn't completely filled, killing the Consul container terminates all these zombies and frees the resources, but obviously this is not an optimal solution. We're running the latest Consul 0.4.1 in the progrium/consul image from Dockerhub. Consul has been amazing so far, and this is the only "glaring" issue we've run into. Anyone have any ideas? Here is an example healthcheck .json file, similar to one we're using:
``` json
{
"id": "fooid",
"name": "fooname",
"tags": [
"footag"
],
"port": 12345
,
"check": {
"script": "/opt/consul/foo_health.sh 12.34.56.78 12345",
"interval": "5s",
"note": ""
}
}
```
The bash script we're using looks like:
``` bash
#!/bin/bash
IP=$1
PORT=$2
OUT=`curl -s --data-binary '{"jsonrpc": "1.0", "id":"healthcheck", "method": "getwork", "params": [] }' -H 'content-type: text/plain;' http://ziftr:abc123@$IP:$PORT/`
echo $OUT | grep '"error":null' > /dev/null
if [ $? -gt 0 ]; then
echo $OUT | sed -n -e 's/.*\"error":{\([^}]\+\).*/\1/p' | sed -n -e 's/.*\"message":"\([^"]\+\).*/\1/p'
echo && echo 'Raw message:' $OUT
exit 2
else
echo OK
exit 0
fi
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.