ECS Agent is not restarted unhealthy containers for Dockerfile healthcheck
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Summary
ECS Agent is not restarted unhealthy containers for Dockerfile healthcheck
### Description
According to an article [Amazon ECS Supports Container Health Checks and Task Health Management](https://aws.amazon.com/about-aws/whats-new/2018/03/amazon-ecs-supports-container-health-checks-and-task-health-mana/) you have announced that Amazon ECS integrates with Docker container health checks to monitor the health of each container using HEALTHCHECK.
So we have ECS cluster with several services (i.e. logger). We added healthcheck for our logger as self hosted service and mapped to port (i.e. 5050), so dockerfile looks like this:
```
FROM microsoft/dotnet:2.1-runtime
WORKDIR ***
COPY . .
HEALTHCHECK CMD curl --fail http://localhost:5050/ || exit 1
ENTRYPOINT dotnet /***.dll
```
But next we read ECS documentation about [Task Definition Parameters](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html) and [Task](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_Task.html) and observed the following:
> The Amazon ECS container agent does not monitor or report on Docker health checks that are embedded in a container image (such as those specified in a parent image or from the image's Dockerfile) and not specified in the container definition. Health check parameters that are specified in a container definition override any Docker health checks that exist in the container image.
Next I'm trying to play with docker containers and stopped one of unhealthy. So container restarted(but not removed) successfully and set unhealthy status again for new one (**please note, ECS task was not restarted**).
The questions are:
1. Will the healthcheck work if we move it to CloudFormation stack template?
2. Do you or will you in future support healthcheck specified specified in a parent image or from the image's Dockerfile?
### Expected Behavior
Unhealthy docker container or ECS task should be restarted
### Observed Behavior
Unhealthy docker container and ECS task are running

### Environment Details
docker info:
```
Containers: 8
Running: 7
Paused: 0
Stopped: 1
Images: 4
Server Version: 18.06.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-1049-aws
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.801GiB
Name: ***
ID: ***
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
```
curl http://localhost:51678/v1/metadata
`{"Cluster":"fnd-Cluster-***-cluster","ContainerInstanceArn":"arn:aws:ecs:eu-west-1:***:container-instance/0a263365-c9ad-40e0-b500-b169913bc3e1","Version":"Amazon ECS Agent - v1.24.0 (8b5e1863)"}`
Contributor guide
Research direction
Start with the linked ECS Task Definition Parameters and Task documentation, then compare the Dockerfile HEALTHCHECK behavior with health checks specified in a container definition or CloudFormation template. Review the reported ECS Agent v1.24.0 metadata endpoint and the observed container/task states. Done means the supported behavior and restart expectations for unhealthy containers and tasks are clearly established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker
- Domain
- cloud, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100