aws / aws/containers-roadmap

[ECS][request] Agent health check reports healthy when ECS shows agent disconnected

Open
#1,071 5 comments 1 reaction 0 assignees View on GitHub
ECS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Summary
The ECS agent healthcheck can report healthy even if the agent is no longer connected to the ECS control plane.

### Description
The ECS agent showed as Connected: false in the ECS console. It continued to run and respond on network ports. Container credentials provided by the agent expired. The ECS-agent logs showed that the ACS connection recycle had stopped about 4 hours earlier.
No errors were logged by the agent. The agent successfully logged calls to /v2/credentials.

### Expected Behavior
An agent that has for whatever reason lost connection with the control plane will show as unhealthy and be recycled by systemd.

### Observed Behavior
The agent reported as healthy.

### Environment Details
Docker info:
```
Client:
Debug Mode: false

Server:
Containers: 24
Running: 22
Paused: 0
Stopped: 2
Images: 16
Server Version: 19.03.6-ce
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: amazon-ecs-volume-plugin local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: ff48f57fc83a8c44cf4ad5d672424a98ba37ded6
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.14.186-146.268.amzn2.x86_64
Operating System: Amazon Linux 2
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 30.41GiB
Name: ip-10-153-72-219
ID: W23X:TVXI:IAFK:HGGR:52AK:4433:6ABW:ST4G:HZX3:SQ2R:5RC6:OU5I
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
```

metadata:
```
{
"Cluster": "REDACTED",
"ContainerInstanceArn": "arn:aws:ecs:us-west-2:REDACTED:container-instance/REDACTED",
"Version": "Amazon ECS Agent - v1.43.0 (1ebf0604)"
}
```

OS:
```
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
```
df -h
```
Filesystem Size Used Avail Use% Mounted on
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 4.0K 16G 1% /dev/shm
tmpfs 16G 1.9M 16G 1% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/nvme0n1p1 89G 7.3G 81G 9% /
tmpfs 512M 0 512M 0% /tmp
/dev/nvme1n1 200G 8.0G 192G 4% /var/lib/docker
SNIP containers
```

Node time was in sync.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the ECS agent health check and the ACS connection recycle behavior described in the report. Reproduce or inspect the disconnected-agent state, then verify that the health check reports unhealthy and that systemd recycles the agent instead of allowing credentials and other requests to continue.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker
Domain
cloud, infrastructure, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.