Agent reports task STOPPED/OutOfMemoryError from a stale OOMKilled flag while the container keeps running
- Dominant language
- Go
- Stars
- 2.2k
- Forks
- 662
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 24
Description
## Summary
On ECS EC2 (not Fargate), the agent reported a task's container as `STOPPED` with reason `OutOfMemoryError: Container killed due to memory usage`, while the container itself kept running normally on the host for 20+ hours afterward, still serving live traffic. ECS's control plane, the target group, and even an explicit `force-new-deployment` of the whole service never noticed or reconciled it.
## Environment
- ECS Agent version: v1.106.2
- AMI: custom (Ubuntu 24.04, kernel `7.0.0-1011-aws`, aarch64/Graviton)
- Launch type: EC2
- Container runtime: Docker via containerd
## What happened
At a given timestamp the agent submitted a `SubmitTaskStateChange` marking the task's container `STOPPED`, reason `OutOfMemoryError: Container killed due to memory usage`. Over 21 hours later, the same container (same runtimeId, confirmed via `docker inspect`) was still running:
```
$ docker inspect --format 'OOMKilled={{.State.OOMKilled}} Running={{.State.Running}} Pid={{.State.Pid}}'
OOMKilled=true Running=true Pid=
```
`docker top` on the container showed its init process (supervisord) and the application's master process (Puma) with uptime matching the container's full lifetime, while two of the application's worker child processes had noticeably shorter uptime (~1h28m less) — those specific workers had been OOM-killed by the kernel and silently respawned by the application's own master process mid-life. The container's own tracked/init process never exited.
## Suspected root cause
Docker's `State.OOMKilled` flag appears to be set `true` when *any* process inside the container's cgroup is OOM-killed by the kernel, not only when the container's own tracked/init process dies — and it appears to stay `true` indefinitely, not tied to a corresponding container exit.
The agent's own code (`agent/dockerclient/dockerapi/docker_client.go`) does:
```go
if dockerContainer.State.OOMKilled {
metadata.Error = OutOfMemoryError{}
}
```
alongside a comment on `"oom"` docker event handling:
> "oom" can either means any process got OOM'd, but doesn't always mean the container dies (non-init processes). If the container also dies, you see a "die" status as well; we'll update suitably there
That comment suggests awareness that an `oom` event alone shouldn't imply the container died, but the `OOMKilled` boolean check above doesn't appear to require a corresponding container `die` to actually have been observed before the agent reports `STOPPED`/`OutOfMemoryError` — so we ended up with a task marked stopped for a container that never stopped.
## Impact
This matters for anything with local self-healing or lock semantics. Since ECS believes the task is gone: it's excluded from ALB target groups, doesn't count toward desired capacity, and — critically — `aws ecs update-service --force-new-deployment` does not detect or remove it. The container keeps running, invisible to the control plane and to any redeploy, and can hold onto external resources (in our case, exclusive RabbitMQ queue locks) indefinitely.
## What we'd expect
Either:
- The agent confirms an actual container `die` (or otherwise verifies the container process is actually gone) before reporting `STOPPED`/`OutOfMemoryError` off the `OOMKilled` flag alone, or
- Some periodic reconciliation between the agent's belief about a task's state and the container runtime's actual state, to catch and correct this kind of drift.
Happy to provide further logs/timestamps if useful.
Contributor guide
Research direction
Start in agent/dockerclient/dockerapi/docker_client.go and inspect how State.OOMKilled becomes OutOfMemoryError, then trace the related "oom" and "die" event handling described in the issue. Compare the agent's reported state with the container runtime state in this scenario; done means the agent no longer reports a running container as STOPPED solely from a stale OOMKilled flag, or the reconciliation gap is clearly addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100