aws / aws/containers-roadmap

ECS agent doesn't report available memory after the initial registration

Open
#1,944 3 comments 0 reactions 0 assignees View on GitHub
ECS
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Summary
The ECS agent doesn't report back to AWS the reserved memory after the initial registration.

### Description
On the first boot, the ECS agent reports the reserved memory successfully even after changing the configuration through `/etc/ecs/ecs.config`. However, if the value of the reserved memory is updated while the system is running the ECS agent doesn't report back the reserved memory, and the AWS web console doesn't show the actual reserved memory.

### Expected Behavior
The AWS web console should report the correct reserved memory values

### Observed Behavior
The AWS web console doesn't report the correct reserved memory values. FWIW, I waited an hour to see if this was a propagation problem with the cache, but it doesn't seem like that's the issue.

### Environment Details

```bash
[root@ ]# docker info
Client:
Context: default
Debug Mode: false

Server:
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 3
Server Version: 20.10.13
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
Default Runtime: runc
Init Binary: docker-init
containerd version: 9cc61520f4cd876b86e77edfeb88fbcd536d1f9d
runc version: 1e7bb5b773162b57333d57f612fd72e3f8612d94
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 4.14.291-218.527.amzn2.aarch64
Operating System: Amazon Linux 2
OSType: linux
Architecture: aarch64
CPUs: 1
Total Memory: 1.878GiB
Name: ip-172-31-24-35.us-west-2.compute.internal
ID: CALM:O3AB:6TA4:ZXDD:ABBP:OHLQ:5SNT:MPOJ:64VR:6K7Y:FNZR:HHNQ
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
```

```bash
[root@ ]# curl http://localhost:51678/v1/metadata -s | jq
{
"Cluster": "default",
"ContainerInstanceArn": "",
"Version": "Amazon ECS Agent - v1.64.0 (*2e05d8f1)"
}
```

```bash
[root@ ]# cat /etc/ecs/ecs.config
ECS_RESERVED_MEMORY=200

```

### Supporting Log Snippets
```bash
# In the instance logs, I can see this for the initial boot:
level=info time=2022-10-21T22:18:29Z msg="Remaining mem: 1922" module=client.go
# After I changed the environment variables, I can see the new value is read by the agent since it reports less memory
level=info time=2022-10-21T22:27:17Z msg="Remaining mem: 1722" module=client.go
```
I would expect the ECS API will return the correct remaining resources values, but it returns the values used to register the instance:

```bash
~ ❯ aws ecs describe-container-instances --container-instances | jq ".containerInstances[0].registeredResources"
[
{
"name": "CPU",
"type": "INTEGER",
"doubleValue": 0.0,
"longValue": 0,
"integerValue": 1024
},
{
"name": "MEMORY",
"type": "INTEGER",
"doubleValue": 0.0,
"longValue": 0,
"integerValue": 1922 # This should be 1722
},
{
"name": "PORTS",
"type": "STRINGSET",
"doubleValue": 0.0,
"longValue": 0,
"integerValue": 0,
"stringSetValue": [
"22",
"2376",
"2375",
"51678",
"51679"
]
},
{
"name": "PORTS_UDP",
"type": "STRINGSET",
"doubleValue": 0.0,
"longValue": 0,
"integerValue": 0,
"stringSetValue": []
}
]
```

Contributor guide

Open the contributing guide

Research direction

Start with the ECS agent's client.go path indicated by the logs and reproduce changing /etc/ecs/ecs.config after registration. Use aws ecs describe-container-instances to verify that registeredResources.MEMORY changes from 1922 to 1722 after the update.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.