Issues with ECS instance memory management

### Summary

Metrics are inconsistent between docker, operating system and different places in the AWS console. ECS keeps placing tasks on instances where they cause `137` error code crashes due to memory exhaustion.

### Description

I recently set up a cluster of three `t3.nano` instances. I started launching small tasks with a single container, a soft limit of 16 MB of memory and no hard limit. Everything was working well until the total memory utilization reached around 60%. Suddenly, tasks seemed to randomly be killed with `137` out-of-memory exceptions.

Upon investigating, I found that, while the ECS cluster page showed around 60% memory utilization, the instances themselves had very little memory left. I saw that the `dockerd` itself was consuming over 60 MB of memory, with one `docker-container` process of ~30MB. Each container seemed to also use a bit more memory than provided by docker stats (~10% deviation) and come with an extra `docker-container` process that consumed an additional 15% on top of that.

Overall, there were roughly 200 MB of memory not accounted for by the ECS metrics, corresponding to more than 40% of total memory available. In that context, it made sense that the tasks would crash and exit each time there was rebalancing or roll over.

After some research, I found the rather poorly documented `ECS_RESERVED_MEMORY` environment variable. I changed the cluster to three `t3.micro` instances and configured them with 200 MB of reserved memory. I was hoping that this would address the issue and allow my cluster to properly balance tasks between instances. Unfortunately, the change didn't seem to have an effect and inconsistencies remain in place.

Please note that the soft limit is set to 16 MB (less than each task consumes) with no hard limit. It seemed to be the only way to avoid wasting memory when tasks remain rather stable, but could spike during some time periods.

### Expected Behavior

- Accurate memory metrics of the tasks / docker containers, including all memory utilized for each task.
- `ECS_RESERVED_MEMORY` deducted from the `Memory available` number of the instance.
- `MemoryUtilization` for the cluster calculated on the basis of this adjusted number.
- Accurate estimation of whether a task can be placed on an instance without issues.

### Observed Behavior

- Memory metrics do not take into account the memory used by `dockerd` (now at around 95 MB for one `t3.micro` with 9 tasks) and the related `docker-container` process (29MB for a total of 124 MB lost memory).
- Memory metrics do not take into account the memory overhead of task processes and the related `docker-container` processes (around 30% memory overhead per task).
- Memory metrics do not take into account the `ECS_RESERVED_MEMORY` setting (at 200 MB for the current configuration).

I do not know if the `ECS_RESERVED_MEMORY` setting will be taken into account when placing tasks, but I doubt it very much. From the little I was able to gather from the documentation, this should put a memory limit on docker itself; since docker lists 957 MB available, it seems to allocate the full system memory to docker still.

I am assuming that the task overhead usually does not matter for applications that consume 1 GB and more; for our use case, however, it does, because each task currently tops at around 27 MB.

The following is pure conjecture, but I would think that the bridge network mode used for the tasks accounts for part of the increased memory usage of the docker daemon itself. With a limit of two `awsvpc` interfaces per container, that's a required setting, though.

As it currently stands, ECS is unusable for our use case, which is running lots of long-running small tasks with decent network I/O and small memory usage per task. We will probably never be able to completely avoid crashing tasks without manually monitoring every instance for real memory utilization.

### Environment Details

`ECS instances` tab
```
Available memory: 597
```

`Metrics` tab
```
MemoryUtilization: 34%
```

`cat /etc/ecs/ecs.config`
```
ECS_CLUSTER=cluster
ECS_RESERVED_MEMORY=200
```

`docker info`
```
Containers: 10
Running: 10
Paused: 0
Stopped: 0
Images: 3
Server Version: 18.06.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.14.114-103.97.amzn2.x86_64
Operating System: Amazon Linux 2
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 957.4MiB
Name: ip-10-0-2-69.eu-west-1.compute.internal
ID: YJFP:MJJU:RAOT:WHKE:EZQA:ZVYS:6IUY:TTIX:3N7F:3FQ2:QU6Z:JKX5
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
```

`docker stats`
```
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
f8e4aeb812ea ecs-task-8aaebe9ab0e0f3bbbd01 0.02% 26.94MiB / 957.4MiB 2.81% 141MB / 76.1MB 0B / 0B 10
d1cf0c15d607 ecs-task-dcbab1bff7b9df92f501 0.00% 25.72MiB / 957.4MiB 2.69% 1.61MB / 1.12MB 0B / 0B 11
ed4760a60664 ecs-task-cacdefd9b3a0e6c26e00 0.00% 26.78MiB / 957.4MiB 2.80% 1.26MB / 902kB 0B / 0B 10
3cda7974535d ecs-task-ca8bf9cc949fd083e701 0.08% 27.3MiB / 957.4MiB 2.85% 291MB / 156MB 0B / 0B 11
d715b4a04a63 ecs-task-ceffd0d398fef0c87e00 0.00% 26.97MiB / 957.4MiB 2.82% 179MB / 96MB 0B / 0B 10
0cb37f1b7461 ecs-task-e0ac828ee1b1d5eeae01 0.00% 26.06MiB / 957.4MiB 2.72% 4.34MB / 2.79MB 0B / 0B 10
e9b140f2872a ecs-task-a2d2bcdadb90f6f43c00 0.00% 26.09MiB / 957.4MiB 2.73% 1.43MB / 997kB 0B / 0B 10
9069de3b4375 ecs-task-949ae9e6f6d4c7858201 0.06% 27.48MiB / 957.4MiB 2.87% 294MB / 156MB 0B / 0B 10
2ae089933a3e ecs-task-eeba8fc4fb87f3cb1100 0.00% 25.39MiB / 957.4MiB 2.65% 1.13MB / 822kB 0B / 0B 10
420f6e325e08 ecs-agent 0.65% 11.98MiB / 957.4MiB 1.25% 0B / 0B 35.1MB / 5.03MB 11
```
If you check this usage against total memory, it's 30%.
If you check this usage against total memory minus reserved memory, it's 38%.
This is the same ratio given by `Available memory` divided by `total memory`, so I'm assuming the `Available memory` is based on the wrong total?

`free -m`
```
total used free shared buff/cache available
Mem: 957 367 181 0 407 422
Swap: 0 0 0
```
As you can see, real ussage is quite a bit higher than what's given by the docker stats.

`top` (sorted by memory)
```
Tasks: 94 total, 1 running, 65 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0,7 us, 0,7 sy, 0,0 ni, 98,3 id, 0,2 wa, 0,0 hi, 0,0 si, 0,2 st
KiB Mem : 980356 total, 185484 free, 377140 used, 417732 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 432208 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2950 root 20 0 1268,1m 95,4m 39,6m S 0,3 10,0 6:21.55 dockerd
4547 root 20 0 830,5m 33,6m 8,4m S 0,0 3,5 2:33.80 task
4122 root 20 0 758,5m 33,5m 8,0m S 0,0 3,5 2:35.27 task
4440 root 20 0 758,5m 33,2m 8,2m S 0,0 3,5 1:35.28 task
4828 root 20 0 758,5m 32,9m 8,0m S 0,3 3,4 1:15.77 task
4601 root 20 0 758,5m 32,9m 8,0m S 0,0 3,4 0:01.90 task
4294 root 20 0 758,5m 32,3m 8,3m S 0,0 3,4 0:03.99 task
4233 root 20 0 758,5m 32,2m 8,0m S 0,0 3,4 0:01.94 task
4753 root 20 0 830,5m 31,4m 7,9m S 0,0 3,3 0:02.13 task
4021 root 20 0 758,5m 31,2m 8,2m S 0,0 3,3 0:01.75 task
3022 root 20 0 1224,1m 28,7m 19,1m S 1,3 3,0 15:05.11 docker-containe
3335 root 20 0 35,5m 23,9m 14,4m S 0,3 2,5 2:43.02 agent
1105 root 20 0 48,2m 12,8m 12,4m S 0,0 1,3 0:00.50 systemd-journal
14135 root 20 0 145,0m 8,4m 7,1m S 0,0 0,9 0:00.01 sshd
3254 root 20 0 452,4m 8,4m 6,8m S 0,0 0,9 0:00.03 amazon-ecs-init
2852 root 20 0 108,2m 7,6m 6,6m S 0,0 0,8 0:00.03 sshd
2750 postfix 20 0 86,4m 6,7m 5,7m S 0,0 0,7 0:00.02 qmgr
14044 postfix 20 0 86,4m 6,5m 5,5m S 0,0 0,7 0:00.00 pickup
1 root 20 0 42,8m 5,3m 3,8m S 0,0 0,6 0:02.37 systemd
2748 root 20 0 86,3m 4,7m 3,7m S 0,0 0,5 0:00.14 master
3273 root 20 0 7,3m 4,5m 3,7m S 0,0 0,5 0:01.56 docker-containe
14137 ec2-user 20 0 145,0m 4,5m 3,2m S 0,0 0,5 0:00.08 sshd
14289 ec2-user 20 0 165,1m 4,5m 3,8m R 0,0 0,5 0:00.01 top
4509 root 20 0 7,4m 4,4m 3,8m S 0,0 0,5 0:01.37 docker-containe
2817 root 20 0 219,6m 4,3m 3,4m S 0,0 0,5 0:02.16 rsyslogd
4788 root 20 0 8,8m 4,3m 3,6m S 0,0 0,5 0:01.39 docker-containe
4192 root 20 0 7,3m 4,3m 3,5m S 0,0 0,4 0:01.48 docker-containe
4082 root 20 0 8,8m 4,2m 3,5m S 0,0 0,4 0:01.41 docker-containe
1930 dbus 20 0 57,0m 4,1m 3,6m S 0,0 0,4 0:00.14 dbus-daemon
2460 root 20 0 103,0m 4,1m 2,1m S 0,0 0,4 0:00.02 dhclient
4402 root 20 0 8,7m 4,0m 3,3m S 0,0 0,4 0:01.39 docker-containe
4245 root 20 0 7,3m 3,9m 3,4m S 0,0 0,4 0:01.32 docker-containe
3976 root 20 0 8,8m 3,9m 3,3m S 0,0 0,4 0:01.30 docker-containe
2082 chrony 20 0 117,7m 3,9m 3,4m S 0,0 0,4 0:00.67 chronyd
4550 root 20 0 7,4m 3,7m 3,2m S 0,0 0,4 0:01.28 docker-containe
4710 root 20 0 8,8m 3,7m 3,1m S 0,0 0,4 0:01.42 docker-containe
2563 root 20 0 103,0m 3,7m 1,7m S 0,0 0,4 0:00.08 dhclient
1334 root 20 0 41,3m 3,5m 2,9m S 0,0 0,4 0:00.13 systemd-udevd
14138 ec2-user 20 0 119,3m 3,5m 3,1m S 0,0 0,4 0:00.02 bash
2908 root 20 0 129,9m 3,0m 2,4m S 0,0 0,3 0:00.04 crond
2006 root 20 0 25,8m 2,9m 2,5m S 0,0 0,3 0:00.09 systemd-logind
1515 root 16 -4 60,8m 2,1m 1,5m S 0,0 0,2 0:00.04 auditd
2913 root 20 0 114,1m 2,0m 1,9m S 0,0 0,2 0:00.00 agetty
1991 root 20 0 97,6m 1,6m 1,5m S 0,0 0,2 0:01.13 irqbalance
2924 root 20 0 114,4m 1,6m 1,5m S 0,0 0,2 0:00.14 agetty
2052 root 20 0 12,8m 0,9m 0,8m S 0,0 0,1 0:01.12 rngd
2057 root 20 0 4,2m 0,1m 0,0m S 0,0 0,0 0:00.00 acpid
```
You see the `docker-containe` for each task process? As well as the agent and one big `docker-containe` for it.

No repository file, test, or entry point is identified; begin by reviewing the ECS_RESERVED_MEMORY setting and comparing the reported Available memory and MemoryUtilization with the supplied docker stats and free -m outputs. Done would require an agreed change that accounts for daemon and task overhead and reserved memory in metrics and placement, with regression coverage.

Written by the indexing model from the issue text.