containerd high cpu when nomad is running

Nobody has claimed this yet.

I'm currently experimenting by running ~100 containers on a single t3.xlarge (aws) instance. There are a mixture of in-house apps running e.g.:

- job: api
- tasks:
- nginx
- php-fpm
- laravel queues and other workers
- job: node:
- tasks:
- nginx
- node

My goal is that each container should handle a single process rather than bundling it all together and trying to manage with say supervisord. The downside being a large number of containers to manage.

I'm noticing high amounts of CPU being consumed when there is very low traffic (current requests are basically consul health-checks)


(2:25 is when all ~100 containers were running)

If I stop the Nomad agent, the containerd CPU levels reduce. In the following screenshot I stoped the Nomad agent at 2:30


I'm assuming this has something to do with some form of stats collection (maybe I'm wrong)

For example if I run `docker stats` and allow it to stream in real time, the containerd CPU level is even worse than when the nomad agent is running.

Assuming I'm heading in the right direction, I've tried playing around with the Nomad Telemetry configuration by increasing the `collection_interval` and also setting both `publish_node_metrics` and `publish_allocation_metrics` to false, but it didn't really help.

Is there any setting I can tweak to lower the containerd CPU levels whilst nomad is running?

I use NewRelic infra to monitor the client instance and containers, so I dont really need to view the metrics in near real-time via Nomad (assumes this is stats related)

Below are the current versions in use:

```
Ubuntu 18.04.3 LTS
Nomad v0.11.1 (b43457070037800fcc8442c8ff095ff4005dab33)

Server: Docker Engine - Community
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: afacb8b7f0
Built: Wed Mar 11 01:24:19 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
```

Start by reproducing the report with about 100 Docker containers on the listed Ubuntu, Docker, and Nomad versions, while comparing containerd CPU with the Nomad agent stopped. Review the Nomad telemetry settings mentioned in the issue and determine whether changing them affects CPU usage; done means identifying the cause or a validated way to reduce the overhead.

Written by the indexing model from the issue text.