containerd / containerd/cgroups

blkio stats empty for zfs on kernel 5.x

Open
#78 0 comments 4 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.2k
Forks
252
PR merge metrics
No merged PRs in 30d

Description

I have two docker instances on the same server, both using:
`Docker version 18.09.3, build 774a1f4`

and

`containerd github.com/containerd/containerd 1.2.4 e6b3f5632f50dbc4e9cb6288d911bf4f5e95b18e`

Kernel:
`5.0.1-050001-generic`

One is using overlay2, the other is using zfs as the graph driver. The instance on overlay2 is running fine, however the zfs instance is not reporting blkio stats:
```json
"blkio_stats": {
"io_service_bytes_recursive": [],
"io_serviced_recursive": [],
"io_queue_recursive": [],
"io_service_time_recursive": [],
"io_wait_time_recursive": [],
"io_merged_recursive": [],
"io_time_recursive": [],
"sectors_recursive": []
},
```

I am trying to debug this further, the blkio controller for cgroups seems to have changed, I am only seeing the following stats inside container cgroups:
```
# ls -la /sys/fs/cgroup/blkio/
total 0
drwxr-xr-x 2 root root 0 Mar 21 07:48 .
dr-xr-xr-x 14 root root 360 Mar 21 07:48 ..
-r--r--r-- 1 root root 0 Mar 21 12:29 blkio.bfq.io_service_bytes
-r--r--r-- 1 root root 0 Mar 21 12:29 blkio.bfq.io_service_bytes_recursive
-r--r--r-- 1 root root 0 Mar 21 12:29 blkio.bfq.io_serviced
-r--r--r-- 1 root root 0 Mar 21 12:29 blkio.bfq.io_serviced_recursive
-rw-r--r-- 1 root root 0 Mar 21 12:29 blkio.bfq.weight
--w------- 1 root root 0 Mar 21 12:29 blkio.reset_stats
-r--r--r-- 1 root root 0 Mar 21 07:48 blkio.throttle.io_service_bytes
-r--r--r-- 1 root root 0 Mar 21 12:29 blkio.throttle.io_service_bytes_recursive
-r--r--r-- 1 root root 0 Mar 21 07:48 blkio.throttle.io_serviced
-r--r--r-- 1 root root 0 Mar 21 12:29 blkio.throttle.io_serviced_recursive
-rw-r--r-- 1 root root 0 Mar 21 12:29 blkio.throttle.read_bps_device
-rw-r--r-- 1 root root 0 Mar 21 12:29 blkio.throttle.read_iops_device
-rw-r--r-- 1 root root 0 Mar 21 12:29 blkio.throttle.write_bps_device
-rw-r--r-- 1 root root 0 Mar 21 12:29 blkio.throttle.write_iops_device
-rw-r--r-- 1 root root 0 Mar 21 12:29 cgroup.clone_children
-rw-r--r-- 1 root root 0 Mar 21 12:29 cgroup.procs
-rw-r--r-- 1 root root 0 Mar 21 12:29 notify_on_release
-rw-r--r-- 1 root root 0 Mar 21 12:29 tasks
```

So none of the blkio.io* files mentioned here exist:
https://www.kernel.org/doc/Documentation/cgroup-v1/blkio-controller.txt

These are the default kernel config options in 5.0.1 (we're using a default mainline kernel):
```
CONFIG_BLK_CGROUP=y
CONFIG_CGROUP_SCHED=y
# CONFIG_DEBUG_BLK_CGROUP is not set
# CONFIG_BLK_CGROUP_IOLATENCY is not set
```

I have tried with both BFQ enabled and disabled (mq-deadline), also tried zfs with and without the `zfs_vdev_scheduler=bfq` option, but both made no difference.

I believe the relevant code is:
https://github.com/containerd/cgroups/blob/master/blkio.go

However I haven't had a chance to properly dive into the code yet, perhaps I'm missing something obvious, can anyone point me in the right direction? I've also posted in the relevant docker issue for checking kernel 5.x compatibility here https://github.com/moby/moby/issues/38887, however this seems to be a containerd issue.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.