Can't create containers if there is a v1 cpuset cgroup with exclusive cores
- Dominant language
- C
- Stars
- 4.1k
- Forks
- 444
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 51
Description
Steps to reproduce:
1. Boot with `systemd.unified_cgroup_hierarchy=0` (in my case it's in a VM with 2 cores)
2. Create a random cgroup `# mkdir /sys/fs/cgroup/cpuset/iamspecial`
3. Assign one of the cores to that cgroup `# echo 1 > /sys/fs/cgroup/cpuset/iamspecial/cpuset.cpus`
4. Make that cpuset exclusive `# echo 1 > /sys/fs/cgroup/cpuset/iamspecial/cpuset.cpu_exclusive`
5. `# crun spec; mkdir -p rootfs/usr/bin; touch rootfs/usr/bin/sh; crun run test1234`
Expected result: `open executable: Permission denied`
Actual result: ``write `cpuset.cpus`: Invalid argument``
---
Some short notes and observations:
- Of course, in step 5. you could replace the rootfs with something more proper like a busybox and expect an actual shell being opened.
- For what it's worth, `runc` fails the same way, although it's more explicit telling us it failed when writing `0-1` to `/sys/fs/cgroup/cpuset/test1234/cpuset.cpus`
- using `--systemd-cgroup` only results in ``write `cpuset.cpus`: Permission denied`` (`runc` still gets `EINVAL`)
- adding `"cpu": { "cpus": "0" }` in the `linux.resources` section fixes the issue in most cases
Below is a table, which summarizes all those combinations:
| | crun | crun --systemd-cgroup | runc | runc --system-cgroup |
|------------------------|--------|-----------------------|--------|----------------------|
| default config | EINVAL | EACCES | EINVAL | EINVAL |
| "cpu": { "cpus": "0" } | EINVAL | OK | OK | OK |
---
So, the main problem here comes from the fact that `crun` (and `runc`) tries to initialize newly created `cpuset.cpus` with value taken from parent cgroup. According to [Linux documentation for cgroups v2](https://docs.kernel.org/admin-guide/cgroup-v2.html#cpuset-interface-files)(!)
> An empty value indicates that the cgroup is using the same setting as the nearest cgroup ancestor with a non-empty “cpuset.cpus” or all the available CPUs if none is found.
I cannot find a similar description for cgroups v1. Is that why that initialization is needed? If so, how does one handle cgroups with exclusive cpus? Does one have to traverse the entire cpuset tree to find available cpus? If so, I feel this issue won't be fixed, given cgroups v1 are obsolete anyway.
The secondary problem seems to be that `--cgroup-manager=cgroupfs` in `crun` ignores `linux.resources.cpu.cpus`. Or rather, it will apply them only after initialization happens, as it uses `initialize_cpuset_subsystem()`. Compare that with `--cgroup-manager=systemd`, which uses `initialize_cpuset_subsystem_resources()` and therefore won't attempt putting all cpus in the `cpuset.cpus` file.
Contributor guide
Research direction
Reproduce the v1 cpuset case from the listed boot, cgroup, and crun commands, then trace initialize_cpuset_subsystem() and initialize_cpuset_subsystem_resources(). Compare cgroupfs and systemd-cgroup handling, including linux.resources.cpu.cpus. Done means container creation no longer fails for exclusive cpus and the requested CPU set is honored.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, linux
- Domain
- operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100