fail to detect cgroup on a LXC container
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
- run tidb-server with systemd
```
[root@lightning ~]# cat /etc/systemd/system/tidb-server.service
[Unit]
Description=tidb
[Service]
Type=simple
Restart=always
ExecStart=/root/.tiup/components/tidb/v8.0.0-alpha-nightly/tidb-server
StandardOutput=journal
CPUQuota=50%
[Install]
WantedBy=multi-user.target
```
below are the cgroup directory of tidb-server. we can see in `/proc/pid/mountinfo` there is no mount of filesystem type = `cgroup` or `cgroup2`
```
[root@lightning ~]# ls /sys/fs/cgroup/cpu/lxc.payload.tispace_jujiajia-lightning/
cgroup.clone_children cgroup.procs cpuacct.usage cpu.cfs_period_us cpu.rt_period_us cpu.shares notify_on_release tasks
cgroup.event_control cpuacct.stat cpuacct.usage_percpu cpu.cfs_quota_us cpu.rt_runtime_us cpu.stat system.slice/ user.slice/
[root@lightning ~]# ls /sys/fs/cgroup/cpu/lxc.payload.tispace_jujiajia-lightning/system.slice/tidb-server.service/
cgroup.clone_children cgroup.procs cpuacct.usage cpu.cfs_period_us cpu.rt_period_us cpu.shares notify_on_release
cgroup.event_control cpuacct.stat cpuacct.usage_percpu cpu.cfs_quota_us cpu.rt_runtime_us cpu.stat tasks
[root@lightning ~]# cat /sys/fs/cgroup/cpu/lxc.payload.tispace_jujiajia-lightning/system.slice/tidb-server.service/cpu.cfs_quota_us
50000
[root@lightning ~]# cat /sys/fs/cgroup/cpu/lxc.payload.tispace_jujiajia-lightning/system.slice/tidb-server.service/cpu.cfs_period_us
100000
```
```
[root@lightning ~]# systemctl status tidb-server
● tidb-server.service - tidb
Loaded: loaded (/etc/systemd/system/tidb-server.service; enabled; vendor preset: disabled)
Drop-In: /run/systemd/system/tidb-server.service.d
└─zzz-lxc-service.conf
Active: active (running) since 三 2024-03-13 20:30:41 CST; 13h ago
Main PID: 278350 (tidb-server)
CGroup: /lxc.payload.tispace_jujiajia-lightning/system.slice/tidb-server.service
└─278350 /root/.tiup/components/tidb/v8.0.0-alpha-nightly/tidb-server
3月 14 10:26:31 lightning tidb-server[278350]: [2024/03/14 10:26:31.896 +08:00] [WARN] [cgmon.go:103] ["failed to get cgroup cpu quota"] [error="failed to detect cgroup root mount and version"]
3月 14 10:26:31 lightning tidb-server[278350]: [2024/03/14 10:26:31.896 +08:00] [WARN] [cgmon.go:128] ["failed to get cgroup memory limit"] [error="failed to detect cgroup root mount and version"]
3月 14 10:26:41 lightning tidb-server[278350]: [2024/03/14 10:26:41.896 +08:00] [WARN] [cgmon.go:103] ["failed to get cgroup cpu quota"] [error="failed to detect cgroup root mount and version"]
3月 14 10:26:41 lightning tidb-server[278350]: [2024/03/14 10:26:41.896 +08:00] [WARN] [cgmon.go:128] ["failed to get cgroup memory limit"] [error="failed to detect cgroup root mount and version"]
3月 14 10:26:51 lightning tidb-server[278350]: [2024/03/14 10:26:51.896 +08:00] [WARN] [cgmon.go:103] ["failed to get cgroup cpu quota"] [error="failed to detect cgroup root mount and version"]
3月 14 10:26:51 lightning tidb-server[278350]: [2024/03/14 10:26:51.896 +08:00] [WARN] [cgmon.go:128] ["failed to get cgroup memory limit"] [error="failed to detect cgroup root mount and version"]
3月 14 10:27:01 lightning tidb-server[278350]: [2024/03/14 10:27:01.896 +08:00] [WARN] [cgmon.go:103] ["failed to get cgroup cpu quota"] [error="failed to detect cgroup root mount and version"]
3月 14 10:27:01 lightning tidb-server[278350]: [2024/03/14 10:27:01.897 +08:00] [WARN] [cgmon.go:128] ["failed to get cgroup memory limit"] [error="failed to detect cgroup root mount and version"]
3月 14 10:27:11 lightning tidb-server[278350]: [2024/03/14 10:27:11.897 +08:00] [WARN] [cgmon.go:103] ["failed to get cgroup cpu quota"] [error="failed to detect cgroup root mount and version"]
3月 14 10:27:11 lightning tidb-server[278350]: [2024/03/14 10:27:11.897 +08:00] [WARN] [cgmon.go:128] ["failed to get cgroup memory limit"] [error="failed to detect cgroup root mount and version"]
[root@lightning ~]# cat /proc/278350/cgroup
11:cpuset:/lxc.payload.tispace_jujiajia-lightning
10:perf_event:/lxc.payload.tispace_jujiajia-lightning
9:memory:/lxc.payload.tispace_jujiajia-lightning
8:freezer:/lxc.payload.tispace_jujiajia-lightning
7:hugetlb:/lxc.payload.tispace_jujiajia-lightning
6:cpuacct,cpu:/lxc.payload.tispace_jujiajia-lightning/system.slice/tidb-server.service
5:devices:/lxc.payload.tispace_jujiajia-lightning/system.slice
4:pids:/lxc.payload.tispace_jujiajia-lightning
3:net_prio,net_cls:/lxc.payload.tispace_jujiajia-lightning
2:blkio:/lxc.payload.tispace_jujiajia-lightning
1:name=systemd:/lxc.payload.tispace_jujiajia-lightning/system.slice/tidb-server.service
[root@lightning ~]# cat /proc/278350/mountinfo
2060 198 253:5 /rootfs / rw,relatime master:81 - ext4 /dev/vg3/containers_tispace_jujiajia--lightning rw,discard,data=ordered
2062 2060 0:140 / /dev rw,relatime - tmpfs none rw,size=492k,mode=755,uid=1000000,gid=1000000
2063 2060 0:139 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
2064 2060 0:188 / /sys rw,relatime - sysfs sysfs rw
2065 2062 0:5 /fuse /dev/fuse rw,nosuid master:2 - devtmpfs devtmpfs rw,size=197320476k,nr_inodes=49330119,mode=755
2066 2062 0:5 /net/tun /dev/net/tun rw,nosuid master:2 - devtmpfs devtmpfs rw,size=197320476k,nr_inodes=49330119,mode=755
2067 2063 0:52 / /proc/sys/fs/binfmt_misc rw,relatime master:77 - binfmt_misc binfmt_misc rw
2068 2064 0:46 / /sys/fs/fuse/connections rw,relatime master:75 - fusectl fusectl rw
2069 2064 0:23 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime master:20 - pstore pstore rw
2070 2064 0:34 / /sys/kernel/config rw,relatime master:21 - configfs configfs rw
2071 2064 0:6 / /sys/kernel/debug rw,relatime master:26 - debugfs debugfs rw
2072 2064 0:17 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime master:7 - securityfs securityfs rw
2073 2062 0:15 / /dev/mqueue rw,relatime master:24 - mqueue mqueue rw
2074 2062 0:47 / /dev/lxd rw,relatime - tmpfs tmpfs rw,size=100k,mode=755
2075 2062 0:45 /tispace_jujiajia-lightning /dev/.lxd-mounts rw,relatime master:72 - tmpfs tmpfs rw,size=100k,mode=711
2076 2064 0:189 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - tmpfs none rw,size=10240k,mode=755,uid=1000000,gid=1000000
2111 2063 0:44 /proc/cpuinfo /proc/cpuinfo rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2112 2063 0:44 /proc/diskstats /proc/diskstats rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2113 2063 0:44 /proc/loadavg /proc/loadavg rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2114 2063 0:44 /proc/meminfo /proc/meminfo rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2115 2063 0:44 /proc/slabinfo /proc/slabinfo rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2116 2063 0:44 /proc/stat /proc/stat rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2117 2063 0:44 /proc/swaps /proc/swaps rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2118 2063 0:44 /proc/uptime /proc/uptime rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2119 2064 0:44 /sys/devices/system/cpu/online /sys/devices/system/cpu/online rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2077 2076 0:44 /cgroup/blkio /sys/fs/cgroup/blkio rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2078 2076 0:44 /cgroup/cpu /sys/fs/cgroup/cpu rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2079 2076 0:44 /cgroup/cpuset /sys/fs/cgroup/cpuset rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2080 2076 0:44 /cgroup/devices /sys/fs/cgroup/devices rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2081 2076 0:44 /cgroup/freezer /sys/fs/cgroup/freezer rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2082 2076 0:44 /cgroup/hugetlb /sys/fs/cgroup/hugetlb rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2083 2076 0:44 /cgroup/memory /sys/fs/cgroup/memory rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2084 2076 0:44 /cgroup/net_cls /sys/fs/cgroup/net_cls rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2085 2076 0:44 /cgroup/perf_event /sys/fs/cgroup/perf_event rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2086 2076 0:44 /cgroup/pids /sys/fs/cgroup/pids rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2087 2076 0:44 /cgroup/systemd /sys/fs/cgroup/systemd rw,nosuid,nodev,relatime master:63 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2088 2062 0:5 /full /dev/full rw,nosuid master:2 - devtmpfs devtmpfs rw,size=197320476k,nr_inodes=49330119,mode=755
2101 2062 0:5 /null /dev/null rw,nosuid master:2 - devtmpfs devtmpfs rw,size=197320476k,nr_inodes=49330119,mode=755
2102 2062 0:5 /random /dev/random rw,nosuid master:2 - devtmpfs devtmpfs rw,size=197320476k,nr_inodes=49330119,mode=755
2103 2062 0:5 /tty /dev/tty rw,nosuid master:2 - devtmpfs devtmpfs rw,size=197320476k,nr_inodes=49330119,mode=755
2104 2062 0:5 /urandom /dev/urandom rw,nosuid master:2 - devtmpfs devtmpfs rw,size=197320476k,nr_inodes=49330119,mode=755
2105 2062 0:5 /zero /dev/zero rw,nosuid master:2 - devtmpfs devtmpfs rw,size=197320476k,nr_inodes=49330119,mode=755
2106 2062 0:190 / /dev/pts rw,nosuid,noexec,relatime - devpts /proc/self/fd/38 rw,gid=1000005,mode=620,ptmxmode=666,max=1024
2107 2062 0:190 /ptmx /dev/ptmx rw,nosuid,noexec,relatime - devpts /proc/self/fd/38 rw,gid=1000005,mode=620,ptmxmode=666,max=1024
2108 2062 0:190 /0 /dev/console rw,nosuid,noexec,relatime - devpts /proc/self/fd/38 rw,gid=1000005,mode=620,ptmxmode=666,max=1024
2109 2063 0:140 /.lxc-boot-id /proc/sys/kernel/random/boot_id ro,nosuid,nodev,noexec,relatime - tmpfs none rw,size=492k,mode=755,uid=1000000,gid=1000000
203 2062 0:191 / /dev/shm rw,nosuid,nodev - tmpfs tmpfs rw,uid=1000000,gid=1000000
218 2060 0:192 / /run rw,nosuid,nodev - tmpfs tmpfs rw,mode=755,uid=1000000,gid=1000000
222 218 0:270 / /run/user/0 rw,nosuid,nodev,relatime - tmpfs tmpfs rw,size=3355444k,mode=700,uid=1000000,gid=1000000
```
### 2. What did you expect to see? (Required)
detect and can read limit from cgroup
### 3. What did you see instead (Required)
```
3月 13 20:30:41 lightning tidb-server[278350]: [2024/03/13 20:30:41.895 +08:00] [INFO] [printer.go:47] ["Welcome to TiDB."] ["Release Version"=v8.0.0-alpha-479-gcd60e7f] [Edition=Community] ["Git Commit H
ash"=cd60e7fadb0a4f251ffb61464b9885c003756e02] ["Git Branch"=HEAD] ["UTC Build Time"="2024-03-13 07:01:24"] [GoVersion=go1.21.4] ["Race Enabled"=false] ["Check Table Before Drop"=false]
3月 13 20:30:41 lightning tidb-server[278350]: [2024/03/13 20:30:41.898 +08:00] [ERROR] [cpu.go:67] [GetCgroupCPU] [error="failed to detect cgroup root mount and version"]
3月 13 20:30:51 lightning tidb-server[278350]: [2024/03/13 20:30:51.897 +08:00] [WARN] [cgmon.go:103] ["failed to get cgroup cpu quota"] [error="failed to detect cgroup root mount and version"]
3月 13 20:30:51 lightning tidb-server[278350]: [2024/03/13 20:30:51.897 +08:00] [WARN] [cgmon.go:128] ["failed to get cgroup memory limit"] [error="failed to detect cgroup root mount and version"]
```
### 4. What is your TiDB version? (Required)
master
Contributor guide
Assessment
This issue has not been assessed yet.