containerd / containerd/cgroups

support filter subsystem on `cgroup.Stat()` function

Open
#370 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.2k
Forks
252
PR merge metrics
No merged PRs in 30d

Description

When we check metrics using the **CRI interface**, it queries all subsystems. However, in practice, we might only need **CPU** and **memory** information, making the other queries unnecessary and redundant. The metrics interface is often called frequently and periodically.

The call chain is usually **ContainerStats -> cgroup.Stat(cntid)** or **PodStats -> cgroup.Stat(cndids...)**. For **PodStats**, this triggers multiple **ContainerStats** calls.

We've encountered an issue where **`/proc/partitions`** is read too frequently which will cause **spin locks** during the read process.

bt script code:

```
bpftrace -e '
BEGIN {
@start_time = nsecs;
printf("Tracing container processes opening /proc/partitions... (Start Time: %d)\n", @start_time);
}

kprobe:vfs_open
/ strncmp(comm, "containerd", 10) == 0 && str(((struct path *)arg0)->dentry->d_name.name) == "partitions" /
{
@open_count[comm, str(((struct path *)arg0)->dentry->d_name.name)] = count();
}

END {
$end_time = nsecs;
printf("\nTracing finished. (End Time: %d)\n", $end_time);

$duration = ($end_time - @start_time) / 1000000000; // 转换为秒
printf("Total duration: %d seconds\n", $duration);

printf("\nTracing results:\n");
print(@open_count);
}
'

// output
Tracing finished. (End Time: 1684141913)
Total duration: 9 seconds

Tracing results:
@open_count[containerd-shim, partitions]: 322
```

// vfs_read kstack

```
@last_stack[120]: (
klist_next+180 <------- spin_lock
class_dev_iter_next+64
disk_seqf_start+128
show_partition_start+44
seq_read+168
proc_reg_read+128
__vfs_read+72
vfs_read+156
ksys_read+108
__arm64_sys_read+36
el0_svc_handler+176
el0_svc+8
,
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.