cloudflare / cloudflare/ebpf_exporter

A suspicious multi-thread race condition of cfs-throttling-trace.bpf.c

Open
#442 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.6k
Forks
282
Avg merge
5d 7h
Merged PRs (30d)
8

Description

In `examples/cfs-throttling-trace.bpf.c` we use cgroup id as the key of `traced_cgroups` map :
```c
SEC("usdt/./tracing/demos/cfs-throttling/demo:ebpf_exporter:cfs_set_parent_span")
int BPF_USDT(cfs_set_parent_span, u64 trace_id_hi, u64 trace_id_lo, u64 span_id)
{
u32 cgroup = bpf_get_current_cgroup_id();
struct span_parent_t parent = { .trace_id_hi = trace_id_hi, .trace_id_lo = trace_id_lo, .span_id = span_id };

bpf_map_update_elem(&traced_cgroups, &cgroup, &parent, BPF_ANY);

return 0;
}
```

When unthrottle, we use cgroup id as key to get from the map :
```c
SEC("fentry/unthrottle_cfs_rq")
int BPF_PROG(unthrottle_cfs_rq, struct cfs_rq *cfs_rq)
{
u32 cgroup = cfs_rq->tg->css.cgroup->kn->id;
u64 throttled_ns = cfs_rq->rq->clock - cfs_rq->throttled_clock;
struct span_parent_t *parent = bpf_map_lookup_elem(&traced_cgroups, &cgroup);
```

But in a multi-threaded environment. It seems not aways true. May be thread id and cgroup id be composited as key?

Contributor guide

Open the contributing guide

Research direction

Start with examples/cfs-throttling-trace.bpf.c and trace how cgroup ids are used when updating and looking up traced_cgroups. Reproduce the mismatch in a multi-threaded workload, then determine whether the map key remains valid across the two entry points. Done means the race is explained and the issue has a verified keying or synchronization direction.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, linux
Domain
observability-sre, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.