[libc][linux][aarch64][cndvar] wait-time amplification under pessimistic CPU idle-state selection
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
For some condvar based queuing operations, I noticed that `menu` governor amplifies the wait time along the futex chain when running on Cortex-A725 on GB10.
Roughly, the problem is caused by a poor interaction between queue-based condvar in llvm-libc/musl and the idle governor. Cores go into deep sleep state so the wake up cost accumulates along the way, causing the broadcasting latency surges from run to run. Switching to multiway wakeup and removing requeue mitigate the amplification, but that means that we need to give up FIFO and introduce thundering herds. Not sure if we want to fix it in libc. Using TEO governor or manually setup idle parameters also workaround the problem. Personally, I think that kernel or micro-architecture is to blame.
If anyone runs into problem with realistic workload, please let me know.
```c
#define _GNU_SOURCE
#include
#include
#include
#include
#include
#include
enum { WORKERS = 31, WARMUP = 50, ROUNDS = 300 };
static pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t work = PTHREAD_COND_INITIALIZER;
static pthread_cond_t done = PTHREAD_COND_INITIALIZER;
static unsigned ready, generation, completed;
static int stop;
static void check(int error) {
if (error)
abort();
}
static void pin(unsigned index) {
static const int cpus[] = {0, 1, 2, 3, 4, 10, 11, 12, 13, 14};
cpu_set_t set;
CPU_ZERO(&set);
CPU_SET(cpus[index % 10], &set);
check(pthread_setaffinity_np(pthread_self(), sizeof(set), &set));
}
static double now_us(void) {
struct timespec t;
check(clock_gettime(CLOCK_MONOTONIC, &t));
return t.tv_sec * 1e6 + t.tv_nsec / 1e3;
}
static void *worker(void *arg) {
pin((unsigned)(uintptr_t)arg + 1);
unsigned seen = 0;
check(pthread_mutex_lock(&mutex));
++ready;
check(pthread_cond_signal(&done));
for (;;) {
while (!stop && seen == generation)
check(pthread_cond_wait(&work, &mutex));
if (stop)
break;
seen = generation;
if (++completed == WORKERS)
check(pthread_cond_signal(&done));
}
check(pthread_mutex_unlock(&mutex));
return NULL;
}
int main(void) {
pthread_t threads[WORKERS];
pin(0);
for (unsigned i = 0; i < WORKERS; ++i)
check(pthread_create(&threads[i], NULL, worker, (void *)(uintptr_t)i));
check(pthread_mutex_lock(&mutex));
while (ready != WORKERS)
check(pthread_cond_wait(&done, &mutex));
double start = 0;
for (unsigned i = 0; i < WARMUP + ROUNDS; ++i) {
if (i == WARMUP)
start = now_us();
completed = 0;
++generation;
check(pthread_cond_broadcast(&work));
while (completed != WORKERS)
check(pthread_cond_wait(&done, &mutex));
}
double elapsed = now_us() - start;
stop = 1;
check(pthread_cond_broadcast(&work));
check(pthread_mutex_unlock(&mutex));
for (unsigned i = 0; i < WORKERS; ++i)
check(pthread_join(threads[i], NULL));
printf("%.3f us/round\n", elapsed / ROUNDS);
}
```
NVIDIA GB10 / DGX Spark, Linux `7.0.0-1013-nvidia`, `acpi_idle` driver, `menu` governor. Three fresh processes per configuration, with run order randomized. Times are mean wall-clock microseconds per complete broadcast/acknowledgment round.
| Implementation | Idle setting | Run 1 | Run 2 | Run 3 |
|---|---|---:|---:|---:|
| musl 1.2.5 | Normal | 2,700.357 | 3,644.437 | 4,965.469 |
| musl 1.2.5 | PM QoS = 0 µs | 187.994 | 188.988 | 188.289 |
| LLVM libc in-tree | Normal | 164.859 | 166.404 | 2,378.183 |
| LLVM libc in-tree | PM QoS = 0 µs | 172.783 | 172.274 | 172.183 |
Contributor guide
Research direction
Start with the supplied C reproducer, especially main, worker, and the pthread_cond_wait/broadcast calls, and run it on the stated Cortex-A725/GB10 configuration. Compare musl and in-tree LLVM libc with normal idle settings and PM QoS set to 0 µs; done requires confirming the amplification and determining whether the actionable owner is libc, the kernel, or the idle governor.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, linux
- Domain
- operating-systems, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100