llvm / llvm/llvm-project

[libc][linux][aarch64][cndvar] wait-time amplification under pessimistic CPU idle-state selection

Open
#224,142 1 comment 0 reactions 0 assignees View on GitHub
libc
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

For some condvar based queuing operations, I noticed that `menu` governor amplifies the wait time along the futex chain when running on Cortex-A725 on GB10.

Roughly, the problem is caused by a poor interaction between queue-based condvar in llvm-libc/musl and the idle governor. Cores go into deep sleep state so the wake up cost accumulates along the way, causing the broadcasting latency surges from run to run. Switching to multiway wakeup and removing requeue mitigate the amplification, but that means that we need to give up FIFO and introduce thundering herds. Not sure if we want to fix it in libc. Using TEO governor or manually setup idle parameters also workaround the problem. Personally, I think that kernel or micro-architecture is to blame.

If anyone runs into problem with realistic workload, please let me know.

```c
#define _GNU_SOURCE
#include
#include
#include
#include
#include
#include

enum { WORKERS = 31, WARMUP = 50, ROUNDS = 300 };
static pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t work = PTHREAD_COND_INITIALIZER;
static pthread_cond_t done = PTHREAD_COND_INITIALIZER;
static unsigned ready, generation, completed;
static int stop;

static void check(int error) {
if (error)
abort();
}

static void pin(unsigned index) {
static const int cpus[] = {0, 1, 2, 3, 4, 10, 11, 12, 13, 14};
cpu_set_t set;
CPU_ZERO(&set);
CPU_SET(cpus[index % 10], &set);
check(pthread_setaffinity_np(pthread_self(), sizeof(set), &set));
}

static double now_us(void) {
struct timespec t;
check(clock_gettime(CLOCK_MONOTONIC, &t));
return t.tv_sec * 1e6 + t.tv_nsec / 1e3;
}

static void *worker(void *arg) {
pin((unsigned)(uintptr_t)arg + 1);
unsigned seen = 0;
check(pthread_mutex_lock(&mutex));
++ready;
check(pthread_cond_signal(&done));
for (;;) {
while (!stop && seen == generation)
check(pthread_cond_wait(&work, &mutex));
if (stop)
break;
seen = generation;
if (++completed == WORKERS)
check(pthread_cond_signal(&done));
}
check(pthread_mutex_unlock(&mutex));
return NULL;
}

int main(void) {
pthread_t threads[WORKERS];
pin(0);
for (unsigned i = 0; i < WORKERS; ++i)
check(pthread_create(&threads[i], NULL, worker, (void *)(uintptr_t)i));

check(pthread_mutex_lock(&mutex));
while (ready != WORKERS)
check(pthread_cond_wait(&done, &mutex));
double start = 0;
for (unsigned i = 0; i < WARMUP + ROUNDS; ++i) {
if (i == WARMUP)
start = now_us();
completed = 0;
++generation;
check(pthread_cond_broadcast(&work));
while (completed != WORKERS)
check(pthread_cond_wait(&done, &mutex));
}
double elapsed = now_us() - start;
stop = 1;
check(pthread_cond_broadcast(&work));
check(pthread_mutex_unlock(&mutex));

for (unsigned i = 0; i < WORKERS; ++i)
check(pthread_join(threads[i], NULL));
printf("%.3f us/round\n", elapsed / ROUNDS);
}
```

NVIDIA GB10 / DGX Spark, Linux `7.0.0-1013-nvidia`, `acpi_idle` driver, `menu` governor. Three fresh processes per configuration, with run order randomized. Times are mean wall-clock microseconds per complete broadcast/acknowledgment round.

| Implementation | Idle setting | Run 1 | Run 2 | Run 3 |
|---|---|---:|---:|---:|
| musl 1.2.5 | Normal | 2,700.357 | 3,644.437 | 4,965.469 |
| musl 1.2.5 | PM QoS = 0 µs | 187.994 | 188.988 | 188.289 |
| LLVM libc in-tree | Normal | 164.859 | 166.404 | 2,378.183 |
| LLVM libc in-tree | PM QoS = 0 µs | 172.783 | 172.274 | 172.183 |

Contributor guide

Open the contributing guide

Research direction

Start with the supplied C reproducer, especially main, worker, and the pthread_cond_wait/broadcast calls, and run it on the stated Cortex-A725/GB10 configuration. Compare musl and in-tree LLVM libc with normal idle settings and PM QoS set to 0 µs; done requires confirming the amplification and determining whether the actionable owner is libc, the kernel, or the idle governor.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, linux
Domain
operating-systems, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.