app-server 0.153.4: 1.18M futex calls/sec on confirmed musl __malloc_lock, >50% kernel CPU

app-server 0.153.4: 1.18M futex calls/sec on confirmed musl __malloc_lock, >50% kernel CPU

What issue are you seeing?

The released Linux Codex Rust app-server develops a severe allocator contention storm across Tokio workers. A fresh 45-second capture recorded 53,098,607 futex calls against musl's global __malloc_lock: approximately 1.18 million calls/sec, and 99.71% of all futex calls. During an overlapping 15-second CPU-accounting window, the app-server consumed 5.82 CPU cores, with 53.82% of its CPU time in the kernel.

This is active synchronization overhead, not merely idle threads sleeping in futexes. Most sampled waits return EAGAIN without sleeping, most wakes wake nobody, and an independent on-CPU profile shows musl lock operations, allocator work, futex kernel paths, and kernel spinlock contention.

Related: #41312 reports a musl allocator storm in CLI/TUI 0.150.1. This report supplies an app-server 0.153.4 reproduction, exact release-symbol identification of __malloc_lock, and application-to-kernel call stacks. These may share the same underlying allocator defect; please consolidate if preferred. This report does not claim a separate allocator defect or a new regression boundary.

Environment

• codex-cli 0.153.4, released x86_64-unknown-linux-musl binary, installed through @openai/codex-linux-x64.
• Launch: codex -c features.code_mode_host=true app-server --listen unix://
• Fedora Linux 44, kernel 7.1.10-200.fc44.x86_64.
• Intel Core i9-7920X, 12 cores / 24 logical CPUs; process allowed CPUs 0-23.
• 82 threads at the beginning of the fresh capture; hot threads are tokio-rt-worker.
• Desktop-connected app-server with numerous MCP/plugin tools and code-mode activity. This was a live active workload, not a controlled idle test. No single triggering prompt or minimal application-level reproducer has been isolated.
• Capture: 2026-09-06, 06:27:53Z–06:28:45Z, including 5-second discovery, 45-second targeted capture, and overhead.
• Installed executable SHA-256: 56ef98ab4032d317ab26e9b5e5a175650717351edb16ed9cde0cb6d1734d62da.

Fresh futex capture

The capture instruments sys_enter_futex for this process, counts all operations and addresses, samples target-address entry/exit pairs at 1/4096, and sparsely samples user stacks at 1/65536. Counts below are from the explicit main-capture sections, excluding discovery and any residual maps printed at shutdown.

Measurement	Result
All futex calls	53,253,315
Calls against __malloc_lock	53,098,607
Share at that address	99.7095%
Approximate target rate	1,179,969/sec
Target FUTEX_WAIT calls	17,879,652
Target FUTEX_WAKE calls	35,218,955
Sampled WAIT returns: EAGAIN	3,634 / 4,304 = 84.43%
Sampled WAIT returns: success	670 / 4,304
Sampled WAKE returns: nobody woken	8,078 / 8,692 = 92.94%
Sampled WAKE returns: one waiter woken	614 / 8,692

EAGAIN means the futex word changed before the kernel could sleep. WAKE returning zero means no kernel-blocked waiter was woken. These are syscall counts and sampled outcomes; they are not measurements of total blocked duration or per-syscall CPU time.

Exact lock identification using matching release symbols

The running binary is stripped, but its .gnu_debuglink points to codex.debug. I downloaded the 0.153.4 Linux symbols and verified the GitHub release asset SHA-256:

4564f74e132c123ed511c71f185ba82e4f918dd49b3bcd470c7a6301eab6ab65

The extracted codex.debug also matches the executable's debuglink CRC32, 0xa528bd4b.

PIE load bias:                   0x7f644c400000
Dominant live futex address:      0x7f645bab7f30
ELF-relative address:             0x000000000f6b7f30
nm result:                       000000000f6b7f30 B __malloc_lock

Sampled WAIT return instruction:  0x000000000c5dd5be
  __futexwait -> __lock
  musl src/thread/__lock.c:44

Sampled WAKE return instruction:  0x000000000c5dd5fc
  __wake -> __unlock
  musl src/thread/__lock.c:59

Disassembly confirms FUTEX_WAIT_PRIVATE and FUTEX_WAKE_PRIVATE with wake count 1. The dominant address is in the executable's static zero-initialized storage, not a SQLite shared-memory mapping or a per-task heap mutex.

CPU evidence

Per-thread /proc/PID/task/TID/stat user/system deltas, summed only for the target process over 15.0098 seconds:

User CPU:                 40.34 seconds
System CPU:               47.02 seconds
Total CPU utilization:    582.02% (5.82 cores)
System fraction:          53.82%

That window overlapped BPF tracing, so instrumentation can affect the absolute CPU cost. A subsequent separate 20-second perf capture, with BPF tracing finished, independently shows the same allocator/futex/spinlock behavior. Both useful perf captures used --no-inherit so child tool processes do not contaminate app-server attribution.

Userspace-only profile (cycles:u, 99 Hz, 15 seconds, 3,472 samples, zero lost samples), selected self percentages:

Symbol	User-cycle profile share
__libc_free	13.70%
__libc_malloc_impl	12.02%
__lock	10.55%
memcpy	8.18%
alloc_slot	8.17%
__unlock	5.11%
rust_stemmers::snowball::snowball_env::SnowballEnv::find_among_b	3.58%

Separate user+kernel profile (cycles, 99 Hz, 20 seconds, 9,011 samples, zero lost samples), selected self percentages:

Symbol	All-cycle profile share
native_queued_spin_lock_slowpath	8.43%
__lock	7.98%
syscall_return_via_sysret	7.94%
entry_SYSRETQ_unsafe_stack	6.89%
__libc_malloc_impl	5.64%
__libc_free	4.95%
__unlock	3.03%
futex_hash	2.02%
futex_wake	1.32%
futex_q_lock	1.12%
futex_wait_setup	1.00%

The two percentage columns have different denominators and were recorded in separate windows; they must not be added together. Kernel CPU also includes work other than futexes.

Application call paths recovered from DWARF stacks

Representative sampled allocation path, caller to callee, with intermediate formatting/collection frames omitted:

ToolRegistry::dispatch_any_with_terminal_outcome
  CodeModeExecuteHandler::handle_call
    McpHandler::cached_code_mode_definitions
      OnceLock::initialize
        codex_tools::code_mode::collect_code_mode_tool_definitions
          render_code_mode_sample_for_definition
            render_code_mode_sample
              alloc::fmt::format / String::write_str / RawVec growth
                __libc_realloc
                  __libc_malloc_impl
                    __lock

The exact release source has the per-handler cache and synchronous definition construction in McpHandler::cached_code_mode_definitions. Other recovered stacks show JsonSchemaTypeRenderer working under that initialization.

A separate representative stack shows a different subsystem reaching the same contended allocator while tearing down a models HTTP client:

OpenAiModelsEndpoint::list_models
  drop ModelsClient / ReqwestTransport / reqwest Connector
    SSL_CTX_free
      X509_STORE_free / certificate ASN.1 cleanup
        __libc_free
          __lock
            __x64_sys_futex
              futex_wait
                futex_wait_setup
                  _raw_spin_lock
                    native_queued_spin_lock_slowpath

These stacks identify actual participants, not proof that either call path alone causes the entire storm. English stemming is also visible in the profile; repeated tool-search indexing is a candidate contributor, but cache invalidation/rebuild frequency has not been measured. The confirmed finding is contention on musl __malloc_lock shared by concurrent allocator users.

What steps can reproduce or diagnose the bug?

1. Run the released 0.153.4 musl binary as a desktop-connected app-server, with code mode enabled and a substantial MCP/plugin tool catalog.
2. Exercise concurrent turns/tool activity until the Rust app-server shows sustained CPU use. This describes the observed workload, not a guaranteed minimal trigger.
3. Capture the affected app-server PID, rather than the Electron process or a child tool. On this run I used bash ~/codex-futex-capture.sh 2446393 45; the essential probes were per-PID sys_enter_futex address/op/TID counts and sparse target-address sys_exit_futex outcomes.
4. For an independently usable initial probe, replace PID below and record counts plus measured elapsed time:

sudo bpftrace -e '
BEGIN { @start = nsecs; }
tracepoint:syscalls:sys_enter_futex /pid == PID/ {
  @addresses[(uint64)args.uaddr] = count();
  @operations[args.op & 0x7f] = count();
}
interval:s:1 /nsecs - @start >= 45000000000/ {
  printf("elapsed_ns=%llu\n", nsecs - @start);
  print(@addresses, 10);
  print(@operations);
  clear(@addresses); clear(@operations); clear(@start);
  exit();
}'

5. Resolve the dominant address against the matching release's codex.debug, accounting for PIE load bias. Do not reuse this report's absolute address after a restart.
6. Capture on-CPU stacks separately from BPF tracing:

sudo perf record --no-inherit -F 99 -g --call-graph dwarf,16384 \
  -e cycles -p "$PID" -o codex.perf.data -- sleep 20

Use matching codex.debug and kernel symbols when reporting. Plain frame-pointer ustack unwinding of the stripped release produced unreliable deeper frames; the representative application stacks above come from DWARF perf capture instead.

Expected behavior and investigation direction

Concurrent app-server work should not spend millions of syscalls per second fighting the process-wide allocator lock. Increasing available worker parallelism should not create this level of synchronization overhead or delay ordinary tool/turn progress.

Please investigate the allocator configuration of the shipped musl app-server and concurrent allocation volume. The concrete call paths above provide places to inspect tool-definition construction/caching and HTTP-client/context lifetimes. An allocator comparison against the same workload would help quantify the underlying defect; no allocator substitution, reduced-worker experiment, or remediation was performed in this report.

Start with codex-rs/core/src/tools/handlers/mcp.rs at McpHandler::cached_code_mode_definitions and inspect the shipped musl allocator configuration and concurrent allocation paths. Reproduce or diagnose the issue with the provided bpftrace futex probe and separate DWARF perf capture, using the matching codex.debug symbols. Done should include measured allocator contention and an evidence-based remediation direction; no specific fix is defined yet.