openai / openai/codex

[Linux] Renderer pinned at 100% CPU: heartbeat messages re-parsed from inside a store equality comparator on every incoming message

Open
#45,827 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

app automations bug Linux performance
Dominant language
Rust
Stars
125k
Forks
19.4k
PR merge metrics
PR metrics pending

Description

What version of the Codex App are you using (From "About Codex" dialog)?

26.901.20858 (Arch package chatgpt 26.901.20858-1) — Electron 42.3.0, Chromium 152.0.7977.64, V8 15.2.124.18

What subscription do you have?

Paid plan (exact tier not confirmed in this report)

What platform is your computer?

Linux 7.1.9-arch1-2 x86_64 unknown — Wayland, 8 cores / 31 GB

What issue are you seeing?

A renderer process pins one core at ~100–125% CPU for minutes at a time. The window stops responding; the JS heap oscillates between ~750 MB and ~3186 MB. It recurs every few minutes on a long-lived conversation.

This is not a resource problem: ~20 GB RAM available, /proc/pressure near zero, no OOM kills, no crashpad dumps, no coredumps. One core is pinned, the other seven idle.

I profiled it during a live wedge and can name the hot path.

A sampling profile cannot be taken the usual wayProfiler.stop needs the very JS thread that is wedged, and Runtime.enable times out. I used browser-process-driven tracing instead (Tracing.start/Tracing.end on ws://127.0.0.1:9222/devtools/browser/…, category disabled-by-default-v8.cpu_profiler, ring buffer), which does not need the renderer's main thread to respond.

Renderer pid 452260, single isolate, 12.4 s busy out of a 20 s window:

 36.2%  onMessage      (self time, no JS children -> native: message parse)
 34.8%  oSt            (heartbeat parser)
  4.2%  getBoundingClientRect
  1.4%  (garbage collector)

Rendering is only 7% of busy time (132 React render passes in 20 s). The render path is not what saturates the thread.

oSt is reached by two paths, roughly equal:

17.6%   X4t -> TS -> bS -> (anon) -> oSt                        (per incoming message)

17.1%   j4t -> updateConversationState -> notifyConversationCallbacks
          -> emitConversation -> ... -> uy.set -> iBt -> Yzt -> Xzt -> read
          -> Ny.isEqual -> Iy.isEqual -> b_r -> bS -> (anon) -> oSt   (inside an equality check)

The second path is the important one: a derived value is recomputed inside a store equality comparator.

What steps can reproduce the bug?

Keep one conversation alive for several days with an hourly heartbeat automation writing into it. In my case: 14 337 items / 88 MB, including 647 heartbeat messages (rrule = FREQ=HOURLY;BYMINUTE=0,15,30,45). The wedges begin once the thread is large and recur every few minutes while a turn streams.

Verbatim from the shipped bundle app://-/assets/app-initial-d237ce07a86a.js (no source maps are shipped in app.asar, so these are minified names):

function cSt(e,t){return RegExp(`<${t}>\\s*([\\s\\S]*?)\\s*<\\/${t}>`,`i`).exec(e)?.[1].trim()??null}

function oSt(e){let t=e.trim();
  if(!t.startsWith(`<heartbeat>`)||!t.endsWith(`</heartbeat>`))return null;
  let n=cSt(t,`current_time_iso`),r=cSt(t,`instructions`);
  return n==null||r==null?null:
    {automationId:cSt(t,`automation_id`),currentTimeIso:n,instructions:r}}   // new object every call
  • cSt compiles a fresh RegExp on every call — three calls per heartbeat item, so three compilations per item per pass. The pattern depends only on the tag name, which is one of three constants.
  • oSt returns a freshly allocated object literal on every call.
  • bS(e,t,n) is a full list reconciliation: it builds a Set and a Map over both item lists and flatMaps the result — O(total items) per call.

So per incoming stream message the thread performs at least two full passes over the entire conversation item list, each re-parsing every heartbeat message from scratch. Messages arrive continuously while a turn streams, and updateConversationState -> notifyConversationCallbacks -> emitConversation -> bS is itself a notify cycle.

The same anti-pattern exists in a second, duplicated copyOun, used for <realtime_delegation> voice messages with tags input, transcript_delta, source:

function Oun(e,t){let n=RegExp(`<${t}>\\s*([\\s\\S]*?)\\s*<\\/${t}>`,`i`).exec(e)?.[1].trim()??null;return n==null?null:kun(n)}

In an earlier trace of a different wedge, RegExp: <transcript_delta> was the top self-time entry at 10.3%, with GC at 7.9%. This is not specific to one feature.

What is the expected behavior?

Settled messages are parsed once. Smallest change with the largest effect, in order:

  1. Do not recompute derived values inside an equality comparator. oSt (and the bS reconciliation it sits in) should not be reachable from Iy.isEqual. Compare cheap identities (item id + revision) instead.
  2. Memoize oSt/sSt and bun/yun per item (id + content). A heartbeat message is immutable once written; it never needs re-parsing.
  3. Hoist the regexes in cSt and Oun to module constants — both rebuild them on every call, and both are duplicates of the same helper.
  4. Make the per-message update path incremental rather than O(total items).
Additional information

Measured, not guessed — and here is what I could not establish. I did not capture which conversation the wedged renderer was displaying; the renderer recovered and no longer held that state. And 6 ms/render of parsing (measured standalone, see below) does not by itself explain a wedge sustained at 100% for minutes — something drives the cycle continuously. The profile shows where the time goes once that is happening; it does not prove what starts it.

Ruled out by benchmark — please do not chase these:

  • Catastrophic backtracking. <TAG>\s*([\s\S]*?)\s*</TAG> was measured from 32 KB to 512 KB in three shapes: well-formed, missing closing tag, and missing closing tag with long whitespace runs. All linear, all under 7 ms. The pattern is not the problem; the call frequency is.
  • Conversation database size. Archiving the history from 1282 MB down to 669 MB changed nothing.
  • Voice capture. Wedges occur with the realtime session closed and zero microphone capture streams (pactl source-output count 0), and one occurred with no voice session that day at all. It is the stored transcript text being re-parsed, not live audio.
  • Context compaction. Compactions at 04:46 and 08:01 produced no runaway, and one at 10:27 came after the 10:26 runaway began.

A tracing caveat that cost me hours: with toplevel + devtools.timeline + v8.execute categories enabled, 79.4% of the profile falls into unattributed (program) and the renderer reports 5 V8 isolates. With disabled-by-default-v8.cpu_profiler alone, (program) disappears and the profile is readable. Also attribute samples per tid — each isolate has its own v8:ProfEvntProc thread, and summing across them mixes workers into the main thread's numbers.

Standalone benchmark of the voice-side helpers running verbatim in plain Node 26 (same V8 major), at realistic sizes:

   1 voice message   ->   0.0 ms per render
 203 voice messages  ->   1.4 ms per render
 911 voice messages  ->   6.0 ms per render

Linear in message count — pure overhead re-deriving data that has not changed.

Possibly related: #36752 reports the same shape on Windows (Git Changes panel recalculating a very large branch comparison on every event). Different subsystem, same class of defect: expensive work redone on every event instead of being cached.

Impact. This is a daily-driver machine running long-lived agent sessions. Before I put a watchdog in place that kills only the runaway renderer, the app required manual restarts every 30–90 minutes, and in-flight work was lost each time. I am happy to share the ~120-line tracer script or the raw trace if useful.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with app://-/assets/app-initial-d237ce07a86a.js and trace oSt, cSt, Oun, bS, and the Iy.isEqual path described in the report; no source maps are shipped. Confirm where settled heartbeat and realtime-delegation items are repeatedly parsed, then verify that unchanged items are not re-derived during equality checks and that the renderer no longer performs repeated full-list work during streaming.

Written by the indexing model from the issue text.

Assessment

Tech stack
electron, javascript
Domain
desktop, frontend, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.