`LocalQueue.enqueueBatch` spins forever on a dangling steal tag, and the queue becomes permanently unstealable

Summary

If a LocalQueue's steal tag is ever left pointing at a steal that no thread is performing, nothing in the runtime repairs it, and the pool degrades to a permanent stop:

1. enqueueBatch spins in an unbounded while (true) whose only exit is tail - stealTag <= LocalQueueCapacityMinusBatch, a condition only a completing steal can restore. Any worker that touches that queue with a batch burns 100% CPU forever.
2. enqueue computes free capacity from the steal tag too, so the queue looks permanently full and every fiber offered to it spills to the external queue as a batch — manufacturing the batches that trap workers in (1).
3. stealInto refuses the queue outright (if (steal != real) return null), so it is permanently unstealable and searching workers are funnelled into the external-queue fallback, which is a call site of enqueueBatch.
4. A worker trapped via WorkStealingThreadPool.stealFromOtherWorkerThread keeps searching status for the life of the process. notifyShouldWakeup requires (st & SearchMask) == 0, so one latched searcher is enough to stop parked workers ever being notified again, and stealTimers — reachable only from the searching path — stops rescuing timers.

A single corrupted 16-bit field is therefore unrecoverable for the lifetime of the JVM.

Version: cats-effect 3.7.0, Scala 3.3.7, Corretto JDK 25.0.2, Linux x86_64. LocalQueue.scala is byte-identical from v3.6.0 through series/3.x HEAD (checked 2026-08-26), so this applies to 3.7.1 and current series/3.x as well.

⚠️ We can reproduce the consequence chain, not the cause. What plants the tag in a live process is open — see What we could not determine below.

Reproduction

StealTagWedge.scala — fault injection, no library changes. It sets Head.head on live LocalQueues so the steal tag trails the real head by more than 225, with no steal in flight. Everything else is an ordinary IORuntime.global under load. A heartbeat fiber (IO.sleep(100.millis) in a loop) measures timer delivery, and the run deadline is enforced by a plain OS thread so it cannot itself be starved.

62 runs on a 4-vCPU machine (2 physical cores + hyperthreading, threadCount = 4):

arm	poisoned queues	runs	verdict
control (identical load, no injection)	0 of 4	12	12 healthy, full timer rate
inject	1 of 4	4	4 healthy — one trapped worker is survivable
inject	2 of 4	27	19 healthy, 8 wedged or degraded
inject	4 of 4	19	19 wedged — zero timer deliveries, permanent

Every run in which a spinner was observed at stealFromOtherWorkerThread wedged or degraded. A variant that stops feeding the external queue at injection time — so live workers must search, and can then reach stealTimers — still wedges at 4 of 4 queues, so the result is not an artifact of the feeder.

A representative wedged run with 2 of 4 queues poisoned:

t= 8s ticks=80 (+0) spinners=2/4 via[1@WorkStealingThreadPool.stealFromOtherWorkerThread:246,
                                     1@WorkerThread.lookForWork$1:372]
       poisonedQ=3/4 searching=1 localQ=5654
... ticks frozen at 80 for the remainder; localQ frozen at 5654; gap stays exactly 400

The injected gap holds at its injected value for the whole run under continuous load, confirming that nothing repairs it. The 5,654 fibers already in the affected queues are never dequeued again.

Impact

This surfaced twice in one day in a long-running JVM service, an hour and two hours into otherwise healthy runs under steady load. The process stopped doing any work, with no error, no OOM, no exception, nothing logged, and no thread crashed. Heap was healthy throughout and there were zero full GCs. Only a process restart cleared it. The application failure was a 30-second IO.sleep armed one millisecond before the seizure that never fired.

Why nothing recovers

// LocalQueue.scala, unchanged since v3.6.0
def enqueueBatch(batch: Array[Runnable], worker: WorkerThread[?]): Runnable = {
  val tl = tail                                     // plain, unsynchronized load
  while (true) {
    val hd = Head.updater.get(this)
    val steal = msb(hd)
    val len = unsignedShortSubtraction(tl, steal)   // tail - stealTag
    if (len <= LocalQueueCapacityMinusBatch) {      // 256 - 32 + 1 = 225
      ... transfer the batch and return ...
    }
    // otherwise: loop, on the assumption an ongoing steal will advance `steal`
  }
}

The loop's correctness rests entirely on the steal tag belonging to a steal that is in progress. dequeue and drainBatch deliberately preserve a lagging steal tag while advancing real, so a gap only widens. (A gap already above 225 also implies the corruption predates the visible spin — a legitimate steal window is at most half the queue.)

The fault then feeds itself:

operation	behaviour on a queue with a dangling tag
enqueue	looks full forever ⇒ every fiber spills to external, as batches
enqueueBatch	spins forever ⇒ traps whichever worker picks a batch up
stealInto	steal != real ⇒ refuses ⇒ the queue is permanently unstealable
dequeue	reads only real ⇒ still works; the owner drains what is there, then it is empty forever

Workers are consumed on contact, and the supply of contacts is generated by the fault.

Why the pool dies rather than merely degrading

There are two call paths into enqueueBatch, and which one traps the worker decides the outcome:

path	frame	searching while spinning?
A	WorkerThread.lookForWork$1 (the direct external.poll sites)	no — transitionWorkerFromSearching is called before enqueueBatch
B	WorkStealingThreadPool.stealFromOtherWorkerThread:246	yes — transitionWorkerFromSearching is only called after the call returns

Path B leaves the worker counted as searching for the life of the process, which disables worker wakeup on its own:

// notifyShouldWakeup
(st & SearchMask) == 0 && ((st & UnparkMask) >>> UnparkShift) < threadCount

SearchMask == 0 is required, so a single latched searcher permanently suppresses notification of parked workers. (transitionWorkerToSearching's 2 * searching >= threadCount is a further and stricter gate; it is not needed to explain the failure.) And since stealTimers is reachable only from the searching path, timers stranded on a consumed worker's heap can never be rescued.

What a live process looks like

jstack at the seizure, for contrast with the injection rig:

"io-compute-2" #58 daemon prio=5 cpu=3711320.84ms elapsed=11370.97s runnable
   java.lang.Thread.State: RUNNABLE
        at cats.effect.unsafe.LocalQueue.enqueueBatch(LocalQueue.scala:340)
        at cats.effect.unsafe.WorkStealingThreadPool.stealFromOtherWorkerThread(WorkStealingThreadPool.scala:246)
        at cats.effect.unsafe.WorkerThread.lookForWork$1(WorkerThread.scala:436)
        at cats.effect.unsafe.WorkerThread.run(WorkerThread.scala:942)

"io-compute-3" #88 daemon prio=5 cpu=3481346.14ms elapsed=4367.52s runnable
        ... identical four frames ...

• io-compute-3 accumulated 3,481 s of CPU in 4,367 s of life ⇒ ~100% CPU since the seizure instant.
• io-compute-0 and io-compute-1 are parked and stay parked.
• ⚠️ stealInto appears nowhere in the dump — no thread is performing the steal these two are waiting on.
• The fiber dump shows zero RUNNING fibers, and the same ~466 YIELDING fibers frozen across two dumps 13 minutes apart.

The transition is instantaneous rather than gradual: normal throughput, then cats-effect's own starvation warning 7 seconds later, then silence.

What we could not determine

What sets the tag. The injection proves the consequence chain — missing repair, unstealability, latched searching, absorbing state — and models none of the cause. Candidates we could not confirm:

• the worker→blocker transferState handoff (the incident had 39 blocker transitions, and two of four worker slots were mid-run replacements);
• a 16-bit ABA on the packed head under preemption on two physical cores;
• a thread dying inside stealInto — least likely, since nothing was observed crashing.

Detection

• The CPU starvation checker cannot see this. It is itself an IO.sleep, so in this state it is as undeliverable as everything else. It logged exactly one warning, at the moment of seizure, then went silent — its silence is not evidence of health. That single warning is, however, a usable signal.
• WorkStealingThreadPoolMetrics.searchingThreadCount() staying non-zero while TimerHeapMetrics.totalTimersExecutedCount() stops advancing is a direct, cheap read of this state — but any such probe has to run outside the IO runtime to be trustworthy.
• Exposing LocalQueue's steal and real head tags through LocalQueueMetrics would make the invariant tail - stealTag <= 225 externally checkable.

Possible mitigation

A bound on the spin — after N iterations, treat the tag as dangling and either repair it (steal := real) or spill the batch to the external queue — would turn an unrecoverable process into a recoverable one without needing the cause first.

Start with LocalQueue.scala, especially enqueueBatch, enqueue, and stealInto, then trace the call paths through WorkStealingThreadPool.stealFromOtherWorkerThread and WorkerThread.lookForWork$1. Run the linked StealTagWedge.scala reproduction and use the reported jstack and runtime metrics to verify the failure; done requires an agreed recovery behavior that prevents a dangling tag from permanently trapping workers and makes the pool resume work.