Local pipe: concurrent arrays oversubscribe the CPU budget (stateless local_worker_limit)

Summary

local_worker_limit() (arc/job/pipe/pipe_run.py) decides how many pipe workers may run
concurrently, but it is stateless across pipe arrays: it derives the count from the full
machine-wide CPU/memory budget on every call, with no accounting of workers already running from
other concurrent arrays. When ARC runs more than one local pipe array at a time, the combined
worker count can exceed the configured budget.

Details

The limit is computed as:

by_cores = local_cpu_budget() // max(1, int(cpus_per_worker))
# ... further capped by available memory ...

local_cpu_budget() returns the server's cpus (the machine-wide budget) fresh on each call.
Each pipe array is a separate process/invocation, so there is no shared in-process state to
consult — every array computes its limit as if it were the only one running.

Example

Budget = 20 cores, 5 cores/worker. Three concurrent arrays (e.g. an sp batch, a freq batch,
and a ts batch) each derive 20 // 5 = 4 workers → 12 workers = 60 cores requested against a
20-core budget. The budget bounds workers within one array but not across concurrent arrays.

Root cause

The derivation has no cross-process view of currently-running workers. A single-process semaphore
would not help, because the arrays are independent processes with no shared in-memory counter.

Proposed fix

Cross-process coordination that all local arrays consult before spawning a worker — e.g. an
on-disk/lock-file token bucket keyed to the CPU budget, or a lightweight local scheduler. Each
array acquires tokens for the cores it takes and releases them as workers finish, so the machine
never exceeds the configured budget regardless of how many arrays run.

Workaround

Set pipe_settings['local_max_workers'] to cap the derived value manually when running concurrent
arrays.

References

• PR #924 (PySCF in-core adapter / local pipe backend) — review thread by @calvinp0 that surfaced this.

Start in arc/job/pipe/pipe_run.py with local_worker_limit() and local_cpu_budget(), then trace how local pipe arrays spawn workers. The fix is complete when concurrent arrays coordinate across processes so acquired worker resources never exceed the configured CPU budget and are released as workers finish; review the local_max_workers workaround for compatibility.