iOfficeAI / iOfficeAI/AionCore

Team managers remain active for hours with unresolved Codex code-mode calls after PID resource exhaustion

Open
#950 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
105
Forks
169
Avg merge
5h 58m
Merged PRs (30d)
84

Description

## Summary

Two Codex-backed team managers remained in an active turn for more than 11 hours with unresolved code-mode `exec` calls wrapping `team_read_messages`. One manager accumulated 17 incoming messages while still showing the original turn as running.

At the start of the incident, the Linux PID controller rejected process/thread creation in the application's systemd cgroup. Codex also logged `Resource temporarily unavailable (os error 11)` and code-mode host startup/exit errors in that interval.

**The resource rejection and unresolved calls are confirmed separately. The exact causal path is not yet reproduced or attributed to a particular manager's code-mode host.** This may require a Codex fix rather than an AionCore transport fix; filing here to track the team-runtime impact and recovery behavior.

## Environment

- AionUi Web CLI 2.1.61, Linux x86_64, systemd.
- AionCore 0.1.72 with the bounded streaming-preview backport from #947. This was not an unmodified current-main deployment.
- Codex CLI 0.150.1, app-server transport, GPT-5.6 family.
- Two teams, approximately eleven configured members including their managers. Members run development tasks, including test/build subprocesses.
- Locally configured service `TasksMax=512`, counting both processes and threads. **This is our deployment setting, not an asserted upstream default.**
- 2 vCPU / 8 GiB host; service `MemoryHigh=5 GiB`, `MemoryMax=6 GiB`.

## Observed sequence (not yet a deterministic minimal reproduction)

1. Run both teams concurrently and let managers assign development work.
2. Several member runtimes start near the same time.
3. The kernel reports `cgroup: fork rejected by pids controller` for the application's service.
4. Codex logs process creation failures with OS error 11. Two managers issue code-mode calls equivalent to:

```javascript
const r = await tools.mcp__aionui_team__team_read_messages({});
text(r);
```

One actual call supplied a message cursor. No private cursor or call/session identifiers are included here.

5. Both session traces have an outbound `custom_tool_call` named `exec` and no matching output for over 684 minutes. The calls began about 0.55 seconds apart, roughly two seconds after the kernel refusal.
6. Other Codex processes log `code-mode host exited during handshake` / `code-mode host closed its stdout`; a code-mode host also traps. Those PIDs have **not** been correlated conclusively with the two managers.
7. Manually cancelling one manager produces `aborted by user after 41106.8s`. Its next turn successfully reads team messages and completes. The other manager remains pending, with 17 incoming messages verified in the SQLite chat projection, until a later service crash/restart.

No automated retry or queue replay was used in the investigation.

## Evidence boundary

- A read-only check of the real Codex session traces deterministically detects the missing results at a historical cutoff. This is **incident verification, not a regression test or a synthetic reproduction of the cause**.
- Earlier `team_read_messages` requests succeeded in milliseconds. No matching server-side `tools/call invoked` trace was found for the stalled calls. However, that log occurs after an awaited agent lookup; absence alone cannot locate the stall.
- AionCore's coordinator keeps an active batch until settlement; the Codex reader intentionally has no blanket mid-turn deadline. Neither behavior by itself proves a bug: long-running valid tools must remain supported.
- Separate later `aionui-web` SIGABRT exits are under investigation. Available records do not establish their cause or connect them to this stall. They should not be conflated with an OOM or with #947.
- A service task-quota increase to 1,024 was applied without restarting the service or changing its memory limits. This is mitigation, not proof of recovery correctness.

## Expected behavior / candidate acceptance criteria

- A confirmed failed execution host or failed nested dispatch must not leave an unfinishable tool/turn permanently active.
- Propagate a meaningful terminal error and settle the corresponding active work when failure is known.
- If the backend is merely quiet or an operation is legitimately long-running, do not kill it using an arbitrary overall turn timeout.
- Preserve pending messages and user data. Never silently replay a possibly side-effectful tool call whose completion is unknown.
- Add a regression test that distinguishes host/dispatch failure from a healthy long-running operation, and verifies the real team coordinator can recover after explicit interruption.

## Related reports reviewed

- openai/codex#38203 reports outstanding code-mode calls left pending after a host crash, on Windows Desktop. Similar symptom; different platform and unproven common cause.
- openai/codex#39149 describes an unanswered MCP approval elicitation in a headless client. Also a possible indefinite-wait mechanism, but not demonstrated in this incident.
- #787 concerns team read-message behavior; the observed call here never returned a result in the Codex trace.

Private prompts, task contents, domains, credentials, workspace paths, team names and session IDs are intentionally omitted.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at AionCore's coordinator active-batch settlement and the Codex reader's awaited execution path, then compare them with the historical Codex session traces and SQLite message projection described here. Done means a regression test distinguishes failed host or nested dispatch from a healthy long-running operation, settles interrupted work with a terminal error, preserves pending messages, and avoids replaying calls with unknown completion.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, rust
Domain
backend, distributed-systems, testing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.