MemberJunction / MemberJunction/MJ
A single runaway agent run can exhaust the Node heap and abort the whole MJAPI process
- Dominant language
- TSQL
- Stars
- 29
- Forks
- 6
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 323
Description
**Summary**
A single agent run can exhaust the Node heap and abort the entire MJAPI process. Every other user of that instance loses their session, every in-flight run is orphaned, and the browser shows `Server unavailable — viewing cached data` followed by `GraphQL Error (Code: unknown)`.
This is not a hypothetical. It happened on a development instance while testing an unrelated fix, triggered by one misconfigured agent.
**Evidence**
macOS crash report, faulting thread:
```
abort
node::OOMErrorHandler(char const*, v8::OOMDetails const&)
v8::internal::V8::FatalProcessOutOfMemory(...)
v8::internal::Heap::FatalProcessOutOfMemory(char const*)
v8::internal::Heap::CollectGarbage(...)
v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(...)
v8::internal::FactoryBase::NewRawTwoByteString(int, AllocationType)
v8::internal::Factory::NewStringFromTwoByte(...)
v8::internal::JsonStringify(...) <-- JSON.stringify
v8::internal::Builtin_JsonStringify(...)
```
`EXC_CRASH / SIGABRT`, `Abort trap: 6`. V8 could not allocate a string while serializing and aborted the process.
The agent run that caused it recorded **107 steps: 106 consecutive `Execute Agent Prompt` and not one Action step.** The agent called the model, appended the response to the conversation, and called again. The serialized payload grew each turn until it exceeded the heap.
Timeline:
| Time (UTC) | Event |
|---|---|
| 04:21:27 | Run starts |
| 04:22:57 | Last `LastHeartbeatAt` written |
| ~04:23:17 | `SIGABRT`, process gone |
The run is still `Status='Running'` in the database with a stale heartbeat, waiting on the watchdog sweep.
**Why the existing guards did not help**
| Guard | Value in this run | Why it did not fire |
|---|---|---|
| `AIAgent.MaxIterationsPerRun` | not set | optional, nullable, no default |
| `AIAgent.MaxCostPerRun` | not set | optional, nullable |
| `AIAgent.MaxTokensPerRun` | not set | optional, nullable |
| `BaseAgent.DEFAULT_ABSOLUTE_MAX_ITERATIONS` | 5000 | only 106 iterations were reached |
| Node heap ceiling | **unset** | `packages/MJAPI` passes no `--max-old-space-size` |
Two structural problems are visible here.
**1. Every run-level guard is opt-in.** An agent that sets none of `MaxIterationsPerRun`, `MaxCostPerRun` or `MaxTokensPerRun` runs with no per-agent bound at all. The only backstop is the system-wide absolute iteration net.
**2. The absolute net is sized in the wrong unit.** `DEFAULT_ABSOLUTE_MAX_ITERATIONS = 5000` (`packages/AI/Agents/src/base-agent.ts:480`) counts iterations, but the failure is measured in bytes. A conversation that grows ~1 KB per turn exhausts a default V8 heap long before turn 5000. In this run the heap died at 106. The net cannot catch this class of failure at any iteration value that is also useful.
There is no size-based guard in the agent loop — no payload-size ceiling, no conversation-length ceiling. Searching `base-agent.ts` for `maxPayloadSize`, `payloadSizeLimit`, `MAX_PAYLOAD` or `truncateConversation` returns nothing.
**Blast radius**
MJAPI is a shared, multi-tenant process. One user's runaway agent aborts it for everybody. In development that is an interrupted session. In a deployment without a process supervisor it is an outage; with one it is a restart that still drops every open WebSocket and orphans every concurrent run.
Note the interaction with #4222: when MJAPI dies, in-flight completions are lost with no replay, so affected conversations hang until the client reconciles.
**Reproduce**
1. Create a Loop agent with `MaxIterationsPerRun` NULL and one action available.
2. Give it an instruction whose success condition the model cannot satisfy, so it keeps re-planning. In our case the model wrote a `While` loop with condition `payload.delayCount < 5` and nothing ever incremented `delayCount`.
3. Run it and watch RSS climb. The process aborts in roughly two minutes.
The trigger was an operator mistake — an agent created without a cap. That is the point: one bad agent record, reachable by anyone who can create agents, is enough to take down the server.
**Suggested fixes**
Roughly in order of cost against benefit.
1. **Give MJAPI an explicit heap ceiling.** `--max-old-space-size` in `packages/MJAPI` start script. Today the process inherits the V8 default while `MJExplorer`, which serves one developer, sets 16384. This does not prevent the abort by itself but makes the limit intentional and tunable per deployment.
2. **Add a size-based guard to the agent loop.** Bound serialized payload and conversation length, and fail the single run when it is exceeded. This is the guard that actually matches the failure mode; iteration counts cannot.
3. **Default `MaxIterationsPerRun` when an agent leaves it NULL.** A sane default (10–25) means a misconfigured agent fails its own run rather than the process.
4. **Re-examine `DEFAULT_ABSOLUTE_MAX_ITERATIONS = 5000.`** Whatever value is chosen, it should be reachable before the heap is exhausted, or it is not a safety net.
5. **Consider isolating agent execution** so a runaway run cannot abort the shared process. Larger change; worth scoping separately.
Items 1 and 3 are small and would each have prevented this specific incident. Item 2 is the one that closes the class.
Contributor guide
Research direction
Start with packages/AI/Agents/src/base-agent.ts around DEFAULT_ABSOLUTE_MAX_ITERATIONS and the existing per-run guard fields, then inspect the packages/MJAPI start script for its Node heap settings. Trace how a conversation is serialized during the agent loop and define a bounded failure for the individual run; done means the reproduced runaway run cannot abort the shared MJAPI process.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100