apache / apache/maka

bug(runtime): sandbox boundary answers wait on the graph wake lease held by the round awaiting them

Open
#3,866 2 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
TypeScript
Stars
5.4k
Forks
502
Avg merge
1d 2h
Merged PRs (30d)
706

Description

## Summary

When an agent-graph supervisor round is woken and that round raises a sandbox-boundary prompt, answering the prompt deadlocks. The answer path blocks on the supervisor wake coordinator's session-activity lease, which is held by the very round that is parked waiting for that answer. Because the answer holds Session admission while it blocks, every later operation on that Session queues behind it and the Session becomes unusable.

Reproduced on `main` (`v0.2.0`, `d92d277`).

## Symptom

The Session freezes permanently: the transcript stays on `Loading…`, the boundary prompt never renders again, nothing in that Session responds. Restarting the app clears it. Leaving the prompt unanswered is harmless — the Session simply stays parked and the UI keeps working.

Image

## Reproduction, and where it goes wrong

Reproduce Prompt:

```md
学习 https://github.com/AlexsJones/llmfit
使用本地已经安装好的 llmfit 结合
https://huggingface.co/models
中各种优化模型,
以本地 arm 版本 M1max 芯片为目标,
以 32G 内存为限制,
综合对比, 推荐一种 Qwen3.8-27B 超量化版
可以通过 Unsloth 加载, 提供至少 20tok/s 的本地模型推理速度

前面的调研用 agent graph
```

| # | What you do / see | What happens internally |
|---|---|---|
| 1 | Send a task that instructs the model to **use the agent graph** and that **requires reading paths outside the workspace** | The first round is user-initiated; it never touches the activity lease. It plans and dispatches operators. |
| 2 | Operators run and return one by one | Each milestone can schedule a supervisor wake. |
| 3 | A supervisor round starts (**"Agent graph reached a supervisor checkpoint"**) and raises «Allow access outside the workspace?» | `#deliverWake` **acquires the activity lease** and awaits the round it started. The round parks on the prompt, so the lease stays held. |
| 4 | **Leave the prompt unanswered.** Let the remaining operators finish | Operators run in their own Sessions and never touch this lease. The parked round still holds it. |
| 5 | Once all operators are done, click **Allow for this task** | `interaction.answer` takes Session admission and completes its first five steps. |
| 6 | *(nothing visibly changes yet)* | ⬅ **The faulty moment.** The final step, `#onSandboxBoundarySettled` → `notifyPermissionResponse` → `#settlePermissionResponse`, calls `activityRegistry.acquire(rootSessionId)`. That lease is held by the parked round from step 3 — the round waiting for this answer. No timeout, no abort path. |
| 7 | Open the child task, then navigate back | This forces the UI to re-subscribe. `subscription.open` queues behind the stuck `interaction.answer`, so `readActiveInteractions` never returns and the interaction queue never hydrates. |
| 8 | The transcript stays on `Loading…`; the prompt never renders again | The Session is unusable until restart. |

Step 6 is worth calling out: **nothing visible happens at the moment the deadlock forms.** The freeze only surfaces at step 7, when the UI next re-subscribes — which is why the existing reports describe it as an intermittent hang with no clear steps.

Two things follow from step 3:

- A prompt raised by the **first** round (step 1) does **not** deadlock — that round holds no lease.
- Whether the operators have finished is irrelevant; they never touch this lease. What matters is only whether a wake-started round is parked when the prompt is answered.

The two paths acquire the two mechanisms in opposite order: the wake path takes the activity lease and then repeatedly takes admission for its work, while the answer path takes admission and then wants the activity lease.

Note what step 6 is trying to do: check whether the round waiting on the permission is still alive, so a lost waiter can be retried (`permission_waiter_lost`). A live waiter is precisely one holding the lease — so the check can only ever observe the "waiter is gone" case. The lease also looks redundant there: `completeAgentGraphSupervisorWakeAttempt` already performs an attempt-scoped compare-and-set inside a DB transaction, and `#scheduleRecoveredWake` — the one action that starts a new round — runs outside the lease.

## Proposed fix

1. **Do not take the session activity lease in `#settlePermissionResponse`.** Rely on the existing transactional, attempt-scoped compare-and-set in the wake store, and treat "attempt is no longer current" as a benign outcome. This removes the inverted edge.
2. **Fire the graph-wake notification without awaiting it** from the answer path, so Session admission is never held across a call into another subsystem. This does not fix the deadlock; it caps the blast radius at "the wake is delayed" rather than "the Session is dead".
3. **Add re-entrancy detection to `SessionActivityRegistry`**, mirroring the guard `SessionAdmissionGate` already has, so a future caller on this path fails loudly during development. Separate change — hardening, not a fix.

Adding a timeout to `acquire` is deliberately not proposed: the cycle would remain, and a timeout would convert a deterministic deadlock into nondeterministic state.

## References

- #1114 — introduced `SessionActivityRegistry` for a caller that runs *between* turns, where "wait until the Session is idle" holds.
- #1935 — hosted sandbox-boundary interactions and wired the settle path, whose only call moment is *mid-round*, inverting that precondition.
- #3312, #3328 — both reports show a pending `request_sandbox_boundary` at the time of the freeze; likely the same underlying issue.

Contributor guide

Open the contributing guide

Research direction

Start at #settlePermissionResponse and trace the answer path through #onSandboxBoundarySettled and notifyPermissionResponse, then compare it with completeAgentGraphSupervisorWakeAttempt and the activity-lease acquisition in #deliverWake. Done means a sandbox-boundary answer no longer waits on the lease held by its parked wake round, the wake result handles a non-current attempt benignly, and the Session remains responsive.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.