anthropics / anthropics/claude-agent-sdk-python
Multi-pod HITL: no recovery mechanism when pod crashes during `can_use_tool` wait
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 1.3k
- Avg merge
- 2d 31m
- Merged PRs (30d)
- 1
Description
## Description
We're building a production multi-pod service using `ClaudeSDKClient` with a Human-in-the-Loop approval flow. We've read through the existing issues (#96, #304) and examples, and we have a working approach for the happy path — but we're stuck on the pod crash/restart scenario.
## Our setup
- Multi-pod Kubernetes deployment (any pod can serve any request)
- `SessionStore` (v0.1.66) for transcript durability across pods
- `can_use_tool` callback as the HITL gate (no timeout — per #96)
## What works (but we don't want)
We can block inside `can_use_tool` with an `asyncio.Future`, coordinated across pods via Postgres `LISTEN/NOTIFY`. When the user approves, the future resolves and the agent continues in the same stream.
However, **we don't want to keep the CLI subprocess alive while waiting for the user**. User decisions can take minutes or hours. In a multi-tenant service with many concurrent sessions, holding a live subprocess per waiting session (memory, CPU, connection) doesn't scale. We'd prefer to stop the process, persist the state, and resume when the user decides.
## What breaks: stop process + resume
When the pod running the agent dies while `can_use_tool` is blocking, the stream and CLI subprocess are lost. There is no SDK mechanism to recover a pending `can_use_tool` gate on a different pod. The only option is to start a new session via `--resume`, but:
- **No built-in HITL recovery on resume** — after a pod crash, the application must manage the full HITL state externally (persist pending decisions, detect already-made decisions on the resumed session, return them from `can_use_tool`, etc.). The SDK has no awareness of interrupted HITL gates.
- **Non-transcript state is lost** — `materialize_resume_session()` only restores transcripts and auth files. Plan files and other config-dir state are not included (see separate issue).
## Questions
1. **Is there a recommended pattern for HITL in multi-pod deployments that survive pod restarts?** The wait-in-callback approach works great until the pod dies. The two-request resume approach works but loses non-transcript state.
2. **Has the team considered a built-in mechanism for durable HITL gates?** For example, a way to persist pending `can_use_tool` decisions in `SessionStore` so that on resume the SDK can automatically resolve them without the application needing to re-implement the coordination logic.
3. **Is there a way to stop/pause the CLI subprocess while waiting for the user's answer?** While `can_use_tool` blocks, the CLI subprocess stays alive consuming resources (memory, connection). In a multi-tenant service with many concurrent sessions waiting for user approval, this doesn't scale well. Is there a way to gracefully stop the process during the wait and resume it later without losing state?
4. **How should the user's HITL decision appear in the chat history?** When a user approves/declines a tool call, we want that choice to show as a message in the conversation (e.g., "Approved"). `can_use_tool` doesn't write to the transcript, and `sdk_get_messages()` only returns `user`/`assistant` messages. What's the recommended way to persist a HITL decision as a visible message at the correct position?
5. **Are there other teams running multi-pod HITL with `ClaudeSDKClient`?** We'd love to learn about patterns others have found for this use case.
## Environment
- `claude-agent-sdk` v0.1.66
- Python 3.14
- Kubernetes, Postgres (asyncpg), FastAPI
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.