OpenHands / OpenHands/software-agent-sdk
[Bug]: TaskToolSet delegation holds the parent ConversationState lock for the whole sub-agent run, saturating the executor pool and freezing the conversation list on agent canvas UI
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 539
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 137
Description
Is there an existing issue for the same bug?
- I have searched existing issues and this is not a duplicate.
Bug Description
When the agent makes a TaskToolSet sub-agent call, the Agent Canvas conversation list stops rendering for the entire duration of the delegated task. It shows the skeleton loader animation but nothing ever loads. Not merely slow, it never loads until the task finishes. The browser shows a timeout toast (signal timed out, i.e. an AbortSignal.timeout abort on a request the server never answers). Other parts of the UI, such as Settings, keep working normally.
Evidence
py-spy dump against the agent-server process (PID owning 127.0.0.1:18000).
Lock holder — the parent's step, executing the task tool inline, inside which the sub-agent runs its own full conversation loop:
Thread 85564 (idle): "asyncio_2"
read (httpcore/_backends/sync.py:128)
...
completion (litellm/main.py:5585)
_transport_call (sdk/llm/llm.py:2153)
completion (sdk/llm/llm.py:1551)
make_llm_completion (sdk/agent/utils.py:682)
step (sdk/agent/agent.py:715)
run (sdk/conversation/impl/local_conversation.py:1945)
_run_until_finished (tools/task/manager.py:437)
_run_task (tools/task/manager.py:392)
start_task (tools/task/manager.py:198)
__call__ (tools/task/impl.py:32)
__call__ (sdk/tool/tool.py:619)
_execute_action_event (sdk/agent/agent.py:1373)
_run_safe (sdk/agent/parallel_executor.py:309)
run_in_caller_context (sdk/agent/parallel_executor.py:234)
run (concurrent/futures/thread.py:73)
Blocked waiters — the conversation-list read path, one thread per poll:
Thread 88827 (idle): "asyncio_5"
wait (threading.py:369)
acquire (sdk/conversation/fifo_lock.py:95)
__enter__ (sdk/conversation/state.py:734)
_compose_conversation_info_sync (agent_server/conversation_service.py:485)
run (concurrent/futures/thread.py:73)
Plus one thread blocked in locked_on_event (agent_server/event_service.py:837), i.e. the parent conversation's own event broadcast is also stalled.
Pool saturation — 20 dumps at 2-second intervals, counting threads parked in fifo_lock.py:95:
sample 1 8
sample 2 8
sample 3 8
sample 4 9
...
sample 20 9
Across the 20 samples: 157 waiters in _compose_conversation_info_sync, 20 in locked_on_event. That is 9 blocked threads plus the holder = 10, which is exactly min(32, cpu_count + 4) on this 6-vCPU VM. The pool is fully consumed; the plateau at 9 is saturation, not stabilisation.
Analysis
ConversationState.__enter__callsself._lock.acquire()with no timeout, so waiters block indefinitely (ConversationState.acquire(blocking, timeout)exists but is not used here).- The agent run loop holds that lock across
agent.step(). For ordinary tool calls this is brief; forTaskToolSetthe tool call is an entire sub-agent conversation, so the hold lasts as long as the delegated task. - With
tool_concurrency_limitdefaulting to 1, the task tool runs viarun_in_caller_context, i.e. on the very thread holding the lock. _compose_conversation_info_synctakes the same lock only to read a handful of fields forConversationInfo, so every sidebar poll parks an executor thread.- Settings survives because it is served from the separate AnyIO worker pool and never touches conversation state.
Environment
Agent server / SDK / tools / workspace: 1.42.1
Python: 3.14.4
Host: Lima VM (vz, Ubuntu), 6 vCPU, 4 GiB RAM, on an Apple M4 Max Mac Studio
Deployment: agent-canvas --public (full mode), ingress :8000 -> agent-server :18000
LLM backend: local llama-server via llama-swap, fronted by LiteLLM
Memory was not a factor. There was over 1.3 GB available throughout, and the VM was mostly idle waiting on the model server. This is lock contention, not resource exhaustion.
Some questions:
- Should the conversation-list read path take the state lock unconditionally at all, or should it use a bounded acquire / lock-free snapshot? A stale status seems strictly better than an unbounded block.
#3169added a dedicated thread pool for conversation execution, but on 1.42.1 the agent run and the blocked list reads share threads namedasyncio_0…asyncio_9(the default loop executor), with a separateThreadPoolExecutor-0_0idle. Is delegation escaping the dedicated pool?- the asymmetry between
astepandstepis real, but on reflection it isn't what causes this. The parent holds its lock across tool execution, not across an LLM call, so a sync release window around LLM calls wouldn't free it. The window that matters here is tool execution, the native step'son_eventcallback mutates state assuming the run loop holds the lock, andswitch_llm's_step_holds_state_lockfast path assumes the same. Happy to test any patch against this reproduction.
Expected Behavior
Conversation list continue to work when an agent enters a task tool set subagent call.
Actual Behavior
Agent Canvas web UI becomes practically unusable. GET /api/conversations never returns for the duration of the sub-agent run. 10 of 10 (in my case on a Lima VM with 6 vcpus) default-executor threads end up blocked.
Steps to Reproduce
- Run Agent Canvas (full mode,
agent-canvas --public) with a local model backend. (for long model processing time) - Give the agent a task that causes it to delegate via
TaskToolSet. (I was able to reproduce with the following prompt with muse glimmer "please help me test a behavior of agent canvas: make a task tool set subagent call (blocking type) and ask the subagent to return the ls result on the home path, then report to me.") - While the sub-agent is running, refresh the page or open the web UI again in a new page.
- The conversation list shows a skeleton loader indefinitely and a
signal timed outtoast appears. Settings and other views continue to work.
Acceptance Criteria
No response
Installation Method
No response
If you selected "Other", please specify
No response
SDK Version
1.42.1
Version Confirmation
- I have confirmed this bug exists on the LATEST version of OpenHands SDK
Python Version
No response
Model Name (if applicable)
any model (only reproduced with locally hosted models)
Operating System
Linux
Logs and Error Messages
No response
Minimal Code Sample
No response
Screenshots and Additional Context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tools/task/manager.py, sdk/conversation/state.py, sdk/conversation/fifo_lock.py, and agent_server/conversation_service.py, tracing the lock from the parent agent step through TaskToolSet execution and conversation-list reads. Reproduce with the provided TaskToolSet prompt while polling GET /api/conversations. Done means the conversation list and event broadcast remain responsive throughout the delegated run without exhausting the executor pool.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100