OpenHands / OpenHands/software-agent-sdk

[Bug]: TaskToolSet delegation holds the parent ConversationState lock for the whole sub-agent run, saturating the executor pool and freezing the conversation list on agent canvas UI

Open
#4,537 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug performance priority:high
Dominant language
Python
Stars
1.1k
Forks
539
Avg merge
1d 19h
Merged PRs (30d)
137

Description

Is there an existing issue for the same bug?
  • I have searched existing issues and this is not a duplicate.
Bug Description

When the agent makes a TaskToolSet sub-agent call, the Agent Canvas conversation list stops rendering for the entire duration of the delegated task. It shows the skeleton loader animation but nothing ever loads. Not merely slow, it never loads until the task finishes. The browser shows a timeout toast (signal timed out, i.e. an AbortSignal.timeout abort on a request the server never answers). Other parts of the UI, such as Settings, keep working normally.

Evidence

py-spy dump against the agent-server process (PID owning 127.0.0.1:18000).

Lock holder — the parent's step, executing the task tool inline, inside which the sub-agent runs its own full conversation loop:

Thread 85564 (idle): "asyncio_2"
    read (httpcore/_backends/sync.py:128)
    ...
    completion (litellm/main.py:5585)
    _transport_call (sdk/llm/llm.py:2153)
    completion (sdk/llm/llm.py:1551)
    make_llm_completion (sdk/agent/utils.py:682)
    step (sdk/agent/agent.py:715)
    run (sdk/conversation/impl/local_conversation.py:1945)
    _run_until_finished (tools/task/manager.py:437)
    _run_task (tools/task/manager.py:392)
    start_task (tools/task/manager.py:198)
    __call__ (tools/task/impl.py:32)
    __call__ (sdk/tool/tool.py:619)
    _execute_action_event (sdk/agent/agent.py:1373)
    _run_safe (sdk/agent/parallel_executor.py:309)
    run_in_caller_context (sdk/agent/parallel_executor.py:234)
    run (concurrent/futures/thread.py:73)

Blocked waiters — the conversation-list read path, one thread per poll:

Thread 88827 (idle): "asyncio_5"
    wait (threading.py:369)
    acquire (sdk/conversation/fifo_lock.py:95)
    __enter__ (sdk/conversation/state.py:734)
    _compose_conversation_info_sync (agent_server/conversation_service.py:485)
    run (concurrent/futures/thread.py:73)

Plus one thread blocked in locked_on_event (agent_server/event_service.py:837), i.e. the parent conversation's own event broadcast is also stalled.

Pool saturation — 20 dumps at 2-second intervals, counting threads parked in fifo_lock.py:95:

sample  1   8
sample  2   8
sample  3   8
sample  4   9
...
sample 20   9

Across the 20 samples: 157 waiters in _compose_conversation_info_sync, 20 in locked_on_event. That is 9 blocked threads plus the holder = 10, which is exactly min(32, cpu_count + 4) on this 6-vCPU VM. The pool is fully consumed; the plateau at 9 is saturation, not stabilisation.

Analysis

  • ConversationState.__enter__ calls self._lock.acquire() with no timeout, so waiters block indefinitely (ConversationState.acquire(blocking, timeout) exists but is not used here).
  • The agent run loop holds that lock across agent.step(). For ordinary tool calls this is brief; for TaskToolSet the tool call is an entire sub-agent conversation, so the hold lasts as long as the delegated task.
  • With tool_concurrency_limit defaulting to 1, the task tool runs via run_in_caller_context, i.e. on the very thread holding the lock.
  • _compose_conversation_info_sync takes the same lock only to read a handful of fields for ConversationInfo, so every sidebar poll parks an executor thread.
  • Settings survives because it is served from the separate AnyIO worker pool and never touches conversation state.

Environment

Agent server / SDK / tools / workspace: 1.42.1
Python: 3.14.4
Host: Lima VM (vz, Ubuntu), 6 vCPU, 4 GiB RAM, on an Apple M4 Max Mac Studio
Deployment: agent-canvas --public (full mode), ingress :8000 -> agent-server :18000
LLM backend: local llama-server via llama-swap, fronted by LiteLLM

Memory was not a factor. There was over 1.3 GB available throughout, and the VM was mostly idle waiting on the model server. This is lock contention, not resource exhaustion.

Some questions:

  1. Should the conversation-list read path take the state lock unconditionally at all, or should it use a bounded acquire / lock-free snapshot? A stale status seems strictly better than an unbounded block.
  2. #3169 added a dedicated thread pool for conversation execution, but on 1.42.1 the agent run and the blocked list reads share threads named asyncio_0asyncio_9 (the default loop executor), with a separate ThreadPoolExecutor-0_0 idle. Is delegation escaping the dedicated pool?
  3. the asymmetry between astep and step is real, but on reflection it isn't what causes this. The parent holds its lock across tool execution, not across an LLM call, so a sync release window around LLM calls wouldn't free it. The window that matters here is tool execution, the native step's on_event callback mutates state assuming the run loop holds the lock, and switch_llm's _step_holds_state_lock fast path assumes the same. Happy to test any patch against this reproduction.
Expected Behavior

Conversation list continue to work when an agent enters a task tool set subagent call.

Actual Behavior

Agent Canvas web UI becomes practically unusable. GET /api/conversations never returns for the duration of the sub-agent run. 10 of 10 (in my case on a Lima VM with 6 vcpus) default-executor threads end up blocked.

Steps to Reproduce
  1. Run Agent Canvas (full mode, agent-canvas --public) with a local model backend. (for long model processing time)
  2. Give the agent a task that causes it to delegate via TaskToolSet. (I was able to reproduce with the following prompt with muse glimmer "please help me test a behavior of agent canvas: make a task tool set subagent call (blocking type) and ask the subagent to return the ls result on the home path, then report to me.")
  3. While the sub-agent is running, refresh the page or open the web UI again in a new page.
  4. The conversation list shows a skeleton loader indefinitely and a signal timed out toast appears. Settings and other views continue to work.
Acceptance Criteria

No response

Installation Method

No response

If you selected "Other", please specify

No response

SDK Version

1.42.1

Version Confirmation
  • I have confirmed this bug exists on the LATEST version of OpenHands SDK
Python Version

No response

Model Name (if applicable)

any model (only reproduced with locally hosted models)

Operating System

Linux

Logs and Error Messages

No response

Minimal Code Sample

No response

Screenshots and Additional Context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tools/task/manager.py, sdk/conversation/state.py, sdk/conversation/fifo_lock.py, and agent_server/conversation_service.py, tracing the lock from the parent agent step through TaskToolSet execution and conversation-list reads. Reproduce with the provided TaskToolSet prompt while polling GET /api/conversations. Done means the conversation list and event broadcast remain responsive throughout the delegated run without exhausting the executor pool.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.