OpenHands / OpenHands/software-agent-sdk

[Bug]: Conversations stall indefinitely when ChatGPT subscription LLM call hangs

Open
#4,997 1 comment 0 reactions 1 assignee View on GitHub

@neubig is already working on this.

Since Sep 14, 2026.

bug llm priority:high ready-for-dev security
Dominant language
Python
Stars
1.1k
Forks
542
Avg merge
1d 19h
Merged PRs (30d)
137

Description

Is there an existing issue for the same bug?
  • I have searched existing issues and this is not a duplicate.
Bug Description

When using a ChatGPT subscription (OAuth) profile, conversations can enter a
permanently stalled state where execution_status remains RUNNING but no
new events are ever produced. The agent-server health check continues to
return 200, and WebSocket connections stay open, but the conversation never
progresses or errors out.

This was observed on two conversations (014d3f78 and 27ea2c04) running
against gpt-5.6-sol-sub (ChatGPT subscription, Codex API endpoint). After
a transient litellm.ServiceUnavailableError at 12:29 UTC (OpenAI upstream
connection error), the conversations continued briefly, then stalled at
~12:34–12:35 UTC and never recovered — no events, no errors, no timeout,
for over 1 hour.

Root Cause Analysis

The stall occurs in the async LLM call path for subscription profiles:

  1. The LLM call is made inside agent.astep(), which is awaited from
    conversation.arun() while holding the conversation's state lock
    (FIFOLock). The lock IS released during the LLM network I/O via
    _released_state_lock_during_io(), so the lock itself is not the
    problem.

  2. The only timeout is litellm's HTTP timeout parameter (set to 300s
    in the profile config). This is passed to litellm.aresponses() /
    litellm.acompletion() as timeout=self.timeout. For non-streaming
    HTTP requests, httpx enforces this as the total request timeout.

  3. However, for streaming responses, httpx/litellm apply the timeout only
    to the initial connection, not to the overall stream duration.
    The
    Codex subscription API (https://chatgpt.com/backend-api/codex/responses)
    requires stream=True (forced by OpenAISubscriptionAuth.create_llm()).
    If the upstream establishes the connection but then stops sending chunks
    (e.g., after a partial outage), the stream read blocks indefinitely —
    the 300s timeout never fires because it only governs the connect phase.

  4. There is no asyncio.wait_for() wrapper anywhere in the call chain:
    amake_llm_completion()llm.aresponses()litellm.aresponses().
    No step-level or iteration-level timeout exists to detect a hung LLM call.

  5. The StuckDetector only checks for repeating action/observation
    patterns
    (e.g., same command run 3 times in a row). It does NOT detect
    a hung LLM call because no events are produced while the stream is
    blocked — the detector never gets a chance to run.

  6. The arun() exception handler at line 2561 catches Exception and
    sets execution_status = ERROR, but since the hung await never raises,
    this handler is never triggered.

Timeline of the observed incident
Time (UTC) Event
12:29:41 litellm.ServiceUnavailableError — OpenAI upstream connection error (Attempt #1, retried)
12:29:45 Credential refresh (retry in progress)
12:30–12:33 Conversations continue normally (retry succeeded, bash commands executing)
12:34:34 Conversation 27ea2c04 starts credential refresh for next LLM call
12:35:05 Conversation 014d3f78 starts credential refresh for next LLM call
12:35:05+ SILENCE — no more events, no errors, no timeouts for 70+ minutes
13:43:00 Browser reconnects WebSocket; resend_mode=since returns same stale events

The ServiceUnavailableError at 12:29 was retried and recovered. But the
next LLM call (starting ~12:34–12:35) apparently connected to the Codex
endpoint but never received a complete response — the stream hung, and
without a stream-level or call-level timeout, the await blocked forever.

Expected Behavior
  • A streaming LLM call that stops receiving data for longer than the
    configured timeout (300s) should be aborted and retried (or raise an
    error that transitions the conversation to ERROR).
  • The conversation should never remain in RUNNING status indefinitely
    with no events being produced.
Actual Behavior

Synthetic reproduction: uv run pytest tests/sdk/llm/test_llm_timeout.py -q on PR #5000 exercises a real local HTTP server that stops delivering stream data and verifies timeout/retry behavior.

The conversation stays in RUNNING status forever. No error event is
emitted. No timeout fires. The only recovery is manually pausing/restarting
the conversation or restarting the agent-server.

Steps to Reproduce
  1. Configure a ChatGPT subscription profile (e.g., gpt-5.6-sol-sub)
  2. Start a conversation and let it run
  3. If the OpenAI Codex endpoint experiences a transient upstream issue
    (connection reset, partial outage), the retry may succeed but a
    subsequent call may hang on the streaming response
  4. The conversation stalls indefinitely

This is hard to reproduce deterministically because it depends on the
upstream Codex API hanging mid-stream. A synthetic reproduction could
use a mock endpoint that accepts the connection but never sends a
complete SSE response.

Acceptance Criteria
  • Streaming LLM calls have a stream-level idle timeout that aborts
    if no data is received for a configurable period (e.g., timeout
    seconds between chunks, not just for the connect phase)
  • Alternatively or additionally, wrap the LLM call in
    asyncio.wait_for() with the configured timeout as a hard cap
  • When the timeout fires, the error is retried per the existing
    retry logic (tenacity), and if retries are exhausted, the
    conversation transitions to ERROR (not stuck in RUNNING)
  • Unit test covers the hung-stream scenario, and fail on main and pass on the branch
Installation Method

Agent Canvas development stack

If you selected "Other", please specify

No response

SDK Version

No response

Version Confirmation
  • I have confirmed this bug exists on the LATEST version of OpenHands SDK
Python Version

No response

Model Name (if applicable)

No response

Operating System

None

Logs and Error Messages

No response

Minimal Code Sample

No response

Screenshots and Additional Context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.