[Bug]: Cloud Automation PR reviewer intermittently posts no review — SecretStr webhook serialization error (non-blocking) + 600s run ceiling killing stalled agent loop (blocker)

Summary

Cloud Automation PR-reviewer runs intermittently fail to post a review: the bot posts a 🔍 Review in progress… comment but never posts the review (Step 4) and never updates the comment to ✅ Review complete. (Step 5). The agent conversation stalls and is killed. On the same PR, some reviews succeed and some don't (see e.g. OpenHands/evaluation#588, where 22:08/22:53/23:08 reviews completed but 23:20/23:34 were orphaned at "in progress").

Investigation with Datadog (prod-runtime runtime-pods + prod-core automation service) points to two distinct runtime-side bugs, only one of which actually blocks the review.

Bug 1 (non-blocking but noisy): Object of type SecretStr is not JSON serializable in the conversation event webhook poster

The agent-server posts conversation events to ${OH_WEBHOOKS_0_BASE_URL}/api/v1/webhooks/events/{conversation_id}. A Pydantic SecretStr is leaking into the event payload, and json.dumps() cannot serialize it, so every event post fails from the first event onward:

warn: Webhook post attempt 1 failed: Object of type SecretStr is not JSON serializable
warn: Webhook post attempt 2 failed: Object of type SecretStr is not JSON serializable
warn: Webhook post attempt 3 failed: Object of type SecretStr is not JSON serializable
warn: Webhook post attempt 4 failed: Object of type SecretStr is not JSON serializable
error: Failed to post events to webhook https://app.all-hands.dev/api/v1/webhooks/events/{conversation_id} after 4 attempts

This fires on a ~5s loop for the entire lifetime of the conversation. It is systemic: 100+ error lines across 3+ distinct runtime-* pods in 6h, and it has been occurring since at least 2026-06-22T02:02:41Z (24h+). Interleaved 401 Unauthorized responses are also logged against the same webhooks/events/{conversation_id} URL.

Important: this bug is NOT what blocks the review. I confirmed this by comparing a completing conversation (95bfb430, pod runtime-gvrbjukuamrmrdwj) with a stuck conversation (3b8197b5, pod runtime-xuheokibrcpgudko): both pods emit the SecretStr is not JSON serializable errors, yet the completing conversation still posted its review and reached "✅ Review complete." The agent runs locally in the sandbox and posts the GitHub review via a direct curl to api.github.com, which does not depend on the event-streaming webhook. So Bug 1 breaks the UI event stream (and wastes CPU on retries) but does not itself prevent the review.

Bug 2 (the actual blocker): agent stalls in an empty-LLM-response loop, then the 600s run ceiling kills it before recovery

This is what differs between completing and stuck runs.

Completing run (95bfb430): start_bash_command at 23:07:58 → conversation created 23:08:15 → review posted ~23:16 (~487s in, under the 600s budget) → SIGTERM at 23:17:07 (normal post-completion sandbox teardown). No Command timed out, no empty-response loop.

Stuck run (3b8197b5): start_bash_command at 23:32:47 → conversation 23:33:04 → at 23:40:16 (430s in) the agent enters an empty-response loop:

warn: LLM produced empty response - continuing agent loop
warn: LLM response contained no tool call and no content - sending corrective feedback

Then at 23:42:47 — exactly 600s after start_bash_command — the run's bash command is killed:

warn: Command timed out after 600 seconds: mkdir -p /workspace/project && tar xzf /tmp/automation-<run_id>.tar.gz ... && ([ ! -f setup.sh ] || bash setup.sh) && <entrypoint>

followed by Received signal SIGTERM (15), shutting down.... No api.github.com / /pulls/ / /reviews activity was ever logged — the review was never posted.

Why only sometimes

The review automation's entire execution (tarball extract + setup.sh + SDK install + agent review + GitHub POST) is one bash command bounded by max_run_duration = 600s (OpenHands/automation → execution.py::execute_in_context → _start_bash(timeout=...); dispatcher.py → effective_timeout = min(automation.timeout, max_run_duration); config.py max_run_duration: int = 600, env AUTOMATION_MAX_RUN_DURATION). presets/{prompt,plugin}/setup.sh reinstalls openhands-sdk/openhands-tools/openhands-workspace from PyPI on every run, consuming a large fixed chunk of that 600s. So the agent has only a few minutes for the actual review. When the LLM starts returning empty responses (intermittent provider behavior), the agent burns the remaining budget in the corrective-feedback loop and the 600s ceiling kills the run before it reaches Step 4/5. When the LLM behaves, the review completes under budget. Hence the intermittent symptom on the same PR.

Datadog shows this is not isolated: 50+ Run timed out (exit_code=-1) on the prod-core automation service and 24 Command timed out after 600 seconds on prod-runtime in 6h, across multiple orgs/users.

Expected behavior

• Bug 1: event payloads posted to webhooks/events/{conversation_id} should serialize SecretStr (and set / other non-JSON-native types) safely — e.g. model_dump(mode="json") / pydantic.TypeAdapter(...).dump_python(..., mode="json"), or json.dumps(..., default=str) as a safety net — so the UI event stream works and the retry storm stops.
• Bug 2: either raise the run budget for review-class automations (raise AUTOMATION_MAX_RUN_DURATION, or make max_run_duration per-automation so heavy reviews can legitimately request more time) and/or stop spending the time budget on per-run SDK install (cache/pre-bake the venv in the runtime image, or run setup.sh before the timed entrypoint starts). Additionally, the agent loop should break out of (or bounded-retry) the empty-LLM-response loop instead of looping until the 600s ceiling kills it.
• Defense in depth: when a review-trigger run is marked FAILED/timed-out, the watchdog should update the orphaned 🔍 Review in progress… comment to a ⚠️ Review did not complete message (with a conversation link), so PRs are never silently stuck.

Reproduction

1. Trigger a Cloud Automation PR review on a non-trivial PR (label review-this / request review from the bot).
2. Tail Datadog: cluster_name:prod-runtime "SecretStr is not JSON serializable" (Bug 1, on nearly every run) and service:automation cluster_name:prod-core "Run timed out" + cluster_name:prod-runtime "Command timed out after 600 seconds" (Bug 2).
3. For a run that gets stuck, confirm on its runtime-* pod: LLM produced empty response loop → Command timed out after 600 seconds → SIGTERM, and no api.github.com review POST. The PR comment stays at 🔍 Review in progress….

Environment

• OpenHands Cloud, prod-runtime runtime-pods + prod-core automation service.
• Reviewer = Cloud event-triggered automation (pull_request.labeled on review-this), prompt preset (presets/prompt/sdk_main.py + setup.sh).
• Evidence window: 2026-06-22, multiple conversations/pods/orgs.

Suggested fix order

1. Fix Bug 1 (SecretStr-safe serialization in the event-webhook poster) — eliminates the retry storm and restores the UI event stream. Low risk.
2. Break/bound the empty-LLM-response loop in the agent loop so a flaky LLM response can't consume the entire run budget.
3. Raise/cache the run budget (raise AUTOMATION_MAX_RUN_DURATION or make per-automation; pre-bake/cache the SDK venv so setup.sh isn't re-installing from PyPI every run).
4. Watchdog-driven orphaned-comment cleanup for FAILED review runs.

---

Note: This issue was created by an AI agent (OpenHands) on behalf of the user investigating the failing reviewer.