Add pre-release load tests as a blocking release gate

Problem

Recent production-grade perf regressions in the agent server caused cross-conversation wedging and serialized execution. Each was caught by review/incident rather than by an automated gate in the release process. Examples that landed on main recently:

• Global lifecycle lock deadlock — #4514, fixed by #4570 (replace global _lifecycle_lock with per-conversation locks). A stuck close()/_prepare_persisted_runtime on one conversation blocked event loading, creation, and delete for all conversations.
• Blocking work on the event loop — #4481 (bash event search glob → scandir + off the event loop) and #4417 (compose ConversationInfo off the event loop to avoid a GC wedge).
• Serialized LLM calls through global config — #4473 (stop serializing calls through global config, tracked in OpenHands/OpenHands#16459).
• Thread-pool exhaustion / lock contention — #4480, #3912, #4513 (test), #4548 (bounded-close mitigation).

We have an opt-in stress suite (tests/agent_server/stress/, -m stress) but it is not wired into the release path, and no pre-release load gate ensures these regressions can't ship again.

Desired Behavior

Add pre-release load tests that run as a required gate in the release workflow. The tests should exercise the real agent-server path (concurrent conversations, LLM calls, event search/listing, conversation create/close) with a cheap fake LLM and controlled latency/delays, so no paid credentials are required.

The release workflow note: the SDK does not currently use release-please itself (that lives in OpenHands/OpenHands and downstream repos like automation); its release path is prepare-release.yml → create-release.yml → pypi-release.yml. Wire the load gate into that release path (or, if the team later migrates the SDK to release-please, the equivalent release-PR workflow).

Acceptance Criteria

• A pre-release load-test suite runs expanded/high-confidence scenarios from tests/agent_server/stress/ (e.g. concurrent conversations, event search/listing under load, conversation create/close storms, event-loop responsiveness) against the real agent-server code path with a cheap fake LLM.
• The suite is added to the release workflow path (prepare-release/create-release, or the equivalent release-please path if/when the SDK adopts it) as a blocking gate before release.
• Each regression(s) listed above is reproducible by cherry-picking the regression onto a fixed tree such that the load tests fail on the pre-fix branches (e.g. pre-#4570, pre-#4481, pre-#4417, pre-#4473) and pass on current main.
• If current main has obvious perf defects surfaced by the new tests, those defects are fixed in the course of deploying the tests (not worked around by loosening budgets).
• Timing/memory/fd budgets are documented and tolerant of CI variance to avoid flakiness (following the existing tests/agent_server/stress/budgets.py pattern).
• The tests require no paid model credentials and are inexpensive enough to run in CI on every release (and, ideally, on PRs touching the agent-server path).
• Existing stress-suite failures are not silently skipped; the gate is required (not advisory) for release.

References

• #4570 — per-conversation locks fix
• #4514 / #4513 / #4548 — lifecycle lock deadlock + mitigations
• #4481, #4417 — blocking work off the event loop
• #4473 — global config LLM serialization (with OpenHands/OpenHands#16459)
• #4477 — existing "cheap load tests" issue (complementary; this issue adds the release gate + regression baseline)
• Existing suite: tests/agent_server/stress/ and tests/agent_server/stress/budgets.py

Notes for assignee

• Reuse the existing stress suite fixtures (conftest.py, probe.py, scripts.py, budgets.py) rather than building a parallel harness.
• Build the "fails before / passes after" proof by checking out the pre-fix commits (or reverting the specific fix) and running the gate — document which commit/branch each scenario targets.

This issue was created by an AI agent (OpenHands) on behalf of the requester.

Start with tests/agent_server/stress/ and its conftest.py, probe.py, scripts.py, and budgets.py to understand the existing stress fixtures and thresholds. Trace the prepare-release.yml, create-release.yml, and pypi-release.yml path, then identify where the required gate belongs. Done means expanded fake-LLM load scenarios run against the real agent-server path, document CI-tolerant budgets and regression evidence, and block release on failure.