OpenHands / OpenHands/software-agent-sdk
Add cheap load tests for concurrent conversations
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 539
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 137
Description
Problem
A regression recently prevented LLM calls from running in parallel, effectively making the agent server single-threaded whenever LLM calls were moderately slow. We do not currently have a cheap load test that would catch this behavior across concurrent conversations.
Related context:
- OpenHands/OpenHands#16459
- OpenHands/software-agent-sdk#4473
Proposed test
Add a deterministic concurrent-conversation load test for the agent server/SDK that:
- Starts multiple conversations at the same time.
- Uses a cheap local/fake LLM implementation with a controlled delay, avoiding paid external LLM calls.
- Verifies LLM requests overlap rather than execute serially.
- Records or asserts aggregate wall-clock time/throughput with enough tolerance to avoid CI flakiness.
- Exercises the real conversation and LLM request path involved in production, rather than testing only an isolated helper.
- Produces useful diagnostics when concurrency regresses.
Acceptance criteria
- A repeatable test runs several conversations concurrently through the real agent-server/SDK path.
- The test fails when LLM calls are unintentionally serialized.
- The test requires no paid model credentials and is inexpensive enough for regular CI execution.
- Timing thresholds are documented and tolerant of normal CI variance.
- The test is included in an appropriate CI workflow.
This issue was created by an AI agent (OpenHands) on behalf of the requester.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the real agent-server/SDK conversation and LLM request path described in the issue, then identify the appropriate CI workflow. Add a deterministic test using a cheap local or fake LLM with controlled delay, and verify it detects serialized calls while remaining tolerant of normal CI variance and producing useful diagnostics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100