OpenHands / OpenHands/software-agent-sdk

Add cheap load tests for concurrent conversations

Open
#4,477 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ready-for-dev
Dominant language
Python
Stars
1.1k
Forks
539
Avg merge
1d 19h
Merged PRs (30d)
137

Description

Problem

A regression recently prevented LLM calls from running in parallel, effectively making the agent server single-threaded whenever LLM calls were moderately slow. We do not currently have a cheap load test that would catch this behavior across concurrent conversations.

Related context:

  • OpenHands/OpenHands#16459
  • OpenHands/software-agent-sdk#4473

Proposed test

Add a deterministic concurrent-conversation load test for the agent server/SDK that:

  • Starts multiple conversations at the same time.
  • Uses a cheap local/fake LLM implementation with a controlled delay, avoiding paid external LLM calls.
  • Verifies LLM requests overlap rather than execute serially.
  • Records or asserts aggregate wall-clock time/throughput with enough tolerance to avoid CI flakiness.
  • Exercises the real conversation and LLM request path involved in production, rather than testing only an isolated helper.
  • Produces useful diagnostics when concurrency regresses.

Acceptance criteria

  • A repeatable test runs several conversations concurrently through the real agent-server/SDK path.
  • The test fails when LLM calls are unintentionally serialized.
  • The test requires no paid model credentials and is inexpensive enough for regular CI execution.
  • Timing thresholds are documented and tolerant of normal CI variance.
  • The test is included in an appropriate CI workflow.

This issue was created by an AI agent (OpenHands) on behalf of the requester.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the real agent-server/SDK conversation and LLM request path described in the issue, then identify the appropriate CI workflow. Add a deterministic test using a cheap local or fake LLM with controlled delay, and verify it detects serialized calls while remaining tolerant of normal CI variance and producing useful diagnostics.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.