OpenHands / OpenHands/software-agent-sdk
Path-gated test jobs omit openhands-sdk/**, so SDK changes can break downstream tests with every required check green
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 542
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 137
Description
Bug Description
Path-gated test jobs in tests.yml omit openhands-sdk/** from their changed-files filters, even though every downstream package depends on openhands-sdk. Because each job's steps are individually guarded by if: steps.changed.outputs.any_changed == 'true', a job whose filter matches nothing runs no tests and still reports success — a green required check that executed nothing.
This is how main went red on agent-server-tests at 56b016e7f, two commits after the change that actually broke it, with every required check green on the breaking PR.
Expected Behavior
A change to openhands-sdk/** runs the test suites of the packages that depend on it, so breaking a downstream test cannot merge with a green required check. A green required check should mean tests ran and passed.
Actual Behavior
Reproduce the breakage the gap allowed through:
git checkout fc9646ad8 # the commit whose CI was green
uv sync
uv run pytest tests/agent_server/test_auto_title_span_metadata.py
Fails deterministically:
tests/agent_server/test_auto_title_span_metadata.py::test_auto_title_llm_span_joins_the_conversation_trace
E AssertionError: assert 'llm.gpt-4o' == 'conversation.generate_title'
Yet agent-server-tests reported success on that commit. Step-level conclusions from the two main runs:
| step | fc9646ad8 (#4836) |
56b016e7f (#4839) |
|---|---|---|
| Detect Agent Server changes | success | success |
| Install deps | skipped | success |
| Run Agent Server tests | skipped | failure |
| job conclusion | success | failure |
#4836 gave Telemetry.on_request its own llm.<model> span, staling a parent assertion in tests/agent_server/test_auto_title_span_metadata.py. Its diff was openhands-sdk/** + tests/sdk/** only, so the gate never opened. #4839 touched agent-server paths and so became the first commit to actually run the test. #4836 is the cause; #4839 is only the detector. It also explains why #4836 updated the SDK-side span test but not the agent-server counterpart — CI never showed the author the second failure.
Steps to Reproduce
The filter gap itself, independent of that one breakage:
grep -A8 "Detect Agent Server changes" .github/workflows/tests.yml
Declared dependencies vs. filters:
openhands-agent-server/pyproject.toml: "openhands-sdk"
openhands-tools/pyproject.toml: "openhands-sdk"
openhands-workspace/pyproject.toml: "openhands-sdk"
| job | openhands-sdk/** in filter |
|---|---|
sdk-tests |
yes |
acp-live-tests |
yes (two specific files) |
tools-tests |
no |
windows-tests |
no |
agent-server-tests |
no |
agent-server-stress-tests |
no |
workspace-tests |
no |
So any SDK-only change can break tools, agent_server or workspace tests while all of their required checks report green. Not specific to spans or to #4836.
Separately: cross-tests filters on tests/** and openhands/**, but there is no top-level openhands/ directory (packages live at openhands-sdk/openhands/...), so that pattern appears dead. tests/** is broad enough that the job still runs in practice.
Acceptance Criteria
-
openhands-sdk/**is added to thechanged-filesfilters fortools-tests,windows-tests,agent-server-tests,agent-server-stress-testsandworkspace-tests. - The dead
openhands/**pattern incross-testsis corrected or removed. - A skipped path-gated job is distinguishable from a passing one at the required-check level (e.g. a gate job that always runs and reports per-suite status), so a green tick means tests ran.
- A regression test or CI assertion covers the dependency direction, so a new downstream package cannot be added with a filter that omits its dependencies.
Installation Method
Source / development checkout
SDK Version
main @ 56b016e7f
The resulting test breakage is fixed separately in #4911; this issue is about the CI gap that let it merge.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in .github/workflows/tests.yml by inspecting the changed-files filters and step guards for tools-tests, windows-tests, agent-server-tests, agent-server-stress-tests, workspace-tests, and cross-tests. Compare those filters with the openhands-sdk dependencies listed in the package pyproject.toml files, then reproduce the current behavior with the provided grep and pytest commands. Done means dependency changes trigger downstream suites, skipped jobs are distinguishable from passing checks, and coverage prevents future filter omissions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ci-cd, devops, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100