OpenHands / OpenHands/software-agent-sdk

Path-gated test jobs omit openhands-sdk/**, so SDK changes can break downstream tests with every required check green

Open
#4,912 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug ci github_actions priority:high ready-for-dev
Dominant language
Python
Stars
1.1k
Forks
542
Avg merge
1d 19h
Merged PRs (30d)
137

Description

Bug Description

Path-gated test jobs in tests.yml omit openhands-sdk/** from their changed-files filters, even though every downstream package depends on openhands-sdk. Because each job's steps are individually guarded by if: steps.changed.outputs.any_changed == 'true', a job whose filter matches nothing runs no tests and still reports success — a green required check that executed nothing.

This is how main went red on agent-server-tests at 56b016e7f, two commits after the change that actually broke it, with every required check green on the breaking PR.

Expected Behavior

A change to openhands-sdk/** runs the test suites of the packages that depend on it, so breaking a downstream test cannot merge with a green required check. A green required check should mean tests ran and passed.

Actual Behavior

Reproduce the breakage the gap allowed through:

git checkout fc9646ad8   # the commit whose CI was green
uv sync
uv run pytest tests/agent_server/test_auto_title_span_metadata.py

Fails deterministically:

tests/agent_server/test_auto_title_span_metadata.py::test_auto_title_llm_span_joins_the_conversation_trace
E   AssertionError: assert 'llm.gpt-4o' == 'conversation.generate_title'

Yet agent-server-tests reported success on that commit. Step-level conclusions from the two main runs:

step fc9646ad8 (#4836) 56b016e7f (#4839)
Detect Agent Server changes success success
Install deps skipped success
Run Agent Server tests skipped failure
job conclusion success failure

#4836 gave Telemetry.on_request its own llm.<model> span, staling a parent assertion in tests/agent_server/test_auto_title_span_metadata.py. Its diff was openhands-sdk/** + tests/sdk/** only, so the gate never opened. #4839 touched agent-server paths and so became the first commit to actually run the test. #4836 is the cause; #4839 is only the detector. It also explains why #4836 updated the SDK-side span test but not the agent-server counterpart — CI never showed the author the second failure.

Steps to Reproduce

The filter gap itself, independent of that one breakage:

grep -A8 "Detect Agent Server changes" .github/workflows/tests.yml

Declared dependencies vs. filters:

openhands-agent-server/pyproject.toml:  "openhands-sdk"
openhands-tools/pyproject.toml:         "openhands-sdk"
openhands-workspace/pyproject.toml:     "openhands-sdk"
job openhands-sdk/** in filter
sdk-tests yes
acp-live-tests yes (two specific files)
tools-tests no
windows-tests no
agent-server-tests no
agent-server-stress-tests no
workspace-tests no

So any SDK-only change can break tools, agent_server or workspace tests while all of their required checks report green. Not specific to spans or to #4836.

Separately: cross-tests filters on tests/** and openhands/**, but there is no top-level openhands/ directory (packages live at openhands-sdk/openhands/...), so that pattern appears dead. tests/** is broad enough that the job still runs in practice.

Acceptance Criteria
  • openhands-sdk/** is added to the changed-files filters for tools-tests, windows-tests, agent-server-tests, agent-server-stress-tests and workspace-tests.
  • The dead openhands/** pattern in cross-tests is corrected or removed.
  • A skipped path-gated job is distinguishable from a passing one at the required-check level (e.g. a gate job that always runs and reports per-suite status), so a green tick means tests ran.
  • A regression test or CI assertion covers the dependency direction, so a new downstream package cannot be added with a filter that omits its dependencies.
Installation Method

Source / development checkout

SDK Version

main @ 56b016e7f


The resulting test breakage is fixed separately in #4911; this issue is about the CI gap that let it merge.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in .github/workflows/tests.yml by inspecting the changed-files filters and step guards for tools-tests, windows-tests, agent-server-tests, agent-server-stress-tests, workspace-tests, and cross-tests. Compare those filters with the openhands-sdk dependencies listed in the package pyproject.toml files, then reproduce the current behavior with the provided grep and pytest commands. Done means dependency changes trigger downstream suites, skipped jobs are distinguishable from passing checks, and coverage prevents future filter omissions.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, devops, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.