OpenHands / OpenHands/software-agent-sdk
OpenHands Fork Audit - CRITICAL: No regression tests prove air-gap works; verifier is source-scan only
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 539
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 137
Description
FLEET AUDIT FINDING: OpenHands Zentra fork — test coverage audit
Location: patches/sdk/verify_sdk_hardening.py (source scan + behavioral proof only); NO integration tests in vendor/
Severity: CRITICAL (P1) — Test coverage gap prevents detecting egress regressions on upstream merge
The Issue:
The hardening is verified by:
- patches/sdk/verify_sdk_hardening.py - a source-scan + behavioral unit-test verifier
- Boot-time assert_sdk_hardened() - runtime sentinel check
But there are NO actual integration/regression tests in the codebase that:
- Run a real agent conversation
- Attempt off-box egress (e.g., openai/ model, remote MCP server, LookupSecret from attacker host)
- Assert denial + fail-loud behavior
Why critical:
- When upstream OpenHands is pulled, there's no test suite to catch if a hardening patch is reverted or egress is re-enabled
- The verifier runs at build-time and boot-time, but silent regressions between vendor merges can slip through
- The gate is "ASSERT sentinel present" not "PROVE behavior works"
Current state:
- verify_sdk_hardening.py asserts ZENTRA-SDK-PATCH-0005 sentinel is in llm.py (the guard exists)
- It does NOT test that an agent conversation with openai/ model actually fails
- It does NOT test that LookupSecret to an attacker host is refused
- It does NOT test that a plugin from github.com/evil/repo is blocked
Proof of gap:
Search vendor/ for tests/: only vendor/openhands-sdk/openhands/sdk/testing (no zentra tests). Run the test suite on the vendor packages—no air-gap regression suite.
Fix guidance:
Add integration test suite (e.g., tests/openhands_fork/test_zentra_airgap.py) that:
- Spawns a test agent-server with ZENTRA_LLM_ENFORCE=1
- Attempts conversations with off-box models (openai/gpt-4o) → expect ZentraLLMPolicyError
- Attempts to add remote MCP servers → expect rejection
- Attempts to use LookupSecret with off-box host → expect failure (may need agent workspace wrapper)
- Attempts to load plugins from github:evil/repo → expect refusal
- Verifies MCP /api/mcp/test dial surface rejects off-box urls
These should be gated on ZENTRA_LLM_ENFORCE so they're independent of the shipped unit but prove the guards work.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read patches/sdk/verify_sdk_hardening.py and inspect vendor/openhands-sdk/openhands/sdk/testing to understand the existing verifier and available test utilities. Add tests/openhands_fork/test_zentra_airgap.py, starting with a test agent-server using ZENTRA_LLM_ENFORCE=1. Done means off-box models, remote MCP servers, LookupSecret hosts, GitHub plugins, and the MCP dial surface are rejected with the expected failure behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, security, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100