OpenHands / OpenHands/software-agent-sdk

OpenHands Fork Audit - CRITICAL: No regression tests prove air-gap works; verifier is source-scan only

Open
#4,262 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
1.1k
Forks
539
Avg merge
1d 19h
Merged PRs (30d)
137

Description

FLEET AUDIT FINDING: OpenHands Zentra fork — test coverage audit

Location: patches/sdk/verify_sdk_hardening.py (source scan + behavioral proof only); NO integration tests in vendor/

Severity: CRITICAL (P1) — Test coverage gap prevents detecting egress regressions on upstream merge

The Issue:
The hardening is verified by:

  1. patches/sdk/verify_sdk_hardening.py - a source-scan + behavioral unit-test verifier
  2. Boot-time assert_sdk_hardened() - runtime sentinel check

But there are NO actual integration/regression tests in the codebase that:

  • Run a real agent conversation
  • Attempt off-box egress (e.g., openai/ model, remote MCP server, LookupSecret from attacker host)
  • Assert denial + fail-loud behavior

Why critical:

  • When upstream OpenHands is pulled, there's no test suite to catch if a hardening patch is reverted or egress is re-enabled
  • The verifier runs at build-time and boot-time, but silent regressions between vendor merges can slip through
  • The gate is "ASSERT sentinel present" not "PROVE behavior works"

Current state:

  • verify_sdk_hardening.py asserts ZENTRA-SDK-PATCH-0005 sentinel is in llm.py (the guard exists)
  • It does NOT test that an agent conversation with openai/ model actually fails
  • It does NOT test that LookupSecret to an attacker host is refused
  • It does NOT test that a plugin from github.com/evil/repo is blocked

Proof of gap:
Search vendor/ for tests/: only vendor/openhands-sdk/openhands/sdk/testing (no zentra tests). Run the test suite on the vendor packages—no air-gap regression suite.

Fix guidance:
Add integration test suite (e.g., tests/openhands_fork/test_zentra_airgap.py) that:

  1. Spawns a test agent-server with ZENTRA_LLM_ENFORCE=1
  2. Attempts conversations with off-box models (openai/gpt-4o) → expect ZentraLLMPolicyError
  3. Attempts to add remote MCP servers → expect rejection
  4. Attempts to use LookupSecret with off-box host → expect failure (may need agent workspace wrapper)
  5. Attempts to load plugins from github:evil/repo → expect refusal
  6. Verifies MCP /api/mcp/test dial surface rejects off-box urls

These should be gated on ZENTRA_LLM_ENFORCE so they're independent of the shipped unit but prove the guards work.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read patches/sdk/verify_sdk_hardening.py and inspect vendor/openhands-sdk/openhands/sdk/testing to understand the existing verifier and available test utilities. Add tests/openhands_fork/test_zentra_airgap.py, starting with a test agent-server using ZENTRA_LLM_ENFORCE=1. Done means off-box models, remote MCP servers, LookupSecret hosts, GitHub plugins, and the MCP dial surface are rejected with the expected failure behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, security, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.