OpenHands / OpenHands/software-agent-sdk

Investigate frozen LLM fixture replay bypass in hello-world cross test

Open
#3,549 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Stale
Dominant language
Python
Stars
1.1k
Forks
539
Avg merge
1d 19h
Merged PRs (30d)
137

Description

Summary

tests/cross/test_hello_world.py appears to load real frozen LLM completion fixtures via create_real_llm_responses_from_fixtures(fncall_raw_logs), but then immediately replaces them with synthetic mock responses before running the main hello-world flow.

real_responses = self.create_real_llm_responses_from_fixtures(fncall_raw_logs)

# Always use mock responses for consistent behavior
# Real fixture data may have different tool call sequences than current agent
real_responses = self.create_mock_llm_responses()

That makes the test look like a real frozen-trajectory replay test while mostly exercising hand-written mocks. We should revisit whether these fixtures can become a first-class deterministic replay suite for real LLM trajectories, or rename/split the current test so the behavior is clearer.

This issue was created by an AI agent (OpenHands) on behalf of the user.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in tests/cross/test_hello_world.py and trace create_real_llm_responses_from_fixtures(fncall_raw_logs) through the hello-world flow, then compare it with create_mock_llm_responses(). Determine whether the frozen fixtures can support deterministic real-trajectory replay; otherwise make the test naming or split clearly distinguish fixture replay from mock behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, testing-qa
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.