OpenHands / OpenHands/software-agent-sdk
Investigate frozen LLM fixture replay bypass in hello-world cross test
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 539
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 137
Description
Summary
tests/cross/test_hello_world.py appears to load real frozen LLM completion fixtures via create_real_llm_responses_from_fixtures(fncall_raw_logs), but then immediately replaces them with synthetic mock responses before running the main hello-world flow.
real_responses = self.create_real_llm_responses_from_fixtures(fncall_raw_logs)
# Always use mock responses for consistent behavior
# Real fixture data may have different tool call sequences than current agent
real_responses = self.create_mock_llm_responses()
That makes the test look like a real frozen-trajectory replay test while mostly exercising hand-written mocks. We should revisit whether these fixtures can become a first-class deterministic replay suite for real LLM trajectories, or rename/split the current test so the behavior is clearer.
This issue was created by an AI agent (OpenHands) on behalf of the user.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in tests/cross/test_hello_world.py and trace create_real_llm_responses_from_fixtures(fncall_raw_logs) through the hello-world flow, then compare it with create_mock_llm_responses(). Determine whether the frozen fixtures can support deterministic real-trajectory replay; otherwise make the test naming or split clearly distinguish fixture replay from mock behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, testing-qa
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100