dimensionalOS / dimensionalOS/dimos
Flaky tests: test_mcp_client (test_prompt, test_image) and test_where_am_i fail intermittently in tests (3.12) CI job
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.5k
- Forks
- 808
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 233
Description
Observed in CI for PR #3960 (run: https://github.com/dimensionalOS/dimos/actions/runs/34088875125, job tests (3.12, ubuntu-latest, false)).
Three agent-related tests failed in the main pytest run, and all three passed on the automatic --lf re-run in the same job, so this looks like flakiness rather than a real regression. The 3.10/3.11/3.13/3.14 jobs passed; the 3.14t job was canceled by fail-fast before finishing.
1. dimos/agents/mcp/test_mcp_client.py::test_prompt
ExceptionGroup: safe_thread_map failed (1 sub-exception) during ModuleCoordinator.start_all_modules(). Root cause in the worker:
File "dimos/agents/testing/agent_test_runner.py", line 51, in start
self.register_disposable(Disposable(self.agent.subscribe(self._on_agent_message)))
File "dimos/core/stream.py", line 303, in subscribe
return self.transport.subscribe(cb, self)
AttributeError: 'NoneType' object has no attribute 'subscribe'
AgentTestRunner.start() subscribed to self.agent before its transport was connected — looks like a startup race.
2. dimos/agents/mcp/test_mcp_client.py::test_image
E AssertionError: assert 'cafe' in 'you are currently on bourbon street.'
response = 'you are currently on bourbon street.'
dimos/agents/mcp/test_mcp_client.py:197
The agent answered a previous test's question ("What street am I on?") instead of describing the picture — suggests cross-test state bleed (shared LLM history / fixture ordering?) or model nondeterminism.
3. dimos/agents/skills/test_google_maps_skill_container.py::test_where_am_i
TimeoutError: RPC call to 'AgentTestRunner/set_transport' timed out after 120.0 seconds
The RPC to set_transport during ModuleCoordinator._connect_streams() (module_coordinator.py:332) never returned — the module appears to have been wedged (possibly by the same startup race as failure 1).
All three failures involve AgentTestRunner / the MCP agent test harness and happened in the same job, so they may share a single root cause (startup/ordering race under pytest-xdist).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with dimos/agents/testing/agent_test_runner.py, dimos/agents/mcp/test_mcp_client.py, and dimos/agents/skills/test_google_maps_skill_container.py; inspect the startup and transport paths referenced in the failures. Run the named tests under the affected Python 3.12 CI configuration and determine whether the shared AgentTestRunner race or test state bleed is reproducible. Done means the three tests pass reliably without automatic --lf reruns or transport timeouts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100