dimensionalOS / dimensionalOS/dimos

Flaky tests: test_mcp_client (test_prompt, test_image) and test_where_am_i fail intermittently in tests (3.12) CI job

Open
#3,962 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.5k
Forks
808
Avg merge
3d 5h
Merged PRs (30d)
233

Description

Observed in CI for PR #3960 (run: https://github.com/dimensionalOS/dimos/actions/runs/34088875125, job tests (3.12, ubuntu-latest, false)).

Three agent-related tests failed in the main pytest run, and all three passed on the automatic --lf re-run in the same job, so this looks like flakiness rather than a real regression. The 3.10/3.11/3.13/3.14 jobs passed; the 3.14t job was canceled by fail-fast before finishing.

1. dimos/agents/mcp/test_mcp_client.py::test_prompt

ExceptionGroup: safe_thread_map failed (1 sub-exception) during ModuleCoordinator.start_all_modules(). Root cause in the worker:

File "dimos/agents/testing/agent_test_runner.py", line 51, in start
    self.register_disposable(Disposable(self.agent.subscribe(self._on_agent_message)))
File "dimos/core/stream.py", line 303, in subscribe
    return self.transport.subscribe(cb, self)
AttributeError: 'NoneType' object has no attribute 'subscribe'

AgentTestRunner.start() subscribed to self.agent before its transport was connected — looks like a startup race.

2. dimos/agents/mcp/test_mcp_client.py::test_image
E       AssertionError: assert 'cafe' in 'you are currently on bourbon street.'
response   = 'you are currently on bourbon street.'
dimos/agents/mcp/test_mcp_client.py:197

The agent answered a previous test's question ("What street am I on?") instead of describing the picture — suggests cross-test state bleed (shared LLM history / fixture ordering?) or model nondeterminism.

3. dimos/agents/skills/test_google_maps_skill_container.py::test_where_am_i
TimeoutError: RPC call to 'AgentTestRunner/set_transport' timed out after 120.0 seconds

The RPC to set_transport during ModuleCoordinator._connect_streams() (module_coordinator.py:332) never returned — the module appears to have been wedged (possibly by the same startup race as failure 1).

All three failures involve AgentTestRunner / the MCP agent test harness and happened in the same job, so they may share a single root cause (startup/ordering race under pytest-xdist).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with dimos/agents/testing/agent_test_runner.py, dimos/agents/mcp/test_mcp_client.py, and dimos/agents/skills/test_google_maps_skill_container.py; inspect the startup and transport paths referenced in the failures. Run the named tests under the affected Python 3.12 CI configuration and determine whether the shared AgentTestRunner race or test state bleed is reproducible. Done means the three tests pass reliably without automatic --lf reruns or transport timeouts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.