Flash unit tests require Gemini credentials before installing their model mocks
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 516
- Avg merge
- 22m
- Merged PRs (30d)
- 5
Description
## Problem
The default deterministic suite promises no device or model credentials in `CONTRIBUTING.md`, but four additional Flash unit-test modules fail without Gemini credentials. This is related to #31, but extends beyond the turn-index snapshot file covered by #32 and #35.
Reproduced on upstream `086078819209c7139d6f833cfdc6d5cc80d9f19a`, Windows, Python 3.12.14, dependencies from the committed `uv.lock`. `GOOGLE_API_KEY`, `GEMINI_API_KEY`, and `OPENAI_API_KEY` were unset. No phone or model service is needed to reproduce.
## Reproduction and before log
Run in a locked development environment:
```sh
uv run pytest -q tests/unit/agents/test_flash_runner.py tests/unit/agents/test_flash_runner_ledger.py tests/unit/agents/test_flash_scrub_edge.py tests/unit/agents/test_flash_step_summarizer.py
```
```text
64 failed, 13 passed in 4.19s
```
Representative failure path (local absolute paths and SDK diagnostic payload omitted):
```text
FlashRunner.__init__
-> VisualStepSummarizer.__init__
-> get_google_llm
-> ModelFactory.create_model
-> ChatGoogleGenerativeAI
pydantic_core.ValidationError:
API key required for Gemini Developer API.
```
There is also a configuration-only test that reaches provider validation:
```text
test_flash_config_and_builder
-> AgentConfigBuilder.build
-> add_profile
-> validate_providers
Exception: Planner requires GOOGLE_API_KEY in .env
```
## Root cause
Tests instantiate the real summarizer before assigning their fake `_llm`. Construction already resolves the utility model, with a fallback to a real Google client. Runner and compressor tests can hit the same constructor indirectly. Replacing the model after construction is too late to isolate credentials. Separately, the builder test uses the default credential validation although it only asserts Flash configuration values.
The fix should isolate both model factories at the summarizer module boundary while retaining the real summarizer, retry, compression, ledger, and runner implementations and their existing assertions. The builder already supports `validate_profiles=False` for configuration-only use. No production credential validation needs to be relaxed.
## Proposed fix validation / after log
A local test-only fix in those four modules produces:
```text
77 passed in 0.35s
```
Full deterministic suite, same environment and test selection:
```text
# Before: upstream 0860788
82 failed, 2109 passed, 4 skipped, 8 deselected
# After: four-module fixture isolation and configuration-only builder option
18 failed, 2173 passed, 4 skipped, 8 deselected, 80 warnings in 91.05s
# Exact failed-node comparison
64 resolved; 0 new failures
```
The remaining failures are outside this patch and already tracked:
| Remaining group | Count | Existing work |
| --- | ---: | --- |
| Turn-index snapshot fixture | 4 | #31, #32, #35 |
| Explorer/video analyzer mock provider values | 12 | #33, #34 |
| Readiness cache timing and ADB manifest | 2 | #18, #17, #23 |
Repository-wide lint also has 11 pre-existing errors covered by #29/#30. These are not introduced by the test fixture fix.
The commands were executed with the existing locked virtual environment's Python executable in separate upstream and fix worktrees; the `uv run` forms above are portable equivalents. Logs above are excerpts, not complete raw logs.
Contributor guide
Research direction
Start by running the four named tests under tests/unit/agents with credentials unset and trace construction through FlashRunner, VisualStepSummarizer, and the model factory. Inspect the summarizer-module fixtures and test_flash_config_and_builder, then verify the real summarizer and existing assertions still run without credentials. Done means the four modules pass and the exact comparison shows no new failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100