Add a committed regression test for the model-fallback path
Nobody has claimed this yet.
- Dominant language
- No language data
- Stars
- 0
- Forks
- 0
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 8
Description
Context
The model-fallback loop (outer model list × inner retry) in .github/workflows/gemini-reviewer.yml was validated only with an ad-hoc mock server during PR #7 (a model returning 404 fell through to the next, which returned 200). There is no committed test, so future changes (e.g. the boilerplate-refactor in #8, or a new provider) can silently break fallback without anyone noticing.
Goal
Add a regression test that spins up a mock OpenAI-compatible server and asserts:
- Fallback works — a removed model (HTTP 400/404/422) is skipped and the next model is used; the comment heading names the model that actually reviewed.
- All-fail path — when every model fails, an
All models failedcomment is posted (not a hang/crash). - Fail-fast on auth — a 401/403 (or permanent 429) aborts instead of burning the list.
Approach
A small script (bash or node) that starts the mock server, exports PROVIDER=openrouter MODELS="removed-model good-model" plus a dummy key/endpoint, sources the relevant part of the workflow's run step, and checks review_report.md. Run it in CI on changes to the workflow, or as a manual/local check.
Acceptance
- Test reproduces the 3 scenarios above.
- It fails if fallback regresses.
- Documented in AGENTS.md/README under testing.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the run step in .github/workflows/gemini-reviewer.yml and inspect how it produces review_report.md. Build the mock-server check for fallback, all-fail, and auth fail-fast cases, then run it locally or in CI. Done means all three scenarios fail on regression and the test is documented in AGENTS.md or README.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, shell
- Domain
- ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100