azurenoops / azurenoops/spin_agent
[Integration] SspAuthoring: FullSspLifecycle + BatchPopulate failing on main (quarantine tracking)
- Dominant language
- C#
- Stars
- 3
- Forks
- 1
- Avg merge
- 18h 9m
- Merged PRs (30d)
- 43
Description
## Summary
Two `SspAuthoringIntegrationTests` are failing on `main` independent of any open PR. They are quarantined in the CI `integration-tests` job (via `fix/ci-integration-gate-quarantine`) to preserve merge velocity while this issue is resolved.
**Quarantine lane must trend to zero — do not add new tests without a companion tracking issue.**
## Failing tests
- `Ato.Copilot.Tests.Integration.Tools.SspAuthoringIntegrationTests.FullSspLifecycle_WriteNarrativeSuggestBatchPopulateProgressGenerateSsp`
- `Ato.Copilot.Tests.Integration.Tools.SspAuthoringIntegrationTests.BatchPopulate_Idempotent_SecondRunSkipsAll`
## Evidence
- First confirmed: CI run 33505808350 (pre-existing on `main`, not introduced by any PR)
- Also failing: run 33535952002, run 33542685428 (413 passed, 4 failed)
- Error class (partially): `InvalidOperationException` — `.GetDouble()` called on nullable `confidence` JSON field returned by `SuggestNarrativeTool`
- Note: PR #855 (`fix/integration-test-cli-exitcode`) guards the assertion in `FullSspLifecycle` but the `BatchPopulate` test and the full SSP lifecycle may have additional failure modes not yet root-caused
## Root cause hypothesis
`SuggestNarrativeTool` returns `confidence: null` when no grounding signal exists (template path, no model call invoked). `FullSspLifecycle` asserted `.GetDouble() > 0` unconditionally. PR #855 guards that assertion, but `BatchPopulate_Idempotent_SecondRunSkipsAll` may fail for a different reason — full local run with verbose output is needed to confirm.
## Required resolution
1. Run both tests locally: `dotnet test ... --filter "FullyQualifiedName~SspAuthoringIntegrationTests" --logger "console;verbosity=detailed"`
2. Capture full exception chain for each failure
3. Fix root cause (test assertion correctness or tool contract — not masking)
4. Remove both test names from the quarantine filter in `.github/workflows/ci.yml`
5. Close this issue
## Owner
Unassigned — needs owner.
Contributor guide
Research direction
Start by running the two named SspAuthoringIntegrationTests with the detailed dotnet test filter from the issue and capture each full exception chain. Inspect SuggestNarrativeTool's nullable confidence behavior and the quarantine entries in .github/workflows/ci.yml. Done means both tests pass for the correct reason and their names are removed from the quarantine filter.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100