azurenoops / azurenoops/spin_agent

[Integration] SspAuthoring: FullSspLifecycle + BatchPopulate failing on main (quarantine tracking)

Open
#856 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
3
Forks
1
Avg merge
18h 9m
Merged PRs (30d)
43

Description

## Summary

Two `SspAuthoringIntegrationTests` are failing on `main` independent of any open PR. They are quarantined in the CI `integration-tests` job (via `fix/ci-integration-gate-quarantine`) to preserve merge velocity while this issue is resolved.

**Quarantine lane must trend to zero — do not add new tests without a companion tracking issue.**

## Failing tests

- `Ato.Copilot.Tests.Integration.Tools.SspAuthoringIntegrationTests.FullSspLifecycle_WriteNarrativeSuggestBatchPopulateProgressGenerateSsp`
- `Ato.Copilot.Tests.Integration.Tools.SspAuthoringIntegrationTests.BatchPopulate_Idempotent_SecondRunSkipsAll`

## Evidence

- First confirmed: CI run 33505808350 (pre-existing on `main`, not introduced by any PR)
- Also failing: run 33535952002, run 33542685428 (413 passed, 4 failed)
- Error class (partially): `InvalidOperationException` — `.GetDouble()` called on nullable `confidence` JSON field returned by `SuggestNarrativeTool`
- Note: PR #855 (`fix/integration-test-cli-exitcode`) guards the assertion in `FullSspLifecycle` but the `BatchPopulate` test and the full SSP lifecycle may have additional failure modes not yet root-caused

## Root cause hypothesis

`SuggestNarrativeTool` returns `confidence: null` when no grounding signal exists (template path, no model call invoked). `FullSspLifecycle` asserted `.GetDouble() > 0` unconditionally. PR #855 guards that assertion, but `BatchPopulate_Idempotent_SecondRunSkipsAll` may fail for a different reason — full local run with verbose output is needed to confirm.

## Required resolution

1. Run both tests locally: `dotnet test ... --filter "FullyQualifiedName~SspAuthoringIntegrationTests" --logger "console;verbosity=detailed"`
2. Capture full exception chain for each failure
3. Fix root cause (test assertion correctness or tool contract — not masking)
4. Remove both test names from the quarantine filter in `.github/workflows/ci.yml`
5. Close this issue

## Owner

Unassigned — needs owner.

Contributor guide

Open the contributing guide

Research direction

Start by running the two named SspAuthoringIntegrationTests with the detailed dotnet test filter from the issue and capture each full exception chain. Inspect SuggestNarrativeTool's nullable confidence behavior and the quarantine entries in .github/workflows/ci.yml. Done means both tests pass for the correct reason and their names are removed from the quarantine filter.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.