azurenoops / azurenoops/spin_agent
[DEF-006] CrossTenantLookupReturns404Tests — all 10 tests silently skip in Docker-less CI via SkippableFact
- Dominant language
- C#
- Stars
- 3
- Forks
- 1
- Avg merge
- 11h 20m
- Merged PRs (30d)
- 70
Description
## Summary
All 10 cross-tenant isolation test cases in `CrossTenantLookupReturns404Tests.cs` use `[SkippableFact]` with `Skip.IfNot(_fx.DockerAvailable)`. In any CI environment without Docker (standard GitHub Actions runner), every test in this suite silently passes as skipped — providing false-green coverage for tenant isolation.
## Location
- `tests/.../CrossTenantLookupReturns404Tests.cs`
- All 10 test methods annotated `[SkippableFact]` + `Skip.IfNot(_fx.DockerAvailable)`
## Severity
**LOW** — No runtime breakage today, but tenant isolation regressions can ship undetected. A cross-tenant data leak would violate FedRAMP SC-4.
> **Note:** This is distinct from #659 (integration test project never running). That issue is about a missing project reference in CI. This issue is about in-suite silent skipping via `SkippableFact` — the test project runs, but these 10 tests always report Skipped.
## Steps to Reproduce
1. Run `dotnet test` in an environment without Docker
2. All 10 `CrossTenantLookup*` tests report "Skipped" (counted as green)
3. A cross-tenant regression would not be caught
## Fix Direction
**Option A:** Add a Docker service container to the CI job so Docker is available and the tests run.
**Option B:** Refactor the fixture to use an in-memory or mock store that doesn't require Docker, removing the `SkippableFact` guard entirely.
**Option C (minimum):** Add a CI assertion that counts skipped tests in this suite and fails the pipeline if all 10 are skipped.
## Evidence Source
- Star-Lord QA audit `84b7431961aa4750`
- Banner consolidated findings `539e6ef45fe842b8`
Contributor guide
Research direction
Start with tests/.../CrossTenantLookupReturns404Tests.cs and run dotnet test in an environment without Docker to confirm that all 10 CrossTenantLookup tests are skipped. Review the fixture's DockerAvailable guard and the available CI test configuration, then choose and implement one of the stated approaches. Done means the suite no longer silently reports all 10 tests as skipped and cross-tenant isolation remains checked in CI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100