azurenoops / azurenoops/spin_agent

[DEF-006] CrossTenantLookupReturns404Tests — all 10 tests silently skip in Docker-less CI via SkippableFact

Closed
#827 0 comments 0 reactions 0 assignees View on GitHub
agent:war-machine bug priority:low
Dominant language
C#
Stars
3
Forks
1
Avg merge
11h 20m
Merged PRs (30d)
70

Description

## Summary

All 10 cross-tenant isolation test cases in `CrossTenantLookupReturns404Tests.cs` use `[SkippableFact]` with `Skip.IfNot(_fx.DockerAvailable)`. In any CI environment without Docker (standard GitHub Actions runner), every test in this suite silently passes as skipped — providing false-green coverage for tenant isolation.

## Location

- `tests/.../CrossTenantLookupReturns404Tests.cs`
- All 10 test methods annotated `[SkippableFact]` + `Skip.IfNot(_fx.DockerAvailable)`

## Severity

**LOW** — No runtime breakage today, but tenant isolation regressions can ship undetected. A cross-tenant data leak would violate FedRAMP SC-4.

> **Note:** This is distinct from #659 (integration test project never running). That issue is about a missing project reference in CI. This issue is about in-suite silent skipping via `SkippableFact` — the test project runs, but these 10 tests always report Skipped.

## Steps to Reproduce

1. Run `dotnet test` in an environment without Docker
2. All 10 `CrossTenantLookup*` tests report "Skipped" (counted as green)
3. A cross-tenant regression would not be caught

## Fix Direction

**Option A:** Add a Docker service container to the CI job so Docker is available and the tests run.

**Option B:** Refactor the fixture to use an in-memory or mock store that doesn't require Docker, removing the `SkippableFact` guard entirely.

**Option C (minimum):** Add a CI assertion that counts skipped tests in this suite and fails the pipeline if all 10 are skipped.

## Evidence Source

- Star-Lord QA audit `84b7431961aa4750`
- Banner consolidated findings `539e6ef45fe842b8`

Contributor guide

Open the contributing guide

Research direction

Start with tests/.../CrossTenantLookupReturns404Tests.cs and run dotnet test in an environment without Docker to confirm that all 10 CrossTenantLookup tests are skipped. Review the fixture's DockerAvailable guard and the available CI test configuration, then choose and implement one of the stated approaches. Done means the suite no longer silently reports all 10 tests as skipped and cross-tenant isolation remains checked in CI.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.