dotnet / dotnet/sdk

Expand snapshot-update automation beyond the completions folder and type

Open
#54,783 0 comments 0 reactions 0 assignees View on GitHub
untriaged
Dominant language
C#
Stars
3.2k
Forks
1.3k
PR merge metrics
PR metrics pending

Description

## Summary

The snapshot-update automation only covers CLI **completion** snapshots living under
`test/dotnet.Tests/CompletionTests/snapshots/`

Other snapshot-based tests in `dotnet.Tests` produce `.verified.*` files outside that one
folder and with other extensions. When those tests' expected output changes, the automation
finds nothing to update and reports "no snapshot files needed to be updated" even though a
snapshot test is failing. The update has to be done by hand instead.

We should generalize the automation so it regenerates and stages **all** snapshot families in
`dotnet.Tests`, not just the completions folder and its shell types.

## Current behavior

The path and extension scope is hard-coded:

1. **Regeneration targets** — `CompareCliSnapshots` / `UpdateCliSnapshots` in
[test/dotnet.Tests/dotnet.Tests.csproj](../../test/dotnet.Tests/dotnet.Tests.csproj) only
operate on `CompletionTests\snapshots\**`. `.received.*` files produced by tests in other
folders are never copied or promoted to `.verified.*`.
2. **Change detection** — the diff is scoped to `test/dotnet.Tests/CompletionTests/snapshots/`
3. **Selection / staging** — only files in the above category are permitted.

## Concrete example

The MTP help-output snapshot test stores its baseline at:

```
test/dotnet.Tests/CommandTests/Test/snapshots/MTPHelpSnapshotTests.VerifyMTPHelpOutput.verified.txt
```

This is:

- in a **different folder** (`CommandTests/Test/snapshots`, not `CompletionTests/snapshots`), and
a .txt file output.

## Snapshot families in the repo today

| Family | Location | Extensions |
| --------------------- | ----------------------------------------------------- | --------------------------------------- |
| CLI completions | `test/dotnet.Tests/CompletionTests/snapshots/` | `.sh`, `.zsh`, `.fish`, `.ps1`, `.nu` |
| MTP help output | `test/dotnet.Tests/CommandTests/Test/snapshots/` | `.txt` |

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.