dotnet / dotnet/sdk

[StaticWebAssets] Zstd compression and Compression Dictionary Transport (CDT) support

Open
#53,855 0 comments 0 reactions 0 assignees View on GitHub
Area-AspNetCore untriaged
Dominant language
C#
Stars
3.2k
Forks
1.3k
PR merge metrics
PR metrics pending

Description

## Summary

Add **zstd compression** and **Compression Dictionary Transport (CDT, [RFC 9842](https://www.rfc-editor.org/rfc/rfc9842))** support to the Static Web Assets SDK. This enables zstd as a publish-time compression format alongside gzip and brotli, and introduces delta compression for returning users by using previous versions of assets as zstd dictionaries.

## Motivation

Blazor WebAssembly and other SPA applications ship significant amounts of static content (framework assemblies, JS bundles, CSS). Currently, the SDK compresses these with gzip and brotli at publish time. Two key limitations exist:

1. **No zstd support**: Zstd is now supported by all major browsers and offers a better speed/ratio tradeoff than gzip. More importantly, zstd supports external dictionaries, which enables CDT.
2. **No delta updates**: Every time users revisit an updated application, they re-download the full compressed content even if only small portions changed. For Blazor WASM apps with ~4MB of framework DLLs, a minor app update still requires re-downloading the entire payload.

Compression Dictionary Transport ([RFC 9842](https://www.rfc-editor.org/rfc/rfc9842)) solves the second problem. When a user has previously visited a site, the browser can use the cached version of a resource as a dictionary for decompressing the updated version, dramatically reducing transfer sizes for subsequent visits.

## Goals

- Add zstd as a publish-time compression format for static web assets, producing `.zst` files alongside `.gz` and `.br`
- Generalize the compression pipeline so that C# tasks are format-agnostic and driven by `CompressionFormat` MSBuild item metadata (making it straightforward to add future formats)
- Implement CDT support that produces `.dcz` (dictionary-compressed zstd) files when a previous build's asset pack is provided
- Generate an asset pack (`.zip`) at publish time containing the manifest and all assets, to be used as dictionary input for future builds
- Emit correct CDT endpoint metadata per RFC 9842 so that ASP.NET Core's content encoding negotiation can serve delta-compressed responses
- Keep dictionary support orthogonal from compression — the `UsesDictionary` metadata on `CompressionFormat` items controls whether a format participates in CDT, independent of the compression logic itself

## Non-goals

- **Runtime zstd decompression support** — this is an ASP.NET Core runtime concern (tracked separately)
- **Automatic dictionary storage/retrieval** — the user provides the previous pack path; CI pipeline integration is out of scope
- **Build-time compression** — zstd is publish-only (like brotli max quality); build-time continues to use gzip for dev server performance
- **Custom user-defined compression formats** — the pipeline is extensible for SDK-defined formats, not arbitrary user plugins
- **N→N+2 or multi-version dictionary chains** — CDT uses only the immediately previous version as dictionary

## Scenarios

### Scenario 1: Zstd compression at publish time

A developer publishes a Blazor WASM app and wants zstd-compressed assets served alongside gzip and brotli.

**No code changes needed** — zstd is automatically included in the publish compression pipeline via the `CompressionFormat` MSBuild items. The publish output includes `.zst` files for all static web assets. ASP.NET Core's content negotiation serves them to browsers that send `Accept-Encoding: zstd`.

### Scenario 2: CDT delta compression for returning users

A developer ships updates to a Blazor WASM app and wants returning users to download only the differences, not full files.

```xml

true
$(ArtifactsPath)\previous\staticwebassets.pack.zip

```

At publish:
1. The SDK reads the previous asset pack and matches current assets to previous versions by their non-fingerprinted route patterns
2. For each changed asset, the SDK compresses the current version using the previous version as a zstd dictionary, producing `.dcz` files
3. The SDK generates endpoint metadata with:
- `Use-As-Dictionary` response header on all content-negotiated responses (identity, gzip, br, zstd) so browsers cache resources for future dictionary use
- `Dictionary-Hash` endpoint property containing the SHA-256 of the dictionary (`:base64:` per RFC 9842 §2.2 Structured Fields Byte Sequence format)
- `Content-Encoding: dcb` selector on delta-compressed endpoints
- `Vary: Accept-Encoding, Available-Dictionary` for correct caching
4. A new asset pack is generated for the current version at `$(StaticWebAssetPublishAssetPackPath)` (default: `$(OutputPath)staticwebassets.pack.zip`)

The CI pipeline saves the asset pack between builds. On the next publish, the saved pack is provided as `StaticWebAssetPreviousAssetPack`.

### Scenario 3: ASP.NET Core routing integration

At request time, ASP.NET Core's `ContentEncodingNegotiationMatcherPolicy`:
1. Reads the `Dictionary-Hash` endpoint property from the manifest
2. Compares it against the `Available-Dictionary` request header sent by the browser
3. If the hash matches, serves the `.dcz` response; otherwise falls back to `br`/`gzip`/`zstd`/identity

This ensures delta-compressed responses are only served when the browser has the correct dictionary cached.

## Detailed design

### Generalized compression framework

The compression pipeline was refactored from hardcoded gzip/brotli logic to a generic, metadata-driven model:

- **`CompressionFormat` MSBuild items**: Each format declares `FileExtension`, `ContentEncoding`, and optionally `UsesDictionary` metadata
- **`ApplyCompressionNegotiation`**: Refactored to use `EndpointGroup` for route-based grouping. Quality ranking, selector generation, and header emission are all driven by format metadata, not format names
- **`ResolveCompressedAssets`**: Parses the `Formats` property and resolves format metadata generically — no hardcoded format switching

### New MSBuild tasks

| Task | Purpose |
|------|---------|
| `ZstdCompress` | `ToolTask` that invokes the SWA CLI tool for zstd compression with optional dictionary support |
| `GeneratePublishAssetPack` | Creates a ZIP containing the publish manifest and all assets, keyed by `assets/{BasePath}/{RelativePath}` |
| `ResolveDictionaryCandidates` | Opens a previous asset pack, matches current assets to previous versions by route pattern, outputs dictionary candidate items |

### Asset pack format

```
staticwebassets.pack.zip
├── manifest.json # Standard StaticWebAssetsManifest
└── assets/
├── _content/MyApp/js/app.js
├── _content/MyApp/css/site.css
└── ... # All publish assets by BasePath/RelativePath
```

### Route-based dictionary matching

Dictionary candidates are matched by **non-fingerprinted route patterns**, not by file paths. This ensures stable matching across versions even when fingerprints change:

1. For each current asset, compute a `MatchPattern` from the endpoint route (e.g., `site.*.js` for `site.{fingerprint}.js`)
2. Scan the previous pack's manifest for assets whose endpoints match the same pattern
3. If a match is found and the content differs, output the previous asset as a dictionary candidate

### CDT endpoint metadata (per RFC 9842)

For assets with dictionary candidates, the negotiation task emits:

```json
{
"Route": "/app.js",
"Selectors": [
{ "Name": "Content-Encoding", "Value": "dcb", "Quality": "0.9" }
],
"EndpointProperties": [
{ "Name": "Dictionary-Hash", "Value": ":pZGm1Av0IEBKARczz7exkNYsZb8LzaMrV7J32a2fFG4=:" }
],
"ResponseHeaders": [
{ "Name": "Vary", "Value": "Accept-Encoding, Available-Dictionary" }
]
}
```

Additionally, `Use-As-Dictionary` response headers are added to all content-negotiated responses for each asset so browsers build up dictionary caches over time.

### MSBuild properties

| Property | Default | Description |
|----------|---------|-------------|
| `StaticWebAssetDictionaryCompression` | `false` | Enable CDT support and the `dcz` compression format |
| `StaticWebAssetPreviousAssetPack` | *(empty)* | Path to the previous version's asset pack ZIP |
| `StaticWebAssetPublishAssetPackPath` | `$(OutputPath)staticwebassets.pack.zip` | Output path for the current version's asset pack |

### CompressionFormat items (defined in targets)

```xml

.gz
gzip

.br
br

.zst
zstd

.dcz
dcb
true

```

## Performance results

| Scenario | vs Brotli | Notes |
|----------|-----------|-------|
| Standalone zstd | +7.9% larger | Expected — zstd trades size for CDT capability |
| CDT delta (app update) | **-21.7%** smaller | Clear win for update payloads |
| Best individual file | **-98.6%** smaller | System.Collections: 8,674 → 118 bytes |

For a typical Blazor WASM app update (adding a feature page):
- **Without CDT**: 624 KB re-download (brotli)
- **With CDT**: 489 KB re-download (dcz) — **135 KB savings**

## Testing

- Unit tests for all new tasks (`ZstdCompress`, `GeneratePublishAssetPack`, `ResolveDictionaryCandidates`)
- Unit tests for generalized `ApplyCompressionNegotiation` with CDT headers
- Unit tests for `ComputeMatchPattern` literal preservation
- Integration tests for zstd compression in build and publish pipelines
- Integration test: `Publish_WithPreviousPack_GeneratesDczForModifiedAssets` — end-to-end CDT flow
- Baseline tests updated for new `.zst`/`.dcz` format registration across all app types (Blazor WASM, standalone, etc.)

## Open items

- [ ] Embedding dictionary hash in `.dcz` file to support multiple dictionaries per asset — needs format design
- [ ] MSBuild error when `StaticWebAssetDictionaryCompression=true` but pack file is missing/invalid
- [ ] Introduce named types (`CompressionFormatInfo`, `PreviousRouteMatch`) to replace tuples in task internals

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.