Improve static file and SPA hosting in the YARP container image

# Improve static file and SPA hosting in the YARP container image

## Context

The YARP container image (`src/Application`) can serve as a static file host and reverse proxy, but it's missing features that every major static hosting platform provides out of the box. Today, a team building an Astro, React, or Angular site behind the YARP container has to accept bare 404s, no compression, no cache control, and no redirect support — all things that nginx, Caddy, Netlify, Vercel, Azure Static Web Apps, and Firebase Hosting handle automatically.

## Scenario: Astro docs site behind YARP (like aspire.dev)

[aspire.dev](https://github.com/microsoft/aspire.dev) is a production Astro Starlight site served by a custom ASP.NET Core static host. Its build produces:

- `index.html` per page (static site generation)
- `_astro/` directory with hashed assets (`_astro/page.C3x9k2.css`, `_astro/hoisted.B7dK1p.js`)
- `404.html` custom error page
- Static assets (favicon, images, fonts)

Using this as a baseline, here's what works and what doesn't when serving the same output through the YARP container:

| Request | nginx | Caddy | YARP today |
|---|---|---|---|
| `GET /` → `index.html` | ✅ | ✅ | ✅ |
| `GET /docs/getting-started/` → directory default document | ✅ | ✅ | ✅ |
| `GET /nonexistent/` → `404.html` with status 404 | ✅ `error_page 404` | ✅ `handle_errors` | ❌ bare empty 404 |
| `GET /_astro/page.C3x9k2.css` → `Cache-Control: immutable` | ✅ location-based | ✅ `header` directive | ❌ no cache headers |
| `GET /index.html` → `Cache-Control: no-cache` | ✅ | ✅ | ❌ no cache headers |
| Pre-compressed `.br`/`.gz` sidecar files served when client accepts | ✅ `gzip_static`/`brotli_static` | ✅ `precompressed br gzip` | ❌ |
| On-the-fly response compression | ✅ `gzip on` | ✅ `encode gzip zstd` | ❌ |
| HSTS + HTTPS redirect in production | ✅ configurable | ✅ automatic | ❌ |
| Declarative redirects (`/old` → `/new`) | ✅ `rewrite`/`return 301` | ✅ `redir` | ❌ |
| SPA fallback with path exclusions | ✅ via location blocks | ✅ via matchers | ⚠️ fallback exists, no exclusions |
| Custom response headers per path | ✅ `add_header` | ✅ `header` directive | ❌ |
| Trailing slash normalization | ✅ `rewrite` | ✅ automatic | ❌ |
| Clean URLs (`/about` → `about.html`) | ✅ `try_files $uri.html` | ✅ `try_files {path}.html` | ❌ |

### What this looks like in nginx and Caddy

Here's a complete production config for the same Astro site on each platform — covering all the features above:

**nginx**
```nginx
server {
listen 443 ssl http2;
server_name docs.example.com;
root /var/www/site;

# HTTPS / HSTS
ssl_certificate /etc/ssl/cert.pem;
ssl_certificate_key /etc/ssl/key.pem;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

# Compression: on-the-fly + pre-compressed sidecars
gzip on;
gzip_types text/plain text/css application/javascript application/json image/svg+xml;
gzip_static on;
brotli_static on;

# Security headers (all responses)
add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "DENY" always;

# Custom 404 page
error_page 404 /404.html;
location = /404.html {
internal;
add_header Cache-Control "no-cache" always;
}

# Redirects
location = /old-page { return 301 /new-page; }
location = /get-started { return 302 /docs/quickstart/; }
location = /install.sh { return 302 https://aka.ms/install.sh; }

# Hashed assets: cache forever
location /_astro/ {
add_header Cache-Control "public, max-age=31536000, immutable" always;
try_files $uri =404;
}

# HTML files: never cache (so deployments take effect)
location ~* \.html$ {
add_header Cache-Control "no-cache" always;
}

# API routes: proxy to backend (no SPA fallback)
location /api/ {
proxy_pass http://backend:5000;
}

# Clean URLs + SPA fallback (excluding /api/ and file extensions)
location / {
try_files $uri $uri.html $uri/ /index.html;
}

# Trailing slash: redirect /about/ → /about (if not a directory)
rewrite ^(.+)/$ $1 permanent;
}

# HTTP → HTTPS redirect
server {
listen 80;
server_name docs.example.com;
return 301 https://$host$request_uri;
}
```

**Caddy**
```caddyfile
docs.example.com {
root * /var/www/site

# Compression: on-the-fly + pre-compressed sidecars
encode gzip zstd br
file_server {
precompressed br gzip
}

# HSTS (Caddy handles HTTPS + certs automatically)
header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"

# Security headers (all responses)
header X-Content-Type-Options "nosniff"
header X-Frame-Options "DENY"

# Hashed assets: cache forever
header /_astro/* Cache-Control "public, max-age=31536000, immutable"

# HTML files: never cache
header /*.html Cache-Control "no-cache"

# Redirects
redir /old-page /new-page 301
redir /get-started /docs/quickstart/ 302
redir /install.sh https://aka.ms/install.sh 302

# Trailing slash: strip (redirect /about/ → /about)
@trailing path_regexp trailing ^(.+)/$
redir @trailing {re.trailing.1} 301

# API routes: proxy to backend (before SPA fallback)
handle /api/* {
reverse_proxy backend:5000
}

# Clean URLs + SPA fallback (excluding /api/ and file extensions)
handle {
try_files {path} {path}.html {path}/ /index.html
file_server
}

# Custom 404 page
handle_errors {
@404 expression {http.error.status_code} == 404
rewrite @404 /404.html
header Cache-Control "no-cache"
file_server
}
}
```

Both configs represent the same production-ready static site with all 9 features. The YARP container should be able to replicate this behavior through configuration alone.

## Features

### 1. Custom 404 page

Serve `wwwroot/404.html` (if it exists) when no route matches, keeping the 404 status code. Today the YARP container returns a bare empty 404 body.

**What others do:**
- nginx: `error_page 404 /404.html;`
- Caddy: `handle_errors { rewrite * /404.html; file_server }`
- Aspire StaticHost: `UseStatusCodePages` + `response.SendFileAsync("404.html")` with `no-cache` headers
- Netlify: automatically serves `404.html` from publish directory

### 2. Cache control for hashed vs unhashed assets

Every SPA/static site build tool (Vite, webpack, Astro, Angular CLI) produces content-hashed filenames like `app.a1b2c3.js`. These should be cached aggressively. `index.html` and other unhashed files should not be cached so deployments take effect immediately.

**Expected behavior:**
- Files with hash-like patterns in the filename (or under `_astro/`, `assets/`, etc.): `Cache-Control: public, max-age=31536000, immutable`
- `index.html` and other HTML files: `Cache-Control: no-cache`

**What others do:**
- nginx: separate `location` blocks with different `Cache-Control` headers
- Caddy: `header` directive with path matchers
- Aspire StaticHost: MSBuild targets set `immutable` on `_astro/` files at publish time

This could be convention-based (detect hashed filenames) or configurable via patterns.

### 3. Response compression

Enable on-the-fly gzip/brotli compression for text-based responses (HTML, CSS, JS, JSON, SVG). SPA bundles are large and highly compressible — serving them uncompressed is a significant performance penalty.

**What others do:**
- nginx: `gzip on; gzip_types text/plain text/css application/javascript ...`
- Caddy: `encode gzip zstd br` (one line)

Since the YARP container is a pre-built image where users mount their own wwwroot at runtime, `MapStaticAssets()` (which requires build-time manifests) is not an option. `UseResponseCompression()` is the right approach here.

### 4. Pre-compressed sidecar file support

If the wwwroot contains pre-compressed sidecar files (e.g., `app.js.br`, `app.js.gz` alongside `app.js`), serve the compressed version when the client's `Accept-Encoding` supports it. This avoids the CPU cost of on-the-fly compression.

**What others do:**
- nginx: `gzip_static on; brotli_static on;` — native, zero-config
- Caddy: `file_server { precompressed br gzip }` — native, one line

Many static site generators and build tools can produce these sidecar files at build time. This is the highest-performance path for static asset delivery.

### 5. HTTPS redirection and HSTS

Enable `UseHsts()` and `UseHttpsRedirection()` in production (non-Development) environments as a security baseline. Opt-out via configuration.

This must be **forwarded-header aware** — the YARP container will often sit behind a CDN, load balancer, or Kubernetes ingress that terminates TLS. Without checking `X-Forwarded-Proto`, the container would redirect in a loop. `UseForwardedHeaders()` should be enabled when HTTPS redirect is on.

**What others do:**
- nginx: `add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload";`
- Caddy: automatic with HTTPS (enabled by default)
- Aspire StaticHost: `UseHsts()` + `UseHttpsRedirection()` with 180-day max-age

### 6. Declarative redirects

Support configurable redirect rules (301/302) via JSON configuration for common scenarios like renamed pages, vanity URLs, and external link shortcuts — without requiring code changes.

**Example config:**
```json
{
"Redirects": {
"/old-page": { "Destination": "/new-page", "StatusCode": 301 },
"/get-started": { "Destination": "/docs/quickstart/", "StatusCode": 302 }
}
}
```

**What others do:**
- nginx: `rewrite ^/old$ /new permanent;` or `return 301 /new;`
- Caddy: `redir /old /new 301`
- Netlify: `_redirects` file with pattern matching
- Vercel: `vercel.json` redirects array

> **Note:** Declarative redirects cover the common case. Use cases that require custom per-request logic (e.g., telemetry tracking on redirects, as [aspire.dev does for install scripts](https://github.com/microsoft/aspire.dev/blob/main/src/statichost/StaticHost/Program.cs)) would still need a custom YARP route or middleware.

### 7. Custom response headers per path

Allow setting custom HTTP headers on responses based on path patterns, configurable via JSON. This covers CORS, security headers, custom caching, and anything else a site needs to set per-route.

**Example config:**
```json
{
"Headers": {
"/*": {
"X-Content-Type-Options": "nosniff",
"X-Frame-Options": "DENY"
},
"/api/*": {
"Access-Control-Allow-Origin": "*"
},
"/_astro/*": {
"Cache-Control": "public, max-age=31536000, immutable"
}
}
}
```

**What others do:**
- nginx: `add_header` per `location` block
- Caddy: `header` directive with path matchers
- Netlify: `_headers` file with path patterns
- Vercel: `vercel.json` `headers` array

This subsumes the cache control feature (#2) — cache headers become one use case of the general custom headers capability.

### 8. Trailing slash normalization

Consistently handle trailing slashes by redirecting to the canonical form. Static site generators like Astro have a `trailingSlash` config (`always`, `never`, `ignore`) and expect the host to enforce it.

**Expected behavior (configurable):**
- `always`: `/about` → `301` to `/about/`
- `never`: `/about/` → `301` to `/about`

**What others do:**
- nginx: `rewrite ^(.+)/$ $1 permanent;` or vice versa
- Caddy: handles automatically based on file existence
- Netlify: `trailingSlash` option in `netlify.toml`
- Azure Static Web Apps: `trailingSlash` in `staticwebapp.config.json` (`always`, `never`, `auto`)
- Astro: `trailingSlash` in `astro.config.mjs` (build-time only, expects host to enforce)

### 9. Clean URLs

Serve `/about` from `/about.html` without requiring the `.html` extension in the URL. This is the default behavior on Netlify and GitHub Pages, and is how most static site generators expect their output to be served when not using directory-based routing.

**Expected behavior:**
- `GET /about` → serves `about.html` if it exists (without redirect)
- `GET /about.html` → optionally redirect to `/about` for canonical URLs

**What others do:**
- nginx: `try_files $uri $uri.html $uri/ =404;`
- Caddy: `try_files {path} {path}.html {path}/`
- Netlify: enabled by default (`Pretty URLs`)
- Vercel: `cleanUrls: true` in `vercel.json`
- Firebase: `cleanUrls: true` in `firebase.json`

> **Interaction with trailing slash:** These features must be ordered to avoid redirect loops. For example, if `cleanUrls` is on and `trailingSlash` is `never`, then `/about/` should redirect to `/about` (not try to serve `/about/.html`). The resolution order should be: trailing slash normalization first, then clean URL resolution.

### 10. SPA fallback with path exclusions

The current SPA fallback (`MapFallbackToFile("index.html")`) is all-or-nothing — it catches every unmatched non-file request. In a combined proxy + static host, this swallows misrouted API calls and other paths that should return 404.

Support configurable exclusion patterns so specific paths bypass the fallback:

**Example config:**
```json
{
"NavigationFallback": {
"Rewrite": "/index.html",
"Exclude": ["/api/*", "/.well-known/*", "/healthz"]
}
}
```

**What others do:**
- Azure Static Web Apps: `navigationFallback.exclude` array in `staticwebapp.config.json`
- nginx: separate `location /api/` block with `proxy_pass` (no fallback)
- Caddy: `handle /api/*` block before the fallback `handle` block
- Netlify: SPA rewrite `/* /index.html 200` is ordered after explicit routes
- Vercel: rewrites are processed in order; earlier rules win

## Request resolution order

These features form a small rule system, not isolated switches. The resolution order matters — both for correctness and to avoid surprising interactions with YARP's reverse proxy routes.

**Proposed order:**

```
1. HTTPS redirect (if enabled and request is HTTP)
2. Trailing slash normalization (redirect to canonical form)
3. Declarative redirects (exact match or pattern)
4. Static file (exact match in wwwroot)
5. Clean URL resolution (try .html extension)
6. Directory default document (try index.html in directory)
7. YARP reverse proxy routes
8. SPA fallback to index.html (if enabled and path not excluded)
9. Custom 404 page (serve 404.html if it exists)
```

**Key principle:** These features apply to the **static hosting pipeline only** — they must not accidentally mutate proxied traffic. Custom headers, cache control, compression, and SPA fallback should not apply to responses that YARP proxies to a backend. YARP's own route configuration handles those.

## Non-goals

- `MapStaticAssets()` — requires a build-time manifest, which doesn't work for a pre-packaged container where users mount their own wwwroot at runtime