DIS: Proxy in and out from legacy cluster

Ensure we add proxies to enable a seamless migration into the new cluster. Need to support both ways.

The proxies should support gradually shifting traffic from one backend service to another thus making it possible to migrate service to DIS without other teams needing to update their configurations at the same time when new services move.

# Cluster migration: gradual traffic shifting without coordinated rollouts

## The problem

Old cluster: every application in `default`. New cluster: one namespace per team. We want teams to move independently — no shared change window, no team having to synchronise its rollout with another team's, and no caller having to change configuration when a dependency moves.

## The approach

For each application, both clusters keep a Service at the legacy name `default/`. A Linkerd `HTTPRoute` on that Service splits traffic by weight between the local workload and a small per-app proxy pod that forwards to the other cluster over the existing firewalled load balancers.

One weight per application, held in Git, generating both clusters' routes, so **callers in both clusters always see the same split**. Moving an application is a PR that walks that number from 0 to 100. Rollback walks it back. Every move is rehearsed in test before prod, which is the main reason the weights exist at all.

## Steady state, mid-shift

`app-a` deployed in both clusters, 20 % of traffic on the new cluster, from *either* side:

```mermaid
flowchart LR
subgraph NEW["NEW cluster — mesh enforced"]
direction TB
CN(["caller
ns team-b"])
subgraph nsdefN["ns default"]
ANCN{{"svc/app-a
HTTPRoute 20/80
no endpoints"}}
PXN["app-a-to-old
proxy, TLS origination"]
end
APPN(["app-a
ns team-a"])
TN["Traefik
Gateway API"]
end

LBO[/"OLD cluster LB
NAT allowlist, self-signed"/]
LBN[/"NEW cluster LB
NAT allowlist, self-signed"/]

subgraph OLD["OLD cluster"]
direction TB
CO(["caller
ns default"])
subgraph nsdefO["ns default"]
ANCO{{"svc/app-a
HTTPRoute 20/80
keeps its selector"}}
APPO(["app-a workload"])
LOC["svc/app-a-local
no route attached"]
PXO["app-a-to-new
proxy, TLS origination"]
end
TO["Traefik
IngressRoute"]
end

CN -->|"app-a.default.svc"| ANCN
ANCN -->|"weight 20"| APPN
ANCN -->|"weight 80"| PXN
PXN --> LBO
LBO --> TO
TO --> LOC
LOC --> APPO

CO -->|"app-a.default.svc"| ANCO
ANCO -->|"weight 80"| APPO
ANCO -->|"weight 20"| PXO
PXO --> LBN
LBN --> TN
TN --> APPN
```

## Lifecycle of one application

```mermaid
stateDiagram-v2
[*] --> OldOnly: registry entry, anchors and routes at weight 0
OldOnly --> Shifting: team deploys into its new-cluster namespace
Shifting --> Shifting: 10, then 50, then 90 -- in test first
Shifting --> Moved: weight 100, old workload scaled to zero
Moved --> DirectAddressing: callers move to app.team-a.svc
DirectAddressing --> [*]: anchors, routes, proxies and ingress pruned

note right of Shifting
same weight applied in both clusters
rollback is the same number, downwards
end note

note right of Moved
old workload kept, scaled to zero, for a week
deleted once metrics are quiet
end note
```

## Rules that make it safe

**Ingress on the receiving side never targets an anchor.** Old-cluster Traefik routes to `-local`; new-cluster Traefik routes to `team-a/app-a`. Both bypass the split, so an inbound cross-cluster request cannot re-enter it. This is the loop guard — enforced in CI, not left to convention.

**Anchors keep their selector in the old cluster.** The route lists the anchor itself as one backend. Every step is additive: adding a route at weight 0 changes nothing, and the Service empties out on its own when the workload is deleted.

**Hop counting is telemetry, not enforcement.** `x-mig-hop` is *added* rather than set, so crossings accumulate. Alert when a request arrives carrying more than one. Nothing loops, but a transaction spanning several half-moved applications can cross the firewall repeatedly and compound timeout budgets — which is the argument for ordering moves along the dependency graph rather than by team availability.

**Deletion needs evidence, not elapsed time.** A shim goes only when its remote backend has seen zero traffic for longer than the longest cronjob schedule that could call it. Mesh coverage is what makes that metric trustworthy, hence mesh-only enforcement in the new cluster.

**Test is a real rehearsal.** The generator runs against both environments from the same source, so a prod move repeats a sequence already executed. This only holds if the test environment has the same firewall and LB path as prod — otherwise the rehearsal validates the routing but not the transport.

## Per application, generated from one weight

| Resource | New cluster | Old cluster |
|---|---|---|
| Anchor Service | `default/`, no endpoints | `default/`, keeps selector |
| Linkerd HTTPRoute | Service-parented, `-migration` | same |
| Proxy | `default/-to-old` | `default/-to-new` |
| Ingress | Gateway API HTTPRoute → `team-a/` | IngressRoute → `-local` |
| Weight | from the team's overlay | same value |

Per-app proxies rather than one shared egress: each has exactly one upstream, so no routing table, no target header, and per-app metrics and blast radius come free. They live in `default` in both clusters — platform-owned and temporary, and the firewall is enforced at the NAT and Traefik service level rather than per pod, so placement is unconstrained.

## Ownership of the weights

The routes themselves cannot be delegated by moving them: a Linkerd HTTPRoute only applies to all callers if it sits in the same namespace as its parent Service. A route placed in the team's namespace becomes a *consumer* route, scoped to traffic originating in that namespace only, and the Gateway API spec is explicit that ReferenceGrant does not govern parentRefs to Services. Anchors and routes therefore stay in `default`.

Ownership lives in Git instead. One repo, delivered by Flux as an OCI artifact, with Kustomize:

```
base/ # anchors, routes, proxies, ingress -- CODEOWNERS: platform
overlays//
teams//weights.yaml # the only file a team edits -- CODEOWNERS: that team
```

This gives teams insight and control without touching cluster RBAC, and keeps the per-team namespace isolation intact.

Things to get right in that repo:

- **A patch that changes nothing must fail CI.** Strategic-merge patches silently no-op when they miss their target, and a silent no-op looks exactly like Flux being slow. Render the manifests in CI, diff them, and fail if a team's PR produces no change — or if it changes anything other than weight fields.
- **Keep the team's file as data, not as a patch.** A free-form patch can reach `backendRefs` and filters. A plain `weight: 20` consumed by the base leaves no patch syntax in the team's hands.
- **One team file must feed both clusters' renders.** If the overlay structure ends up per-cluster, a team can update one and forget the other, and the split goes asymmetric with no error anywhere — the failure mode most likely to survive review, since both files look correct in isolation.
- **CODEOWNERS: last match wins.** Specific team paths first, platform-owned paths last, and verify with a test PR rather than by reading the file. Keep branch protection requiring code-owner review, since CODEOWNERS alone is advisory.
- **Rollback speed is a reconcile interval, not a commit.** Commit → CI → artifact → poll → apply is minutes. Shorten the interval on this Kustomization, and document `flux reconcile` as the emergency path with the platform on-call able to run it.
- **Scope pruning tightly.** The exit criteria depend on Flux pruning anchors, routes and proxies when an app reaches `direct`. Since they all land in `default` alongside other things, confirm the label selector covers only the generated set.
- **Surface applied state to teams.** They can see the desired weight in Git but not whether it is live. Expose the `OCIRepository` and `Kustomization` revision and ready condition, or every merge becomes a platform-team question.

## Visibility

**One dashboard, every application, visible to everyone — not scoped per team.** A team needs to know where its *dependencies* are serving from at least as much as where its own application is, so scoping the view by ownership would withhold exactly the information the migration creates a need for. The weights are already readable in the Git repo, so this exposes nothing that namespace RBAC was protecting; the isolation stays on the resources, while migration state is treated as public.

Per application, two numbers side by side:

| Column | Source |
|---|---|
| Desired weight | the rendered `HTTPRoute`, exported via kube-state-metrics custom resource state from `spec.rules[].backendRefs[].weight` |
| Observed split | Linkerd's `outbound_http_route_backend_*` metrics, labelled by parent Service and backend |
| Remaining callers of the anchor | same metrics, grouped by client workload and namespace |
| Error rate and latency, each side | same |
| Last applied revision | Flux `Kustomization` / `OCIRepository` status |

Desired against observed is the important pairing. They diverge whenever Flux has not reconciled yet, whenever a Kustomize patch silently no-ops, and whenever an unmeshed caller is bypassing the split — three failure modes that are otherwise invisible and look identical from the outside. **Alert on desired ≠ observed sustained beyond the reconcile interval**, and that single alert covers all three.

The remaining-callers column is the deletion evidence: it answers "who still resolves `.default`", which is the question that decides when a shim can be pruned.

Aggregated across applications, the same data is the migration progress tracker — how many are `old-only`, shifting, moved, or fully direct.

## Environment notes

- Both clusters meshed. Kubernetes 1.34, Linkerd edge-26.4.2 in the old cluster, newer in the new. Traefik in both — IngressRoute in the old, Gateway API in the new. We install the Gateway API CRDs ourselves, so neither Linkerd nor Traefik owns them.
- All application traffic is HTTP, so every dependency can be weighted. No gRPC streams, no raw TCP, nothing needing a hard cutover.
- In the new cluster, Linkerd and Traefik both consume `gateway.networking.k8s.io` HTTPRoutes, told apart by `parentRefs`: `kind: Service` is Linkerd doing the split, `kind: Gateway` is Traefik doing ingress. Naming convention `-migration` vs `-ingress`.
- Native sidecars are default from edge-26.5.2 onwards, so the old cluster needs the beta annotation (or `proxy.nativeSidecar=true`) to fix the cronjobs. Expect different pod shapes across the two clusters until then.
- Set `appProtocol` on anchor and proxy Service ports so Linkerd skips protocol detection.

## Addressing

App teams are instructed to use the fully qualified service name, which is valid in both clusters and can therefore be adopted ahead of any migration. Two details for that instruction:

- It is `..svc.cluster.local` — the `svc` label is required. `..cluster.local` is not served by CoreDNS and will not resolve.
- With the default `ndots: 5`, a four-dot name still gets the search list appended first, so `app-a.default.svc.cluster.local` costs three failed lookups before the real one. A trailing dot makes it absolute and resolves in one.

## Open items

- [ ] Whether Linkerd requires a `ReferenceGrant` for the anchor's cross-namespace `backendRef` into a team namespace, and if so who creates it there.
- [ ] Verify an endpoint-less anchor Service works as a Linkerd route parent.
- [ ] Test the LB path for large bodies, and idle and response timeouts.
- [ ] Confirm the test environment's firewall and LB path matches prod closely enough for the rehearsal to mean something.
- [ ] A date by which the shim layer is removed regardless.

TODO:
- Spec out the migration itself
- DIS: Gateway API (namespace)