[Release tracker] TauGrid v0.4.2 deployment: portal refresh, bug fixes, and open-issue closure

### User problem

We need one deployment tracker for the next TauGrid release: capture the changes made since v0.4.0, deliver the refreshed TauGrid Portal experience alongside bug fixes, and account for every originally tracked issue before release closeout.

### Desired outcome

Ship and deploy **TauGrid v0.4.2** with an agreed release commit, published artifacts, a signed-off Portal refresh, and verified fixes. Each tracked bug needs a fix or an explicitly accepted deferral; an unchecked item is not implicitly dropped.

**Version decision:** This request began as v0.4.1. Target v0.4.2 because #243 prepared v0.4.1 and #253 subsequently bumped the unified release. v0.4.0 remains the latest published GitHub release at this triage.

**Scope decision, 2026-09-09 PDT:** All three tracked enhancements (#113, #92, #83) are deferred to the [0.4.3 milestone](https://github.com/Azure/taugrid/milestone/1). They remain open and are not v0.4.2 launch blockers. Bug recommendations below are triage recommendations, not approved deferrals.

**Accepted limitation, 2026-09-10 PDT:** The release owner explicitly deferred automatic recovery from the upstream AzAPI asynchronous terminal-operation identity failure as a documented, non-blocking v0.4.2 limitation. Provisioning can fail visibly and require operator assessment; this is not a fix or a guarantee that every ADX initialization failure recovers automatically. Keep #161 open. This decision does not waive the Function waiter safety fix, actual permission-migration evidence, or combined-candidate acceptance. The original incident's HTTP-versus-LRO response shape remains unknown.

### Proposed approach

## 1. Changes already merged since v0.4.0

Checked boxes in this section mean **merged**, not published, deployed, or release-accepted. Confirm inclusion in the selected release commit.

- [x] #252 and #264 - Portal UX foundations followed by the native React refresh with integrated experiment navigation, comparisons, and evidence.
- [x] #260, #262 and #263 - Workspace-bound dry-run/serving, checkpoint paths, and explicit Deployment command/argument support.
- [x] #254 - Historical RayJob ADX log connection discovery.
- [x] #239 - Agent-readable CLI run status.
- [x] #235 - Metrics offload configuration in run files.
- [x] #234 - Workspace GPU chargeback and Flex GPU setup.
- [x] #225 - Portable cluster credential adapters.
- [x] #228 and #229 - Windows CLI binaries and validation.
- [x] #223 - CLI help improvements and dependency fix.
- [x] #222 and #246 - AKS Terraform guidance and waiter line-ending coverage.
- [x] #233 - Ray 2.56.1 / protobuf 7 compatibility.
- [x] #243 and #253 - Unified release/chart preparation, including Kueue 0.19.2 and gpu-monitoring 0.1.8 with CRD retention and single-namespace contracts.
- [x] #237 and #242 - Kubernetes and Azure Go SDK dependency updates.
- [x] #199, #214, #218, #220, #230, #245, #249, #250, #251, #255, #256, #257 and #258 - Monitoring, tooling, runtime, and CI dependency maintenance.
- [x] #224, #238 and #240 - Blog, documentation cache behavior, and community links.

## 2. Portal new look and bug-fix delivery

- [x] Implement and merge the new look: #264, merged 2026-09-09 at 20:31 PDT.
- [ ] Obtain visual and functional acceptance on the exact release candidate; merging the implementation is not deployment signoff.
- [x] Integrate #241 onto main, preserving current diagnostics and React section-recovery behavior. Merged as `cdcd1df8`; deployed-candidate acceptance remains below.
- [x] Integrate #232 onto main. Merged as `2ed3c292`; deployed-candidate acceptance remains below.
- [ ] Exercise overview, workspace/run navigation, Job/RayJob details, logs, metrics, Stellar comparison/evidence, and GPU chargeback with representative deployed data.
- [ ] Confirm missing telemetry is unavailable rather than zero/healthy; check CPU intervals, GPU/memory availability, and cost coverage.
- [ ] Confirm Ray target/workspace isolation and supported read-only routes.
- [ ] Attach before/after visuals and a data-backed walkthrough; record limitations and acceptance.

## 3. Original issue inventory and dispositions

Original snapshot: **13 issues**. Five are now closed, three enhancements explicitly deferred, and five bugs still open. The two open Portal fix PRs are tracked separately because they have no corresponding issue in this snapshot. A checked closed-issue box records GitHub closure, not live release acceptance.

### Run, serve, and workspace fixes

- [x] #212 - Client dry-run output paths: closed, #260 merged.
- [x] #210 - Serve namespace team-label mismatch: closed; #262 supplies the serve-specific workspace routing fix after #260.
- [x] #209 - Explicit shell command/argument support: closed, #263 merged. Legacy `--args` behavior remains; Deployment uses explicit `--command` / `--arg`.
- [x] #208 - Workspace checkpoint prefix: closed, #263 merged.
- [ ] #111 - Unused client-side scratchMount field absent from the TauWorkspace CRD. Deferral recommendation below.
- [ ] #76 - Durable client-request-ID submission idempotency. Conditional deferral recommendation below.

### ADX and infrastructure fixes

- [ ] #190 - ManagementCommand completion condition; mitigated installation sequencing is not schema-completion proof.
- [ ] #162 - Function reconciliation after transient ADX throttling; Terraform retry mitigation is not an upstream controller fix.
- [ ] #161 - Lifecycle recorder ADX identity propagation. #247 is merged; actual migration evidence remains open. Only automatic terminal-operation recovery is an accepted non-blocking v0.4.2 limitation.

### Enhancements deferred to 0.4.3

- [x] #113 - Producing tau version and payload digest: assigned to milestone 0.4.3; left open.
- [x] #92 - Multiple active workspaces: assigned to milestone 0.4.3; left open.
- [x] #83 - Server-backed saved collections: assigned to milestone 0.4.3; left open.

### Repository administration

- [x] #170 - Migration/opt-out action: GitHub issue closed; separate from product fixes.

## 4. Pre-launch bug triage

Initial source baseline: `d8a4b022` (main including #264), 2026-09-09 PDT. Subsequent exact-candidate qualification at `cdcd1df8` and scope decisions are recorded in this issue's updates. The accepted upstream recovery limitation is recorded above; it is not evidence of a complete #161 fix.

| Item | Recommended disposition | Evidence and launch acceptance |
| --- | --- | --- |
| Portal #241 | **Merged; deployed-candidate acceptance pending** | UID-fenced canonical Pod/Workload/Ray ownership and Event correlation, read RBAC, and unknown/unavailable handling are merged. The final correction preserves same-incarnation troubleshooting evidence when ownership sources are incomplete. Confirm those behaviors on the combined deployed candidate rather than the separate older demo build. |
| Metrics watch #232 | **Merged; deployed-candidate acceptance pending** | Non-scalar chunk survival, checkpoint persistence, later scalar export, completion, and strict malformed/non-finite input handling are merged. Confirm the integrated release's collection and completion path; an older demo is not qualification of the merged fix. |
| #161 / #247 | **Automatic terminal-operation recovery deferred; migration gate remains** | #247 merged native state migration and bounded HTTP-error retry hardening. The owner accepted the unsupported asynchronous terminal-operation retry as a documented non-blocking v0.4.2 limitation, not a complete fix. Still require a real refreshed existing-deployment plan proving the grant is preserved and candidate-specific recorder/history acceptance where enabled. Keep #161 open. |
| #162 / #265 | **Waiter safety fix required** | Fault injection on `cdcd1df8` reproduced false readiness with required Functions missing and deletion of a foreign throttled Function in both waiter engines. #265 implements required-set and ownership safeguards and is still undergoing CI/review. Direct Helm bypasses the mitigation and upstream Azure/adx-mon#1238 remains open. Healthy live Functions or a successful waiter do not establish upstream automatic reconciliation. |
| #190 | **Recommend defer the upstream completion contract** | Main no longer waits on the missing condition and uses skipvalidation, avoiding the known install wait failure. That does not prove ADX schemas are ready; early workload history can be missed. Before first qualification workload, establish schema readiness separately. Upstream Azure/adx-mon#983 remains unmerged; do not promise lossless startup capture. |
| #111 | **Recommend defer, with unsupported-field clarification** | scratchMount exists only in the CLI model/fixture; there is no production consumer and no CRD property. Hand-authored values cannot persist, but this is not an active implemented mount feature regressing. Do not advertise workspace scratch defaults; remove/clarify the unused field and misleading docs in follow-up, or implement the full schema-to-rendering contract. Explicit Job storage is available, not a universal RayJob scratch workaround. |
| #76 | **Recommend defer only with retry restrictions** | Fresh invocations generate new submission IDs; durable request-ID/digest replay is absent. Stable authored workload names prevent ordinary same-name duplicate creation, and same-process uncertain-create recovery exists. Renamed/deleted/cross-kind retries can still execute again. Require callers to preserve namespace/kind/name and inspect uncertain results; do not promise cross-invocation idempotency or allow blind automated retries. Block any release feature requiring that guarantee. Open #31 is not an idempotency fix. |

### Candidate integration cautions and evidence

- #241 was merged after exact-head approval and completed checks; no known code-integration blocker remains. Live acceptance must still confirm UID-fenced ownership and preserved diagnostics on the combined candidate.
- #232 was merged after completed exact-head checks; its old dependency-download failure is not a current blocker. Live acceptance must still cover the combined importer/watch path.
- #247 is merged with a native moved block and saved-plan guard, but the real refreshed migration plan remains unproven. Its earlier successful principal creation followed by Helm readiness timeouts is not full installation acceptance. The accepted terminal-operation recovery limitation does not waive migration safety.
- #162 mitigation lives in `terraform/aks/wait-for-adx-functions-ready.sh` and `terraform/aks/Wait-ForAdxFunctionsReady.ps1`. #265 addresses the reproduced required-set/ownership defects; do not run an unfixed waiter against unrelated resources or mismatched release values.
- #111 evidence spans the CLI workspace type, controller WorkspaceDefaults type, and generated/chart CRDs. #76 evidence is in `cli/internal/cli/run.go` and `run_submit.go`.

## 5. Release and deployment gates

- [ ] Assign release/Portal acceptance owners, bug owners, target environments, and rollout window; explicitly accept or reject remaining proposed bug deferrals. Only the upstream asynchronous recovery limitation and the three enhancements have accepted deferrals.
- [ ] Freeze the release commit and reconcile the complete v0.4.0-to-candidate change list. All tracked enhancements are excluded in favor of 0.4.3; other open PRs are not implicitly included.
- [ ] Confirm unified 0.4.2 charts, images, CLI defaults, examples, and release notes; preserve independently versioned SDK, gpu-monitoring, and adx-mon contracts.
- [ ] Complete candidate CI and issue-specific regression coverage, including Helm 3/4, CRD retention, and relevant CLI/Portal/controller/SDK contracts.
- [ ] Establish a reversible installation/upgrade and ownership transition, with safe Terraform grant migration if #247 is included; identify the actual prior deployed revision.
- [ ] Resolve or explain and bound the previously reported intermittent Portal HTTP 400; lack of fresh recurrence alone is not a root cause.
- [ ] Publish approved v0.4.2 artifacts through release workflows and record versions/digests/workflow links.
- [ ] Deploy through existing ownership mechanisms; record environment, release commit, chart version, image digests, and time.
- [ ] Verify schema/controller/monitoring readiness before representative workloads, then workspace/queue routing, Job/RayJob execution/logs, serving/checkpoints, and Portal workflows.
- [ ] Record go/no-go, roll out, and observe health/error signals for an agreed period.
- [ ] Close only after deployment evidence, Portal acceptance, and every bug's fix or accepted deferral are recorded. Refresh the issue inventory at freeze.

### Alternatives considered

- Retain v0.4.1: rejected because main already prepares v0.4.2.
- Require every enhancement before launch: rejected; all three are explicitly moved to 0.4.3.
- Close bugs merely because a workaround or PR exists: rejected; distinguish mitigation, code integration, and deployment acceptance.

### Additional context

This is release coordination, not an instruction to deploy, publish, merge, grant permissions, or close linked bugs. The triage leaves source unchanged. No live cluster or Azure operations were performed. GitHub Copilot assisted with issue triage and this tracker update.

### Confirmation

- [x] I searched existing issues and roadmap items for this request.
- [x] This request does not include confidential or security-sensitive information.