Azure / Azure/taugrid

Record the producing `tau` version on generated workloads, and stamp payload-digest consistently

Open
#113 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
15
Forks
3
Avg merge
21h 53m
Merged PRs (30d)
131

Description

### User problem

When a run fails or behaves unexpectedly days later, you cannot reconstruct what produced it. Two specific gaps, both verified against `cb7e255`:

**1. No workload records the `tau` version that rendered it.** I traced every version path in the repo:

- `app.kubernetes.io/version` — `cli/internal/runhistory/runhistory.go:622` *reads* it; **no writer exists anywhere in the repo.** In practice it carries the researcher's own git SHA from their manifest labels, which is easily mistaken for a TauGrid version.
- `tau.azure.com/controller-version` — declared at `core/workloadmeta/metadata.go:68`; the only reference is that same read. **No writer.**
- `core/version.Version` — used by `tau version`, cobra `--version`, `reposcaffold`, the portal API, and `run_connection.go:183` (a *client-side* check against the connection descriptor's `minTauVersion`). **None of it reaches the cluster.**

So a generated Job or RayJob carries rich provenance — `tau.azure.com/image`, `image-digest`, `code-sha`, `config-hash`, `tau-command`, `capture-version`, `submission-id` — but nothing that answers "which `tau` built this?"

This is not hypothetical. A recent review of a live cluster had to diagnose two separate symptoms (orphaned per-run ConfigMaps, and Jobs without `ttlSecondsAfterFinished`) by *deducing* that the submitting CLI predated a particular refactor, because no artifact on the cluster stated it. The deduction was only possible because the relevant code paths happen to have observable side effects; the conclusion still cannot be cited, only inferred. A single label would have made it a lookup.

Note this is genuinely a *CLI* version question, not a platform one: the binary that submits a run lives on the researcher's machine and is installed independently of the cluster's charts, so no chart or image version on the cluster answers it either.

**2. `tau.azure.com/payload-digest` is computed and then discarded by the managed-workflow renderer.** This one is an inconsistency between renderers rather than a missing feature:

- `cli/internal/rayjobrender/render.go:373` stamps `payload.AnnotationDigest`.
- `cli/internal/manifest` (the managed-workflow renderer used for **both** the Job and RayJob templates) **already computes** the digests — `payloadEmbeds.ScriptDigest` and `payloadEmbeds.ManifestDigest` at `render.go:1064-1073` — and uses them for in-pod verification via `TAU_PAYLOAD_DIGEST`, but never surfaces them as a workload annotation.
- `cli/internal/jobrender` embeds no payloads, so it is legitimately N/A.

The value exists at render time and is deliberately dropped at exactly the point where it would be useful. Since payload embeds are what make a workload self-contained, the digest is the only thing that identifies *which* script bundle a given workload actually ran.

### Desired outcome

A workload's own metadata is sufficient to identify what produced it:

- The producing `tau` version is recorded on every generated Job and RayJob.
- `tau.azure.com/payload-digest` is stamped consistently by every renderer that embeds a payload, not just `rayjobrender`.

### Proposed approach

1. Stamp the CLI version in `experiment.Metadata.KubernetesMetadata()` (`core/experiment/metadata.go:91`). That is the single point all three run paths already funnel through — `run_managed_workflow.go:387`, `run_job.go:321`, and `run_ray.go:173` all call `experiment.MergeMetadata` with the same capture metadata — so one `addAnnotation` there covers every workload kind uniformly, with no per-renderer changes and no asymmetry risk.

An annotation (not a label) is the right shape: version strings like `v0.3.0-12-gabc1234` and `dev` are not guaranteed to satisfy label-value constraints. Suggested key `tau.azure.com/tau-version`, alongside the existing `capture-version`.

Worth deciding explicitly: what to record for a locally built binary, where `core/version.Version` defaults to `dev`. `dev` is honest and still more useful than nothing, but a commit SHA would be better — `core/version.Commit` is already wired for `-ldflags`.

2. Stamp `payload.AnnotationDigest` in `cli/internal/manifest`'s workload annotations from the already-computed `payloadEmbeds` digests, matching `rayjobrender`. Decide whether the script and manifest digests are reported separately or combined, since `payloadEmbeds` carries two.

Compatibility: both are additive annotations. `runhistory` already tolerates absent annotations (`text(annotations[...])`), so older workloads keep working and nothing needs a migration.

### Alternatives considered

- **Rely on `app.kubernetes.io/version`.** Doesn't work — nothing writes it, and in practice it holds the researcher's SHA, so reusing it would overwrite user data and conflate two different things.
- **Rely on the chart or image version.** Answers a different question. The submitting CLI is installed on the researcher's machine (`make install-tau-cli`) and nothing reconciles it with the cluster's charts, so platform version is not a proxy for CLI version.
- **Infer from behaviour.** What the review actually had to do. It works only when a version difference happens to have observable side effects, and yields an inference rather than a fact.
- **Stamp per-renderer.** Rejected for item 1 — that is exactly how item 2 became inconsistent. `MergeMetadata` is the choke point and should be used.

### Additional context

This is the concrete cost behind the "improve environment provenance" recommendation in the East US 2 review, which had previously been recorded as a low-priority nice-to-have with no demonstrated impact. It has one now.

One correction that came out of the same investigation and is worth recording here, since it is easy to reach the opposite conclusion from cluster observation alone: **batch Jobs and RayJobs are *not* asymmetric in their `tau.azure.com/*` provenance at HEAD.** All three run paths share `experiment.MergeMetadata`, so both kinds receive the same annotation set. Live clusters running older builds may show batch Jobs carrying markedly less than RayJobs, but that is version skew, not current behaviour. The only genuine provenance asymmetry at HEAD is the `payload-digest` gap in item 2, and it is a *renderer* difference rather than a workload-kind difference.

Separately, and deliberately out of scope here: batch Jobs really are less diagnosable than RayJobs at terminal state, but for an unrelated reason — a Kubernetes Job's status carries only `BackoffLimitExceeded` while KubeRay writes the actual cause onto the RayJob object. That is upstream API behaviour, not a TauGrid annotation gap, and #110 addressed it by synthesizing a failure summary from pod state. Mentioned only so the two are not conflated.

Related: #110 (durable failure summaries, workspace TTL default), #111 (`scratchMount` CRD gap).

Contributor guide

Open the contributing guide

Research direction

Start with core/experiment/metadata.go and trace the metadata flow from run_managed_workflow.go, run_job.go, and run_ray.go. Then inspect cli/internal/manifest/render.go and cli/internal/rayjobrender/render.go, along with existing annotation tests if present. Done means generated Jobs and RayJobs expose the CLI version, and payload-embedding renderers expose the payload digest consistently without affecting jobrender.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.