Extend `workload validate` and `workload scan` to validate/scan all GitOps layers in-process

> 🤖 Drafted by Claude Code at the maintainer's request, grounded in a read-through of the `workload validate`/`scan` source and `go.mod` (`main` @ `5dfa22fa`).

## Problem & goal

`ksail workload validate` and `ksail workload scan` are the shift-left quality/security gate for every KSail-managed GitOps repo. But a GitOps repo expresses desired state at **three abstraction layers**, and only one of them is fully expanded before these commands inspect it. The result: the Helm- and operator-rendered manifests — i.e. most of what Flux actually applies — are never validated or scanned offline.

**Goal:** make `validate` and `scan` reason about all three layers **in-process**, so a single `ksail workload validate` / `ksail workload scan` gives full-fidelity coverage. Consumers should not have to bolt `flux-local` + `helm template` + `kubeconform` + a policy engine into their own CI to scan the real output — KSail already owns the Kustomize build, the Flux substitution, the schema validation, and the cluster lifecycle, so it is the natural place to consolidate this.

## The three layers, and where KSail stands today

| Layer | Expanded by | Offline-renderable? | `validate` today | `scan` today |
|---|---|---|---|---|
| **1. Kustomize overlays** | `kustomize build` (pure fn) | Yes, deterministic | ✅ built in-process (krusty) | ⚠️ see note |
| **2. Helm via `HelmRelease`** | helm-controller (`helm template`) | Yes (needs chart + values) | ❌ **not rendered** — HelmRelease validated as an opaque CR | ❌ not rendered |
| **3. Operator CRs** (CNPG `Cluster`, KEDA `ScaledObject`, Flagger `Canary`, `ExternalSecret`, cert-manager `Certificate`, …) | the operator controller, at runtime | **No offline renderer exists** | ⚠️ CR **shape** only (schema) | ⚠️ CR shape only |

Grounded in the source:

- **`validate`** (`pkg/cli/cmd/workload/validate.go`): builds Kustomize in-process via `sigs.k8s.io/kustomize/api/krusty`, applies a custom, schema-aware Flux `${VAR}` / `${VAR:-default}` substitution pass, then schema-validates with `github.com/yannh/kubeconform` against locally-cached schemas (`yannh/kubernetes-json-schema` + `datreeio/CRDs-catalog`). **HelmRelease resources flow through as CRs — the chart is never templated.** Operator CRs are schema-validated only.
- **`scan`** (`pkg/cli/cmd/workload/scan.go`): runs `github.com/kubescape/kubescape/v3` as an in-process library, offline (`ScanTypeRepo`, `Local: true`), against the configured frameworks (`--framework`, default `nsa`). It shares the same blind spot — HelmRelease/operator CRs are opaque — and (note) it should be confirmed/changed to operate on the **krusty-built + substituted** output rather than raw files, so overlay patches are actually reflected.

## Why this matters (what slips through today)

Because the `HelmRelease` CR itself is valid, none of these are caught offline:

- A chart that renders a privileged container, `runAsRoot`, a missing `securityContext`/resources, or a `:latest` image.
- A chart that renders schema-invalid Kubernetes objects (fails only at reconcile time, often with no hard CI gate).
- An overlay patch that **weakens** a base (strips a securityContext, exceeds a replica ceiling) — missed if `scan` reads raw files instead of the built output.

On a representative KSail-managed platform this is ~60 HelmReleases and ~80 operator CRs whose rendered output is currently unscanned shift-left.

## The crux: layer 3 has no offline renderer

This must be stated honestly. CNPG `Cluster`, KEDA `ScaledObject`, Flagger `Canary`, `ExternalSecret`, cert-manager `Certificate`, etc. expand into native resources via **arbitrary controller code at runtime** — there is no `helm template` equivalent. So "all layers in-process" decomposes into:

- **Layers 1 & 2 — fully solvable offline now.** Render Kustomize (already done) + Helm, then run the existing kubeconform/kubescape pipeline over the result.
- **Layer 3 — two honest options:** (a) CR **schema** validation (already done — validates shape, not children); and (b) an **opt-in ephemeral-cluster mode**, where KSail — uniquely able to — spins a throwaway cluster, installs the operators, applies the manifests, and validates/scans the operator-rendered **children** via admission + a cluster scan. (a) is static; (b) is the only way to see the real children, and KSail is the one tool already positioned to offer it as a single command.

## Building blocks already in `go.mod` (mostly wiring, not new deps)

- `sigs.k8s.io/kustomize/api v0.21.1` — krusty build (**used**).
- **`helm.sh/helm/v4 v4.2.0` — direct dep; in-process `helm template` via the Helm SDK engine is available but currently unused.**
- `github.com/yannh/kubeconform v0.7.0` — schema validation (**used**).
- `github.com/kubescape/kubescape/v3` — scanning (**used**).
- `github.com/fluxcd/{helm,kustomize,source}-controller/api` — typed `HelmRelease` / `Kustomization` / `OCIRepository` / `HelmRepository` structs to parse the CRs (no render logic).
- `sigs.k8s.io/controller-runtime v0.24.1`.

So Helm rendering needs a new code path, not a new tool: parse the `HelmRelease` (Flux typed API) → resolve the chart from its `sourceRef` → render with `helm/v4` → feed back into the existing validation/scan pipeline.

## Proposed roadmap

### Phase 1 — Render Helm in `validate` (layer 2; biggest ROI)
- New render path (mirror `pkg/client/kustomize`): parse `HelmRelease` via the helm-controller API types; resolve the chart from its `sourceRef` (start with `OCIRepository` + `HelmRepository`, the common cases; `GitRepository` charts later); render with the `helm/v4` engine using `spec.values` (+ in-repo `valuesFrom` ConfigMaps/Secrets where resolvable).
- Pipe the rendered output through the existing kubeconform pipeline.
- **Graceful degradation:** if a chart can't be resolved offline (private source, cluster-only `valuesFrom`), warn and fall back to CR-schema validation rather than hard-failing; gate strictness behind a flag.

### Phase 2 — Unify `scan` on the rendered output
- Make `scan` consume the same pipeline as `validate` (krusty build + substitution + Helm render) instead of raw files, so kubescape's NSA/MITRE/CIS/PSS checks evaluate the **actual applied** manifests, including overlay patches and chart output.
- Offer a flag to retain raw-file scanning for one release if back-compat is a concern.

### Phase 3 — Layer-3 depth (operator children), opt-in
- (a) Strengthen CR-schema coverage: allow `--schema-location` / CRD dirs for CRDs absent from the catalog (e.g. extract from installed CRDs or a vendored set).
- (b) `--ephemeral` mode: reuse the operator/component-install machinery from #4899 (`installer.Factory`, `Connector`) to stand up a throwaway Kind/KWOK cluster, install the declared operators, apply the manifests, and validate/scan the rendered children (admission policy + `kubescape` cluster scan). KWOK is attractive for cheap API-only expansion where operators don't need real workloads.

### Cross-cutting
- Per-layer attribution in output (which layer a finding came from).
- Cache resolved charts.
- Offline-by-default; any network/cluster access is opt-in.

## Relationship to existing work
- **Roadmap #4988:** this doesn't map cleanly to an existing theme — propose a new theme *"GitOps validation depth / shift-left fidelity"*, or split it: the `scan` half extends **Theme B (supply-chain security)**, the `validate` half is correctness/quality.
- **#4899 (operator-driven component-install lifecycle):** Phase 3(b) directly reuses its `installer.Factory` + `Connector` work.

## Open questions
1. Chart-source resolution offline: `OCIRepository` + `HelmRepository` first; `GitRepository` charts later? Auth handling for private registries?
2. `valuesFrom` referencing cluster-only ConfigMaps/Secrets — warn-and-skip, or require `--values-from` overrides?
3. Should `scan` switch to rendered output **by default** (behavior change) or behind a flag for one release?
4. Layer-3 ephemeral mode: Kind vs KWOK default; how much operator install to automate vs derive from the `Cluster` spec?
5. Flux `postBuild.substituteFrom` (cluster ConfigMaps/Secrets): the substitution pass approximates `substitute` statically today — how should `substituteFrom` be handled (skip/override)?

## Acceptance criteria
- [ ] `validate` renders `HelmRelease` charts in-process (`OCIRepository` + `HelmRepository`) and schema-validates the rendered output.
- [ ] HelmReleases that render invalid / `:latest` / privileged / resource-less manifests are caught by `validate`.
- [ ] `scan` operates on the rendered (krusty + Helm) output; kubescape findings reflect overlay patches and chart output.
- [ ] Clear graceful degradation + per-layer reporting when a layer can't be rendered offline.
- [ ] Optional ephemeral-cluster mode validates/scans operator-rendered children (may land as a follow-up child issue).
- [ ] Docs + tests; no regression to current `validate`/`scan` behavior.
- [ ] No new **required** external CLIs — reuse in-process `helm/v4`, kustomize, kubeconform, kubescape.

---
*Analysis grounded in `pkg/cli/cmd/workload/{validate,scan}.go` and `go.mod` at `main` @ `5dfa22fa`. Drafted with Claude Code.*

Start by reading pkg/cli/cmd/workload/validate.go and scan.go, then inspect go.mod and trace the existing krusty build, substitution, kubeconform, and kubescape paths. Break the broad roadmap into a scoped phase with focused tests. Done means the selected phase meets its listed acceptance criteria without regressing current validate or scan behavior.

Written by the indexing model from the issue text.