RFC: GitOps diff preview on PRs — validate ArgoCD + kro + ACK changes before merge

## Problem Statement

Multiple users have raised the need to **validate the full rendered output of GitOps changes before merging a PR**. Today, reviewing a PR that modifies Helm values, Kustomize overlays, kro ResourceGroupDefinitions, or ACK resource specs requires mentally parsing what ArgoCD will actually apply — which is error-prone and slows down reviews.

**User pain points:**
- "I changed a Helm value but I am not sure what manifests it affects across environments"
- "I modified a kro RGD template but cannot see what the downstream resources will look like"
- "ACK resource changes (S3, RDS, etc.) are hard to review without seeing the final spec"
- "We need an Atlantis-like experience for Kubernetes — plan before apply"

## Proposed Solution

Integrate [argocd-diff-preview](https://github.com/dag-andersen/argocd-diff-preview) (689⭐, presented at ArgoCon NA 2024 + EU 2026) into our CI/CD pipeline to automatically post rendered manifest diffs as PR comments.

### How it works

1. On PR open/update, a workflow clones base branch + PR branch
2. Spins up an ephemeral ArgoCD instance (via kind) or connects to a pre-configured one
3. ArgoCD renders all Application/ApplicationSet manifests from both branches
4. Generates a markdown diff and posts it as a PR comment

### What you see in the diff

| Layer | Visible in diff? | Notes |
|-------|-----------------|-------|
| Helm rendered manifests | ✅ Yes | Full YAML output after value substitution |
| Kustomize overlays | ✅ Yes | Merged patches visible |
| kro RGD definitions | ✅ Yes | Template changes visible |
| kro RGD instances (ResourceGroups) | ✅ Yes | Spec changes visible |
| ACK resource specs (S3Bucket, DBInstance) | ✅ Yes | Desired state changes |
| **kro-generated sub-resources** | ❌ No | Runtime controller output, not in Git |
| **ACK-provisioned AWS resources** | ❌ No | Cloud-side, not in Git |

### Extending to kro-generated resources (future)

To preview what kro will **actually produce** from a RGD change, we would need:
- The upcoming **kro CLI** (`kro render` / `kro plan`) to locally expand RGD templates into sub-resources
- A pipeline step that runs `kro render` on both branches and diffs the output
- This would give a complete "plan" including Deployments, Services, ConfigMaps that kro generates

**Proposed future flow:**
```
PR → argocd-diff-preview (ArgoCD layer) + kro render --diff (kro layer) → full preview
```

## Implementation Options

### Option A: Argo Workflows (recommended for PeEKS)

Since PeEKS uses Argo Workflows for CI/CD:

```
GitHub/GitLab webhook → Argo Events Sensor → Argo Workflow → diff → PR comment
```

- WorkflowTemplate with dind sidecar (ephemeral mode, ~90s)
- Or connect to a dedicated "preview" ArgoCD instance (~30s)
- Post comment via `gh` CLI (GitHub) or GitLab API

### Option B: GitHub Actions

Simple `.github/workflows/argocd-diff.yml` — see argocd-diff-preview docs.

## GitLab Support

argocd-diff-preview is VCS-agnostic — it produces markdown output. For GitLab:
- Replace `gh pr comment` with GitLab MR note API (`curl POST /api/v4/projects/:id/merge_requests/:mr_iid/notes`)
- The tool itself does not interact with any VCS — it just renders manifests and outputs files
- Community has confirmed it works with GitLab CI, Bitbucket, Jenkins

## Scope for appmod-blueprints

1. **Phase 1:** Add WorkflowTemplate + EventSource/Sensor for `gitops/` path changes
2. **Phase 2:** Filter by `FILE_REGEX=gitops/.*` to only render affected applications
3. **Phase 3:** Integrate kro CLI render (when available) for full sub-resource preview
4. **Phase 4:** Document the pattern in the PeEKS workshop as "GitOps Safety" module

## References

- Tool: https://github.com/dag-andersen/argocd-diff-preview
- Maintainer: Dag Andersen (Sr. Platform Engineer @ Egmont, Copenhagen)
- ArgoCon EU 2026 talk: "Argo CD: Previewing Pull Request Changes in SECONDS!"
- Blog: https://dev.to/dag-andersen/rendering-the-true-argo-cd-diff-on-your-prs-10bk
- kro CLI tracking: https://github.com/kubernetes-sigs/kro (future `kro render` command)

## Questions to resolve

- [ ] Ephemeral cluster vs dedicated preview ArgoCD? (trade-off: isolation vs speed)
- [ ] Where to store the WorkflowTemplate? (`gitops/addons/charts/argocd-diff-preview/`?)
- [ ] Do we need private Helm chart credentials in the ephemeral cluster?
- [ ] GitLab vs GitHub — which is primary target for the first implementation?
- [ ] kro CLI timeline — when will `kro render` be available?

/cc @allamand

Start by reading the argocd-diff-preview documentation and inspecting the existing gitops/ layout. Compare the proposed .github/workflows/argocd-diff.yml approach with the Argo WorkflowTemplate and EventSource/Sensor options, then resolve the listed questions about preview-cluster mode, credentials, VCS target, and kro support. Done means an agreed implementation scope and a working PR diff preview for the selected path and platform.

Written by the indexing model from the issue text.