Spike: in-cluster registry for skills — controller reconciles inline (and git) to an image

## Why now

Parked during the PR #157 review as a non-blocking "future spike" note ([review](https://github.com/konveyor/agentic-controller/pull/157#pullrequestreview-4978574257)). ADR 0015 deliberately moved *off* an OCI-for-everything model: git and inline skills are staged as ConfigMaps/ImageVolumes and assembled at pod init by a `skill-loader` init container. That works, but it left three source types (`image`, `git`, `inline`) with three delivery mechanisms and a split between where content lives and where it is validated.

Today the controller validates skill *content* in exactly one place — inline SkillCards. Everything else is validated at pod init by the loader, or not at all. The idea worth spiking: have the controller **resolve and validate a skill once, package it as an ordinary OCI image, push it to a lightweight in-cluster registry, and let every consumer mount the same digest via an ImageVolume**. Inline is the obvious first case (`inline markdown → SKILL.md → scratch image → registry → ImageVolume`), with git as the natural follow-on.

The open question underneath: are ImageVolumes genuinely the superior delivery mechanism (resolve/validate once, immutable digest, uniform mount for every consumer), or are we just enamored with them? This spike is where we decide.

## The seam

- **Today:** `image` → ImageVolume; `git`/`inline` → ConfigMap staged under `/opt/skills-src`, assembled by the loader init container into `/opt/skills`. Inline is the only source the controller validates content for.
- **Spiked path:** controller reconciles `inline` (then `git`) → builds a `FROM scratch` OCI image (SKILL.md + references) → pushes to an in-cluster registry → records the digest on status → consumers mount that digest via ImageVolume. Validation happens once, at reconcile, before push.

The registry endpoint should be a controller flag, not platform detection (the operator owns platform knowledge):

- `--skill-registry` (working name): if set, controller pushes built skill images there. On OpenShift the operator points this at the internal registry.
- If unset, the controller runs an **embedded registry** (e.g. `go-containerregistry/pkg/registry`) on a second port and pushes to itself — the vanilla-k8s fallback.

(These two framings — inline→image→registry, and the `--skill-registry` flag + embedded fallback — came out of the Aug 13 ADR 0015 grilling and the Aug 18–20 PR #157 review.)

## Spike tasks (timeboxed — answer questions, don't productize)

- [ ] Evaluate lightweight in-cluster OCI registry options for ImageVolume-pullable skill storage: embedded `go-containerregistry/pkg/registry`, `project-zot/zot`, `distribution/distribution`, and the OpenShift internal registry. Compare footprint, TLS/insecure-pull requirements for the kubelet, and deploy shape (Deployment+Service vs. embedded-in-controller).
- [ ] Prototype: controller reconciles one **inline** SkillCard → builds a `FROM scratch` image → pushes to the chosen registry → an AgentRun mounts it by digest via ImageVolume. Throwaway branch.
- [ ] Confirm the kubelet can pull from the in-cluster registry for an ImageVolume mount (insecure vs. TLS, and what the node needs).
- [ ] Decide where the `--skill-registry` flag lives and how the embedded-fallback is wired; confirm the controller does no platform detection.
- [ ] Assess git as the second case: does the same resolve-validate-push path generalize, or does per-commit immutability change the tradeoff?
- [ ] Record the ImageVolume prerequisite reality (feature gate is Beta/disabled-by-default in k8s 1.33/1.34, runtime support required) — this bounds where the spiked path can even run. See the CodeRabbit thread on [PR #157](https://github.com/konveyor/agentic-controller/pull/157#discussion_r3808051390).

## Deliverables

- A findings write-up: registry options compared, what worked, what broke, kubelet pull requirements.
- A recommendation: (a) is ImageVolume-by-digest genuinely superior to the current staging + loader design, or just familiar; (b) do we unify all three source types onto controller-side resolve→push, or keep the ADR 0015 split.
- If "yes, unify": a **new ADR** superseding the relevant parts of ADR 0015, followed by implementation issues.

## Explicitly out of scope

- Productizing / making this the default delivery path.
- Private registry credentials (`imagePullSecrets`) — noted on the roadmap, not part of the spike.
- OpenShift-target validation beyond confirming the operator can supply `--skill-registry`.

## Risks / notes

- Adds a stateful-ish component (a registry) to the controller's deployment surface, or an embedded server in-process — either has operational cost.
- Reintroduces "controller builds and pushes images during reconcile," which ADR 0015 deliberately walked away from; the write-up must justify the reversal against ADR 0015's context.
- ImageVolume is Beta/disabled-by-default through k8s 1.34; the spiked path is unreachable on clusters without the gate + runtime support.

## References

- PR #157 approval review, "Future spike (non-blocking): in-cluster registry for skills": https://github.com/konveyor/agentic-controller/pull/157#pullrequestreview-4978574257
- #155 (disconnected mirroring for skill images) · #154 (private git/authenticated images)
- ADR 0015 (skill packaging and delivery — the design this would revisit)
- ImageVolume prerequisites thread: https://github.com/konveyor/agentic-controller/pull/157#discussion_r3808051390

Start by reading ADR 0015 and the referenced PR #157 review, then compare the listed registry options and ImageVolume prerequisites. Prototype the inline SkillCard resolve, validation, image push, and digest mount path, while checking kubelet pull requirements. Done means a findings write-up and recommendation, with a new ADR only if unification is recommended.

Written by the indexing model from the issue text.