Migrate Omnia Image Pipelines from OpenCHAMI image-builder to image-thrillhouse

---

## Traceability Metadata

| Field | Value |
|-------|-------|
| **Specification Name** | Adopt image-thrillhouse Multi-Arch Manifest Builds and Pre-Build Validation Gate in image_build_manager |
| **Spec Number** | BR-91959 |
| **Category** | buildstream |
| **Related To** | BR-86010, BR-54280, BR-55868, BR-22025, BR-21709, BR-19442 |

---

# Adopt image-thrillhouse Multi-Arch Manifest Builds and Pre-Build Validation Gate in image_build_manager

---

## Summary

`image_build_manager` currently builds x86_64 and aarch64 images by invoking `image-thrillhouse build` independently per architecture, each against its own separately rendered layer YAML, with aarch64 delegated to a remote build host. `image-thrillhouse` also exposes a manifest DAG feature — a single YAML describing a graph of layers, including multi-arch layers — that Omnia does not currently use. Separately, `image-thrillhouse validate` (a config validation subcommand built into the tool) is never invoked in the pipeline; only Omnia's own input-file schema validation runs today. This spec covers closing both gaps.

---

## Problem Statement

Because each architecture's build is a fully independent `image-thrillhouse build` invocation, Omnia gets no benefit from `image-thrillhouse`'s native multi-arch manifest handling (single-manifest builds, dependency-aware layer tagging across architectures). And because there is no pre-build `validate` gate, a malformed rendered layer YAML (e.g. from a bad catalog entry or template regression) is only caught when the actual build fails inside the container — wasting build time and registry/S3 round-trips that a fast pre-flight check would avoid.

---

## Value Proposition

### Customer Value

Adopting the native manifest DAG removes the need to maintain two separate build invocations and rendered configs per functional group, reducing the surface area for x86_64/aarch64 drift. Adding the `validate` pre-build gate catches malformed build configs before any package download or container build starts, saving build-cycle time and giving operators a clear, fast failure instead of a mid-build one.

---

## User Personas

- **HPC Site Administrator**: Defines catalogs and build inputs consumed by the image pipeline.
- **HPC Operations Engineer**: Monitors builds and reviews validation/publish results.
- **Cluster Admin**: Consumes the resulting images for functional-group provisioning.

---

## Use Cases

### Use Case 1: Multi-Arch Manifest Build
**Actor**: HPC Site Administrator
**Context**: A functional group needs both x86_64 and aarch64 images from the same base definition.
**Action**: The administrator defines one `image-thrillhouse` manifest describing both architecture layers; the pipeline builds it in a single `image-thrillhouse` invocation instead of two independent ones.
**Result**: Per-architecture tagged images are produced from one manifest, without maintaining two separately rendered layer YAMLs per functional group.

### Use Case 2: Pre-Build Validation Gate
**Actor**: HPC Operations Engineer
**Context**: A catalog change produces a rendered build configuration (layer YAML) for a functional group.
**Action**: Before the build starts, the pipeline runs `image-thrillhouse validate` against the rendered configuration.
**Result**: An invalid configuration is rejected immediately, before any package download or container build begins, with a clear error surfaced to the operator.

---

## Scope Definition

### Goals (Execution Outcomes)

- Adopt `image-thrillhouse`'s native manifest DAG feature so x86_64 and aarch64 images for a functional group can be produced from a single manifest/build invocation where feasible (accounting for aarch64's remote-build-host constraint).
- Add `image-thrillhouse validate` as a pre-build gate for both the native build path and the BuildStream pipeline, run against every catalog-generated/rendered configuration before a build starts.

### Non-Goals (Scope Boundaries)

- Changing the catalog schema or `package_groups.yml` input format.
- Changing S3/Pulp storage locations or retention policy.
- Adding package-manager backends beyond DNF.
- Authoring OpenSCAP scan policies.
- Removing or altering the existing `image-builder` fallback path.

---

## Requirements

### Requirement: Multi-Arch Manifest Support [Req 1]

The pipeline must be able to produce x86_64 and aarch64 images from a single `image-thrillhouse` manifest instead of two independent build invocations, to the extent compatible with aarch64's remote-build-host delegation model.

#### Scenario: One Manifest, Two Architectures
- When a manifest is built with per-arch layer definitions, both x86_64 and aarch64 tagged images are produced without maintaining two independently rendered layer YAMLs per functional group.

### Requirement: Pre-Build Validation Gate [Req 2]

`image-thrillhouse validate` must run against every catalog-generated/rendered layer configuration before a build starts, for both the native build path and BuildStream.

#### Scenario: Invalid Config Rejected Before Build
- When a catalog change or template regression produces an invalid rendered layer configuration, `validate` rejects it before any package download or container build begins.

---

## Acceptance Criteria

- **AC-001**: A functional group's x86_64 and aarch64 images are producible from a single manifest/build invocation (or an explicitly documented, justified exception for the aarch64 remote-build-host case). [Req 1]
- **AC-002**: Every rendered layer configuration is passed through `image-thrillhouse validate` before the corresponding build starts, in both the native path and BuildStream. [Req 2]
- **AC-003**: An invalid rendered configuration is rejected by `validate` with zero package downloads or container builds triggered. [Req 2]

---

## Dependencies

- **BR-86010 (Buildstream Backend)**: BuildStream's `build-image` stage must also gain the `validate` pre-flight step, since it shares the same underlying playbook as the native path.
- **BR-21709 (Flexible Functional Groups)**: Manifest DAG adoption must not change the per-functional-group image mapping contract.
- **OpenCHAMI `image-thrillhouse`** (upstream, pinned `v0.0.24`): manifest DAG and `validate` subcommand behavior for the pinned version must be confirmed before implementation.

---

## Risks

| Risk | Category | Impact | Likelihood | Mitigation |
|---|---|---|---|---|
| aarch64's remote-build-host delegation (SSH to a separate build host) may not compose cleanly with a single local manifest DAG spanning both architectures | Technical | Medium | Medium | Confirm with upstream whether manifest DAG builds support cross-host layer execution; if not, scope Req 1 to "manifest per architecture, invoked from a shared source of truth" rather than one literal cross-arch manifest. |
| `image-thrillhouse validate` behavior/coverage may be limited for early-stage tool versions | Technical | Low | Medium | Verify `validate` catches the failure classes we actually care about (bad package names, malformed repo blocks, invalid publish targets) against the pinned `v0.0.24` before relying on it as the sole pre-build gate. |
| Adding a `validate` step increases per-build wall time | Operational | Low | Low | `validate` is expected to be fast (no package download); measure actual overhead during implementation and skip-cache appropriately if needed. |

---

## Assumptions / Open Questions

### Assumptions

- **A1**: The pinned `image-thrillhouse v0.0.24` supports both the manifest DAG feature and a `validate` subcommand with behavior consistent with current upstream docs.

### Open Questions

- **Q1**: Does `image-thrillhouse`'s manifest DAG support layers that must execute on different physical/remote hosts (as aarch64 currently requires), or is manifest DAG scoped to single-host multi-layer builds only?
- **Q2**: Should `validate` failures be treated as hard pipeline failures in BuildStream (blocking the stage) or soft warnings during an initial rollout period?

---

## Scope & Sequencing

Both requirements are independent and can be implemented and released separately: Req 2 (`validate` gate) has no dependency on Req 1 (manifest DAG) and is lower risk/effort, so it can be sequenced first. Req 1 depends on resolving Q1 (cross-host manifest feasibility) before implementation scope can be finalized.

Start by examining the native image pipeline and the BuildStream build-image stage, then verify the pinned image-thrillhouse v0.0.24 behavior for manifest DAGs, remote-host execution, and validate. Define the implementation scope from those findings; done means both paths validate rendered configurations before builds and produce the required architecture images, with any aarch64 exception documented.

Written by the indexing model from the issue text.