[Track]: Mooncake Build, CI, and Release Roadmap

This RFC tracks Mooncake's build, CI, and release evolution. Contributions and feedback are welcome.

## Focus

The work is divided into four directions:

1. **C++ Build** — one explicit and modular native build graph.
2. **Python Build** — one Python source tree and one standard package backend.
3. **CI** — clear PR, nightly, and release CI responsibilities, implemented with small reusable workflows.
4. **Release** — one reviewable process from release branch to published artifacts.

The overall dependency is:

```text
C++ Build + Python Build --> Reusable CI --> Release
```

Build interfaces come first. CI should consume them instead of implementing another build path, and Release should publish the artifacts already validated by CI.

## 1. C++ Build

**Goal:** use CMake as the single native build graph for standalone components, language bindings, tests, and Python wheels.

### Direction

- **One build graph.** CMake owns feature flags, targets, tests, and install rules for standalone components, language bindings, and Python wheels.
- **Clear component boundaries.** Transfer Engine remains independently buildable; Store exposes one native SDK boundary; and PG/EP native cores do not depend on the PyTorch C++ ABI.
- **One dependency policy.** Prefer maintained, high-performance libraries with few transitive dependencies. Acquire, pin, and link them target-by-target through CMake when practical; keep only toolchains, drivers, and vendor runtimes external with explicit supported versions.

### Tracking

- [ ] #3800
- [ ] #3181
- [ ] #3553
- [ ] #3550
- [x] #3537
- [x] #3276
- [x] #3206
- [x] #2883

## 2. Python Build

**Goal:** make one tracked Python package and one PEP 517 backend the source of every editable install and wheel variant.

### Direction

- **One source tree.** Code shipped under `import mooncake` has one authoritative location; builds neither copy tracked sources nor modify package metadata.
- **One build interface.** PEP 517 is the public entry point for editable installs and every wheel profile, and its backend consumes CMake install outputs.
- **One install contract.** Package dependencies and public imports are explicit, and every wheel is validated from a clean environment outside the source checkout.

### Tracking

- [ ] #3534
- [ ] #3425

## 3. CI

**Goal:** provide three distinct gates over one shared build and artifact contract: PR CI decides whether a change may merge, Nightly CI reports the health of `main`, and Release CI qualifies one exact candidate for publication.

### Current baseline

- `ci.yml` is the main PR workflow for `main` and `release/**` and exposes one aggregate `CI Gate`. It already reuses the standard wheel builder, but still mixes native builds, wheel installation, TENT, EFA, ROCm, and SGLang integration in one large caller. Privileged hardware E2E is a separate label-authorized workflow.
- `nightly.yml` owns the wider wheel matrix, MUSA and container builds, Ascend hardware, full CTest and language-binding tests, coverage, TestPyPI snapshots, and failure notification. These responsibilities are currently coupled into one aggregate gate.
- Release automation is split by package variant. Standard and EFA wheels reuse builders, while ROCm, Ascend, and MUSA keep custom build-and-publish paths. A final tag still starts new builds, and container publication is a separate manual workflow.

### Target ownership

| Tier | Decision | Owns | Does not own |
| --- | --- | --- | --- |
| PR CI | May this change merge? | Fast deterministic checks and representative affected builds/tests | The full platform matrix or publication |
| Nightly CI | Is `main` healthy across supported configurations? | Full matrices, long tests, hardware tests, snapshots, and diagnosis | Merge approval or production credentials |
| Release CI | Are these exact candidate artifacts publishable? | Full core artifacts, validation evidence, and candidate manifest | Rebuilding after approval |

### PR CI

- **Required merge signal.** Every PR update produces one stable gate from deterministic checks plus the affected portable build and test profiles.
- **Artifact-first validation.** Source changes build the reference profile; packaging and shared-build changes build affected core wheels, then validate their installation, public interfaces, and portable integration paths.
- **Trusted hardware path.** Privileged E2E runs only after a maintainer authorizes an exact PR SHA, and consumes artifacts produced for that SHA.

### Nightly CI

- **Broad health coverage.** Run the full supported build matrix, portable and language-binding tests, long integrations, coverage, container builds, and available hardware suites through the same profiles used elsewhere.
- **Independent diagnosis.** Build, test, hardware, coverage, and snapshot results remain independent, retain focused artifacts, and classify failures without one broken job hiding the rest.
- **Non-production snapshots.** TestPyPI publication follows artifact validation, uses no production credentials, and reports separately from the health of `main`.

### Release CI

- **Build once.** Produce the complete core matrix from one immutable candidate commit using the shared build profiles.
- **Qualify exact artifacts.** Validate package identity, compatibility, public interfaces, and native linkage, then emit one immutable manifest containing the source, required profiles, artifact names, and digests.
- **Promote without rebuilding.** One aggregate gate qualifies the manifest; the release publisher verifies it, while extended variants may backfill independently from the same tag.

The proposed core set is standard CUDA, CUDA 13, and non-CUDA wheels for x86_64 and arm64. EFA, ROCm, Ascend, and MUSA are extended variants until a later RFC promotes one into the core release gate.

### CI implementation direction

- **Profiles define builds.** Each profile owns its runner, container, dependencies, CMake flags, package identity, and supported matrix once.
- **Workflows only orchestrate.** Top-level workflows select profiles and gates; reusable jobs build, validate, run hardware tests, and promote artifacts. Vendor paths keep only vendor-specific setup.
- **Trust follows artifacts.** Artifacts carry source SHA, profile, version, and digest; builders have no publication credentials; and trusted workflows never execute unapproved PR code.

### Tracking

- [ ] #3445
- [ ] #3338
- [ ] #2898
- [x] #3485
- [x] #3148
- [x] #3137
- [x] #3283
- [x] #3049
- [x] #2950
- [x] #3184

## 4. Release

**Goal:** make a release a reviewable promotion of one validated commit and its existing artifacts, not a fresh build triggered at the end.

### Proposed flow

1. Create `release/**` from the selected `main` commit; release fixes continue through normal PR CI.
2. Set the final version and run Release CI once. It builds, validates, and records the candidate artifacts in an immutable manifest.
3. Tag the exact candidate commit after validation. The publisher verifies the tag and manifest, then promotes the existing artifacts without rebuilding.
4. Extended variants may retry or backfill from the same tag. Versioned container images consume published wheels and pass entrypoint smoke tests before mutable tags are promoted.
5. A failed published release uses a new version; published files and tags are never overwritten.

### Tracking

- [ ] #3130

## Decisions requested

1. Should standard CUDA, CUDA 13, and non-CUDA x86_64/arm64 wheels form the core release gate?
2. Which maintainer role may authorize privileged PR hardware CI, and what cleanup guarantee must each self-hosted runner provide?

This is a broad RFC rather than a self-contained implementation task. Read ci.yml and nightly.yml first, then review the unchecked tracking issues listed under C++ Build, Python Build, CI, and Release. The work is complete only after the requested release-gate and privileged-CI decisions are resolved and the roadmap is split into actionable implementation issues.

Written by the indexing model from the issue text.