kvcache-ai / kvcache-ai/Mooncake
[RFC]: Add Intel XPU CI support
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Changes proposed
# [CI/Build] Add Intel XPU CI support
Track Intel XPU CI support in this single issue. The CI implementation follows the XPU runtime work in https://github.com/kvcache-ai/Mooncake/issues/4031.
Implementation order: prepare the self-hosted XPU runner, add PR build CI after [#4031 PR1](https://github.com/kvcache-ai/Mooncake/issues/4031#pr1--xpu-platform--build-enablement), add hardware E2E after [#4031 PR2](https://github.com/kvcache-ai/Mooncake/issues/4031#pr2--transport-routing--topology) and [#4031 PR3](https://github.com/kvcache-ai/Mooncake/issues/4031#pr3--store-staging--python-e2e-mvp), then add nightly and release after the paths are stable.
## P1: Self-hosted XPU runner
- Prepare a non-root runner with Intel XPU, driver, oneAPI, Level Zero, and Docker.
- Add runner labels: `self-hosted`, `Linux`, `X64`, `intel`, `xpu`, and `mooncake-xpu`.
Done when the runner can execute a minimal SYCL/Level Zero program and is ready for GitHub Actions jobs.
## P2: PR build CI
- Depends on [#4031 PR1](https://github.com/kvcache-ai/Mooncake/issues/4031#pr1--xpu-platform--build-enablement): XPU platform, `USE_XPU`, and wheel build support.
- Add `.github/workflows/ci_xpu.yml` using a pinned oneAPI image.
- Build the XPU wheel for the supported Python versions.
- Install and import the generated wheel.
- Upload XPU wheel artifacts.
- Call the workflow from `ci.yml` for source changes and add it to `CI Gate`.
Done when source changes run XPU build/package checks in PR CI without needing XPU hardware or repository secrets.
## P3: XPU runtime and hardware E2E
- Depends on [#4031 PR2](https://github.com/kvcache-ai/Mooncake/issues/4031#pr2--transport-routing--topology) and [#4031 PR3](https://github.com/kvcache-ai/Mooncake/issues/4031#pr3--store-staging--python-e2e-mvp): XPU routing and host-staging Store MVP.
- Add `integration-test-xpu.yml` for device discovery and single-node transfer smoke tests.
- Add a maintainer-only `run-xpu-ci` label path in `e2e-ci.yml`.
- Use the PR head SHA and its exact XPU wheel artifact.
- Add exclusive concurrency, cleanup, and failure-log upload.
Done when a maintainer can label a PR to run XPU tests on real hardware and the result is enforced by `E2E Gate`.
## P4: Nightly and release
- Depends on stable P2/P3 and [#4031 PR4](https://github.com/kvcache-ai/Mooncake/issues/4031#pr4--gpudirect-dma-buf-rdma): direct-RDMA implementation.
- Add XPU compile/package coverage to `nightly.yml`.
- Add nightly hardware smoke tests after runner capacity is confirmed.
- Add tag-driven `release-xpu.yaml`.
- Upload verified XPU wheels to GitHub Releases and publish from the official repository only.
Done when nightly and release pipelines publish only verified `mooncake-transfer-engine-xpu` wheels.
## Main files
- `mooncake-common/common.cmake`
- `mooncake-transfer-engine/src/transport/CMakeLists.txt`
- `scripts/build_wheel.sh`
- `mooncake-wheel/pyproject.toml`
- `.github/workflows/ci.yml`
- `.github/workflows/e2e-ci.yml`
- `.github/workflows/nightly.yml`
### Before submitting a new issue...
- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)
Contributor guide
Research direction
Start by reading .github/workflows/ci.yml and the listed build files, especially scripts/build_wheel.sh and moondcake-wheel/pyproject.toml, then inspect the dependency on issue #4031. Trace how existing CI builds and gates wheels before choosing a phase to implement. Done means the selected XPU runner, build, hardware-test, nightly, or release workflow meets its stated checks and artifacts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake, cpp, docker, github-actions, python, shell
- Domain
- build-system, ci-cd, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100