kvcache-ai / kvcache-ai/Mooncake
[RFC]: Separate TONE and ROCm E2E platform lifecycles
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
## Motivation
TONE CUDA/ERDMA and self-hosted ROCm/RoCE currently share a controller and a large platform-dispatching `common.sh`. Their different device isolation, driver installation, process tracking and teardown policies make lifecycle changes difficult to review independently.
## Proposal
- Preserve the TONE entry under `scripts/tone_tests/` and introduce `scripts/rocm_tests/` with independent platform lifecycle implementations and test inventories.
- Keep reusable cases, assets, Python helpers and orchestration under `scripts/e2e/`, rather than duplicating two suites.
- Explicitly reference shared files through `E2E_DIR`; do not use symlinks. Mount platform directories at `/test_run` and shared sources read-only at `/test_run/e2e` on both nodes. Copy platform and shared directories with ordinary `rsync -a`.
- Refresh generated environment/wheels on setup and isolate each case in a subshell to prevent configuration/function leakage.
- Preserve ROCm allocation-scoped devices, pinned SSH, runtime caches, reset/postflight checks and stop-on-unhealthy-environment behavior.
- Update workflow paths, release wheel verification, ownership, labels and focused regression tests.
TONE uses the full inventory; ROCm uses the existing core-4gpu inventory. Running a suite requires a complete repository checkout, not a standalone copy of its directory.
## Alternatives and scope
Duplicating all test scripts would remove cross-platform branches but create two independently drifting test implementations. Keeping the current monolithic lifecycle retains the review and maintenance problem. This proposal splits platform policy while sharing test logic.
Related #4175 only extracts PR/nightly composite actions and explicitly excludes this E2E split. #3600 adds a parallel TENT TONE job and touches some of the same scripts; it will need coordination, but this proposal does not add TENT execution or alter its runtime. #3970 changes the release wheel build backend, not E2E lifecycle policy.
## Validation / rollout
CPU-only cleanup regression tests, six Python unit tests, shell syntax checks and PR-scoped prek hooks pass. Tests cover platform separation, explicit helper loading, no suite symlinks, remote preparation with environment round-tripping and per-case isolation. Actual two-node TONE/ROCm E2E validation and human line-by-line review remain outstanding; the implementation PR will be a draft until reviewed.
## AI assistance
Prepared with an OpenAI coding assistant in pi. Human technical review and end-to-end sign-off are pending.
## Revision after review
Removed the initial symlink implementation in favor of explicit shared paths. The PR now changes 29 files (+2,326 / -2,158), compared with 47 files (+3,277 / -3,083) in the initial version. Much of the displayed reduction comes from recognizing the test scripts as renames.
Contributor guide
Research direction
Start by mapping scripts/tone_tests/, scripts/rocm_tests/, scripts/e2e/, and the shared common.sh dispatch, then inspect the focused regression tests and workflow paths described in the proposal. Done means platform lifecycles and inventories are separated without symlinks, shared helpers remain under scripts/e2e/, and the listed syntax, unit, cleanup, and isolation checks pass; two-node validation and review are still outstanding.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, shell
- Domain
- devops, infrastructure, testing
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100