kvcache-ai / kvcache-ai/Mooncake

[RFC]: Separate TONE and ROCm E2E platform lifecycles

Open
#4,176 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

## Motivation

TONE CUDA/ERDMA and self-hosted ROCm/RoCE currently share a controller and a large platform-dispatching `common.sh`. Their different device isolation, driver installation, process tracking and teardown policies make lifecycle changes difficult to review independently.

## Proposal

- Preserve the TONE entry under `scripts/tone_tests/` and introduce `scripts/rocm_tests/` with independent platform lifecycle implementations and test inventories.
- Keep reusable cases, assets, Python helpers and orchestration under `scripts/e2e/`, rather than duplicating two suites.
- Explicitly reference shared files through `E2E_DIR`; do not use symlinks. Mount platform directories at `/test_run` and shared sources read-only at `/test_run/e2e` on both nodes. Copy platform and shared directories with ordinary `rsync -a`.
- Refresh generated environment/wheels on setup and isolate each case in a subshell to prevent configuration/function leakage.
- Preserve ROCm allocation-scoped devices, pinned SSH, runtime caches, reset/postflight checks and stop-on-unhealthy-environment behavior.
- Update workflow paths, release wheel verification, ownership, labels and focused regression tests.

TONE uses the full inventory; ROCm uses the existing core-4gpu inventory. Running a suite requires a complete repository checkout, not a standalone copy of its directory.

## Alternatives and scope

Duplicating all test scripts would remove cross-platform branches but create two independently drifting test implementations. Keeping the current monolithic lifecycle retains the review and maintenance problem. This proposal splits platform policy while sharing test logic.

Related #4175 only extracts PR/nightly composite actions and explicitly excludes this E2E split. #3600 adds a parallel TENT TONE job and touches some of the same scripts; it will need coordination, but this proposal does not add TENT execution or alter its runtime. #3970 changes the release wheel build backend, not E2E lifecycle policy.

## Validation / rollout

CPU-only cleanup regression tests, six Python unit tests, shell syntax checks and PR-scoped prek hooks pass. Tests cover platform separation, explicit helper loading, no suite symlinks, remote preparation with environment round-tripping and per-case isolation. Actual two-node TONE/ROCm E2E validation and human line-by-line review remain outstanding; the implementation PR will be a draft until reviewed.

## AI assistance

Prepared with an OpenAI coding assistant in pi. Human technical review and end-to-end sign-off are pending.

## Revision after review

Removed the initial symlink implementation in favor of explicit shared paths. The PR now changes 29 files (+2,326 / -2,158), compared with 47 files (+3,277 / -3,083) in the initial version. Much of the displayed reduction comes from recognizing the test scripts as renames.

Contributor guide

Open the contributing guide

Research direction

Start by mapping scripts/tone_tests/, scripts/rocm_tests/, scripts/e2e/, and the shared common.sh dispatch, then inspect the focused regression tests and workflow paths described in the proposal. Done means platform lifecycles and inventories are separated without symlinks, shared helpers remain under scripts/e2e/, and the listed syntax, unit, cleanup, and isolation checks pass; two-node validation and review are still outstanding.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, shell
Domain
devops, infrastructure, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.