kvcache-ai / kvcache-ai/Mooncake

[RFC][TENT] Align Ascend Direct Transport with classic TE (async + AutoConnect only)

Open
#3,919 1 comment 0 reactions 1 assignee Claimed by @ascend-direct-dev View on GitHub
RFC
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

## Changes proposed

TENT's Ascend Direct Transport is an early single-engine HIXL client. Classic TE (non-TENT, the current development trunk) has landed dummy-real multi-engine, fabric mem, LocalCommRes, AutoConnect failure semantics, register rollback, and timeout handling. Those PRs were not ported to TENT.

This RFC proposes rewriting the TENT backend to match **behavior**, not the classic class tree (`TransferExecutorBase` / sync executor / thread-pool dispatcher / `LocalCopyEngine`). TENT already has submit+poll workers and failover; copying those pieces would fight the runtime.

### In scope
- Async only: `TransferAsync` + `getTransferStatus` poll. No `TransferSync`.
- AutoConnect only: `Initialize` always sets `AutoConnect=1`. No `Connect()`, no `ASCEND_AUTO_CONNECT=0`.
- `LocalCommRes` defaults to `{"version":"1.3"}`; `ASCEND_LOCAL_COMM_RES` may override.
- Dummy-real: one HIXL engine per local NPU, dest-addr routing, RoCE serialized with a per-engine mutex (not a dedicated thread pool).
- Fabric mem + `GlobalResourceConfig`, gated so a P2P TE does not inherit Store's process-global fabric flag.
- Submit all-or-nothing validation, timeout units in milliseconds, fail-entire-route on AutoConnect errors, `quiesce()`, register rollback.
- Mock HIXL unit tests in-tree. RoCE e2e stays **out of tree**.

### Out of scope
- Synchronous ADXL transfer
- Local ACL memcpy path (same-engine also uses `TransferAsync`)
- Short connection
- `ASCEND_BUFFER_POOL` (incompatible with async; TENT staging covers constrained HCCS)
- Transport-level 2x retry (TENT runtime failover owns retries)

### Why not share classic TE sources
Classic TE and TENT use different `Transport` contracts, metadata, and threading. Sharing `TransferExecutorBase` would pull sync, local copy, and a dispatcher TENT does not want.

### Testing
- In-repo gtest with a fake HIXL client and mocked ACL (no NPU required; needs CANN headers / `USE_ASCEND_DIRECT`).
- Hardware RoCE e2e kept outside the repository.

### Size
Expected well over 500 LOC excluding tests, hence this RFC.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.