kvcache-ai / kvcache-ai/Mooncake
[Bug]: Standalone Ascend DummyClient loses device routing context in transfer workers
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Bug Report
## Environment
- Mooncake baseline: v0.3.13
- Transport: Ascend Direct
- Deployment: DummyClient + standalone mooncake_client
- Store mode: Ascend agent mode
## Topology
DummyClient
→ IPC/RPC
→ standalone mooncake_client
→ RealClient
→ Mooncake Store
→ Transfer Engine worker
## Problem
The DummyClient knows the physical NPU device ID, and the standalone
RealClient can set the ACL context on its RPC/SHM registration thread.
However, Store transfer work is subsequently executed by another worker
thread. ACL contexts are thread-local and are not inherited by that worker.
AscendDirectTransport calls aclrtGetDevice() from the transfer worker to
determine the local engine. If the worker has no current ACL context, it fails
with:
Invalid_Argument: rtGetDevMsg execution failed, the context is a null pointer
The transfer returns Status::Context, causing Store put/get operations to fail.
## Expected behavior
Device identity should be carried explicitly:
DummyClient physical_device_id
→ RPC device_id
→ Store Slice.device_id
→ TransferRequest.device_id
→ logicalDeviceForPhysicalId()
→ selected Ascend engine and context
Transfer routing should not depend on an ACL context inherited by an
asynchronous worker.
## Verification
The fix was manually tested on Mooncake v0.3.13 in an Ascend Linux environment.
- put/get passed on every visible NPU
- no aclrtGetDevice null-context failure
- no Status::Context transfer failure
### Before submitting...
- [x] Ensure you searched for relevant issues and read the [documentation]
Contributor guide
Research direction
Start by tracing AscendDirectTransport and its aclrtGetDevice() call from the transfer worker through Store, Slice.device_id, TransferRequest.device_id, and logicalDeviceForPhysicalId(). Use the DummyClient-to-RealClient RPC/SHM path to follow device identity end to end. Done means put/get works on every visible NPU without a null ACL-context failure or Status::Context transfer failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 47/100