kvcache-ai / kvcache-ai/Mooncake

[Bug]: Standalone Ascend DummyClient loses device routing context in transfer workers

Open
#3,972 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Bug Report

## Environment

- Mooncake baseline: v0.3.13
- Transport: Ascend Direct
- Deployment: DummyClient + standalone mooncake_client
- Store mode: Ascend agent mode

## Topology

DummyClient
→ IPC/RPC
→ standalone mooncake_client
→ RealClient
→ Mooncake Store
→ Transfer Engine worker

## Problem

The DummyClient knows the physical NPU device ID, and the standalone
RealClient can set the ACL context on its RPC/SHM registration thread.

However, Store transfer work is subsequently executed by another worker
thread. ACL contexts are thread-local and are not inherited by that worker.

AscendDirectTransport calls aclrtGetDevice() from the transfer worker to
determine the local engine. If the worker has no current ACL context, it fails
with:

Invalid_Argument: rtGetDevMsg execution failed, the context is a null pointer

The transfer returns Status::Context, causing Store put/get operations to fail.

## Expected behavior

Device identity should be carried explicitly:

DummyClient physical_device_id
→ RPC device_id
→ Store Slice.device_id
→ TransferRequest.device_id
→ logicalDeviceForPhysicalId()
→ selected Ascend engine and context

Transfer routing should not depend on an ACL context inherited by an
asynchronous worker.

## Verification

The fix was manually tested on Mooncake v0.3.13 in an Ascend Linux environment.

- put/get passed on every visible NPU
- no aclrtGetDevice null-context failure
- no Status::Context transfer failure

### Before submitting...

- [x] Ensure you searched for relevant issues and read the [documentation]

Contributor guide

Open the contributing guide

Research direction

Start by tracing AscendDirectTransport and its aclrtGetDevice() call from the transfer worker through Store, Slice.device_id, TransferRequest.device_id, and logicalDeviceForPhysicalId(). Use the DummyClient-to-RealClient RPC/SHM path to follow device identity end to end. Done means put/get works on every visible NPU without a null ACL-context failure or Status::Context transfer failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
47/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.