microsoft / microsoft/onnxruntime

[Build] NPU Multi-Card Concurrent Compilation Causes OM File Race Condition (Read-While-Write Error)

Open
#26,778 1 comment 0 reactions 0 assignees View on GitHub
build contributions welcome ep:CANN
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

When compiling an ONNX model to OM on multiple NPUs concurrently, we observed a reproducible race conditon: the faster card finished OM generation first and starts writing the file, while slower cards detect that the OM file has appeared and immediately attempt to load it.

We also created a script where card 0 compiles first and all other cards continuously poll the directory; as soon as the OM file shows up, the other cards start inference, and the error reproduces reliably. Because the file is still being written by the fast card, the slower cards read an incomplete OM, leading to model-load failures or accuracy issues.

### Urgency

_No response_

### Target platform

Linux x86_64
CANN:8.3.RC1
torch:2.5.1+cpu
torch_npu:2.5.1
onnxruntime-cann:1.24.0 (This requires building ONNX Runtime from source)

### Build script
inswapper onnx :https://huggingface.co/Devia/G/blob/main/inswapper_128.onnx

python script:
```
import os
import time
import glob
import numpy as np
import torch
import torch_npu
import onnxruntime as ort

RESWAPPER_PATH = "./insightface/inswapper_128.onnx"
OM_DIR = "./"
POLL_INTERVAL = 0.2
MAX_WAIT = 300

def list_om():
return set(glob.glob(os.path.join(OM_DIR, "*.om")))

def main():
local_rank = int(os.environ.get("LOCAL_RANK", "0"))
torch.npu.set_device(local_rank)

sess = ort.InferenceSession(
RESWAPPER_PATH,
providers=[
(
"CANNExecutionProvider",
{
"device_id":local_rank ,
"arena_extend_strategy": "kNextPowerOfTwo",
"enable_cann_graph": True,
"precision_mode": "must_keep_orgin_dtype",
"op_select_impl_mode": "high_precision",
},
),
],
)

target = np.random.randn(1, 3, 128, 128).astype(np.float32)
source = np.random.randn(1, 512).astype(np.float32)
feed = {sess.get_inputs()[0].name: target, sess.get_inputs()[0].name: source}

init = list_om()

if local_rank == 0:
sess.run(None, feed)
else:
start = time.time()
while True:
now = list_om()
if now - init:
break
if time.time() - start > MAX_WAIT:
break
time.sleep(POLL_INTERVAL)

out = sess.run(None, feed)


if __name__ == "__main__":
main()
```

shell script:
``` torchrun --nproc_per_node=8 xxx.py```

### Error / output

Status Message: CANN error executing aclmdLoadFromFile()

### Visual Studio Version

_No response_

### GCC / Compiler Version

_No response_

Contributor guide

Open the contributing guide

Research direction

Use the supplied Python torchrun reproducer as the entry point and inspect the CANNExecutionProvider path that creates and loads .om files, especially the aclmdLoadFromFile() failure. Verify the race with concurrent NPU processes; done means readers never consume an incompletely written OM file and the reproducer completes successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.