microsoft / microsoft/onnxruntime

[Training] RuntimeError: ORTModule's extensions were not detected

Open
#16,066 4 comments 0 reactions 1 assignee Claimed by @mindest View on GitHub
ep:CUDA training
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

I got an error while doing the below code :

```py
trainer = ORTTrainer(
model = model,
args = args,
train_dataset = tokenized_dataset["train"],
eval_dataset = tokenized_dataset["validation"],
data_collator = data_collator,
compute_metrics = compute_metrics,
tokenizer = tokenizer,
feature = 'token-classification',
)
trainer.train()
```

ORTModuleInitException: ORTModule's extensions were not detected at '/local_disk0/.ephemeral_nfs/envs/pythonEnv.../lib/python3.9/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions' folder. Run `python -m torch_ort.configure` before using `ORTModule` frontend.

I have 0 idea where to look at and start debugging it.

### To reproduce

Please run the above code and I am doing it in the below libraries

```
%pip install torch==2.0.1+cu118 -f https://download.pytorch.org/whl/cu118/torch_stable.html
%pip install onnx ninja
%pip install onnxruntime-training==1.14.0 -f https://download.onnxruntime.ai/onnxruntime_stable_cu114.html

%pip install --upgrade protobuf==3.20.2
# %pip install optimum[onnxruntime-gpu]==1.7.0
%pip install evaluate
%pip install seqeval
%pip install datasets
%pip install torch-ort
%pip install --upgrade accelerate
%pip install optimum
```

### Urgency

I have to finish it by end of this week at least and need to figure this out fast.

### ONNX Runtime Installation

Released Package

### ONNX Runtime Version or Commit ID

1.7.0

### PyTorch Version

3.9

### Execution Provider

CUDA

### Execution Provider Library Version

cuda 11.3

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.