Azure / Azure/azureml-examples

Whisper STT: Expected a torch.device with a specified index or an integer, but got:cuda

Open
#2,518 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Jupyter Notebook
Stars
2k
Forks
1.7k
Avg merge
18h 18m
Merged PRs (30d)
2

Description

### Operating System

Linux

### Version Information

Python Version: 3.9.6
azure-ai-ml package version: 1.8.0

### Steps to reproduce

1. `sdk/python/foundation-models/system/inference/automatic-speech-recognition/asr-online-endpoint.ipynb`
2. Change the machine type from `Standard_DS5_v2` to `Standard_NC8as_T4_v3`
3. Deploy the endpoint
4. Run inference with `response = workspace_ml_client.online_endpoints.invoke( ... )`

### Expected behavior

Audio is transcribed

### Actual behavior

```
Exception has occurred: Exception
Expected a torch.device with a specified index or an integer, but got:cuda
StopIteration: 0

During handling of the above exception, another exception occurred:

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

File "/Users/danielve/azureml-examples/inference/automatic-speech-recognition/asr-online-endpoint-inference.py", line 28, in
response = workspace_ml_client.online_endpoints.invoke(
Exception: Expected a torch.device with a specified index or an integer, but got:cuda
```

### Addition information

This issue seems to arise when using GPUs. I have enough quota, so that shouldn't be the problem.

Contributor guide

Open the contributing guide

Research direction

Start with sdk/python/foundation-models/system/inference/automatic-speech-recognition/asr-online-endpoint.ipynb and inference/automatic-speech-recognition/asr-online-endpoint-inference.py, reproducing the deployment on Standard_NC8as_T4_v3. Trace the Whisper endpoint's GPU device handling and the invoke response error. Done means the endpoint deploys and transcribes audio successfully on the GPU machine type.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.