asteroid-team / asteroid-team/asteroid
Extremely slow when evaluating
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 450
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I am running stage 3 of [egs/librimix/ConvTasNet/run.sh](https://github.com/asteroid-team/asteroid/blob/master/egs/librimix/ConvTasNet/run.sh#L119).
I used ```--compute_wer 1 --eval_mode max``` to evaluate WER.
However, it is running extremely slow.
```
2%|█▉ | 58/3000 [46:02<29:01:05, 35.51s/it
```
It takes more than one day to complete.
I checked with ```nvidia-smi```, and it was computing with GPU. However, I think only the separation process is running with GPU. I looked through the code [eval.py](https://github.com/asteroid-team/asteroid/blob/master/egs/librimix/ConvTasNet/eval.py#L121) and found out that numpy arrays are fed to the wer_tracker. So I think that for the asr part, it is evaluating in CPU mode. Is there any reason this can't be calculated with GPUs?
By the way, I see that [eval.py](https://github.com/asteroid-team/asteroid/blob/master/egs/librimix/ConvTasNet/eval.py) is evaluating with the ["Shinji Watanabe/librispeech_asr_train_asr_transformer_e18_raw_bpe_sp_valid.acc.best"](https://github.com/asteroid-team/asteroid/blob/master/egs/librimix/ConvTasNet/eval.py#L52) asr model. Is it possible to switch to other kinds of asr models by modifying [line 52](https://github.com/asteroid-team/asteroid/blob/master/egs/librimix/ConvTasNet/eval.py#L52)?
Thanks
Contributor guide
Research direction
Start by reproducing stage 3 from egs/librimix/ConvTasNet/run.sh with --compute_wer 1 --eval_mode max, then inspect egs/librimix/ConvTasNet/eval.py around lines 52 and 121 and the wer_tracker path. Measure where time is spent and determine whether the requested GPU execution or ASR-model switching is supported; done requires a documented cause and an agreed, tested change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python, pytorch
- Domain
- audio-video-rtc, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100