asteroid-team / asteroid-team/asteroid

Extremely slow when evaluating

Open
#540 8 comments 0 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
Python
Stars
2.6k
Forks
450
PR merge metrics
No merged PRs in 30d

Description

Hi, I am running stage 3 of [egs/librimix/ConvTasNet/run.sh](https://github.com/asteroid-team/asteroid/blob/master/egs/librimix/ConvTasNet/run.sh#L119).
I used ```--compute_wer 1 --eval_mode max``` to evaluate WER.
However, it is running extremely slow.
```
2%|█▉ | 58/3000 [46:02<29:01:05, 35.51s/it
```
It takes more than one day to complete.
I checked with ```nvidia-smi```, and it was computing with GPU. However, I think only the separation process is running with GPU. I looked through the code [eval.py](https://github.com/asteroid-team/asteroid/blob/master/egs/librimix/ConvTasNet/eval.py#L121) and found out that numpy arrays are fed to the wer_tracker. So I think that for the asr part, it is evaluating in CPU mode. Is there any reason this can't be calculated with GPUs?

By the way, I see that [eval.py](https://github.com/asteroid-team/asteroid/blob/master/egs/librimix/ConvTasNet/eval.py) is evaluating with the ["Shinji Watanabe/librispeech_asr_train_asr_transformer_e18_raw_bpe_sp_valid.acc.best"](https://github.com/asteroid-team/asteroid/blob/master/egs/librimix/ConvTasNet/eval.py#L52) asr model. Is it possible to switch to other kinds of asr models by modifying [line 52](https://github.com/asteroid-team/asteroid/blob/master/egs/librimix/ConvTasNet/eval.py#L52)?

Thanks

Contributor guide

Open the contributing guide

Research direction

Start by reproducing stage 3 from egs/librimix/ConvTasNet/run.sh with --compute_wer 1 --eval_mode max, then inspect egs/librimix/ConvTasNet/eval.py around lines 52 and 121 and the wer_tracker path. Measure where time is spent and determine whether the requested GPU execution or ASR-model switching is supported; done requires a documented cause and an agreed, tested change.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python, pytorch
Domain
audio-video-rtc, machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.