vosk can become very slow when spoken language is different from model

When e.g. using vosk-model-en-us-daanzu-20200905 on a modern CPU (single thread), performance can be up to about 15x realtime. However, when the spoken language switches to something different from the model (and possibly with background noise) I see Vosk sometimes slowing down to .1x realtime or less. Is it possible to avoid this slow-down scenario somehow? If it is useful I can collect some audio samples that exhibit this but my guess is that this is fundamental to the implementation.