Cannot reproduce the result in speech translation

Describe the bug

Hi, I tried to reproduce the result of CVSS speech translation, but I got verty high training loss at around epoch 18. The loss is around 200.

I just follows the instruction step by step here (https://github.com/speechbrain/speechbrain/tree/1bd75e4b5e10e0d45bcf06f9b9b4564341530dbe/recipes/CVSS/S2ST) using the CVSS fr-en dataset.

Do you know whether my training loss is so high and what me be the potential problem?

Expected behaviour

Do you know whether my training loss is so high and what me be the potential problem?

To Reproduce

No response

Environment Details

My conda environment is here
Package Version Editable project location

absl-py 2.1.0
accelerate 0.28.0
aiohttp 3.8.5
aiosignal 1.3.1
alabaster 0.7.16
alembic 1.13.1
aniso8601 9.0.1
annotated-types 0.5.0
antlr4-python3-runtime 4.9.3
anyio 3.7.1
appdirs 1.4.4
arrow 1.2.3
asciitree 0.3.3
asteroid 0.6.0
asteroid-filterbanks 0.4.0
asttokens 2.4.0
async-timeout 4.0.3
attrdict 2.0.1
attrs 23.1.0
audioread 3.0.0
auraloss 0.4.0
Babel 2.14.0
backcall 0.2.0
backoff 2.2.1
beautifulsoup4 4.12.2
bitarray 2.6.0
black 19.10b0
blessed 1.20.0
blobfile 2.1.1
boto3 1.34.25
botocore 1.34.25
braceexpand 0.1.7
Brotli 1.1.0
cached-property 1.5.2
cachetools 5.3.2
cdifflib 1.2.6
cdpam 0.0.6
certifi 2023.7.22
cffi 1.15.1
cftime 1.6.2
charset-normalizer 3.2.0
ci-sdr 0.0.2
click 8.1.7
cmake 3.27.5
colorama 0.4.6
colorlog 6.8.0
comm 0.1.4
ConfigArgParse 1.7
contourpy 1.1.1
coverage 7.4.4
croniter 1.4.1
ctc-segmentation 1.7.4
cycler 0.11.0
Cython 3.0.2
datasets 2.16.1
DateTime 5.2
dateutils 0.6.12
debugpy 1.8.0
decorator 5.1.1
deepdiff 6.5.0
DeepFilterLib 0.5.6
DeepFilterNet 0.5.6
dill 0.3.7
diskcache 5.6.3
Distance 0.1.3
distro 1.8.0
docker-pycreds 0.4.0
docopt 0.6.2
docutils 0.20.1
editdistance 0.6.2
einops 0.6.1
espnet 202308
espnet-tts-frontend 0.0.3
exceptiongroup 1.1.3
executing 1.2.0
expecttest 0.2.1
fairscale 0.4.13
fairseq 0.12.2
fairseq2 0.2.0
fairseq2n 0.2.0
faiss-cpu 1.7.4
fast-bss-eval 0.1.3
fastapi 0.103.1
fasteners 0.19
fasttext 0.9.2
ffmpeg-python 0.2.0
ffprobe 0.5
filelock 3.12.4
fire 0.6.0
flake8 7.0.0
Flask 2.2.5
Flask-RESTful 0.3.10
fonttools 4.42.1
frozenlist 1.4.0
fsspec 2023.9.1
ftfy 6.1.3
future 0.18.3
g2p-en 2.1.0
gdown 5.0.0
gitdb 4.0.10
GitPython 3.1.36
google-auth 2.26.2
google-auth-oauthlib 1.2.0
greenlet 3.0.3
grpcio 1.60.0
h11 0.14.0
h5py 3.9.0
httpcore 1.0.2
httpx 0.25.2
huggingface-hub 0.23.3
humanfriendly 10.0
hydra-core 1.3.2
HyperPyYAML 1.2.1
idna 3.4
ijson 3.2.3
imagesize 1.4.1
importlib-metadata 4.13.0
importlib-resources 6.0.1
inflect 7.0.0
iniconfig 2.0.0
inquirer 3.1.3
inquirerpy 0.3.4
ipykernel 6.25.2
ipython 8.15.0
ipywidgets 8.1.1
isort 5.13.2
itsdangerous 2.1.2
jaconv 0.3.4
jamo 0.4.1
jams 0.3.4
jedi 0.19.0
jieba 0.42.1
Jinja2 3.1.2
jiwer 3.0.3
jmespath 1.0.1
joblib 1.3.2
jsonschema 4.19.0
jsonschema-specifications 2023.7.1
julius 0.2.7
jupyter_client 8.3.1
jupyter_core 5.3.1
jupyterlab-widgets 3.0.9
kaldi-python-io 1.2.2
kaldiio 2.18.0
kiwisolver 1.4.5
kornia 0.7.1
latexcodec 2.0.1
Levenshtein 0.22.0
librosa 0.9.2
lightning 2.0.9
lightning-cloud 0.5.38
lightning-utilities 0.9.0
lit 16.0.6
llama3 0.0.1
llvmlite 0.40.1
loguru 0.7.2
lxml 4.9.3
Mako 1.3.0
Markdown 3.5.2
markdown-it-py 3.0.0
markdown2 2.4.12
MarkupSafe 2.1.3
marshmallow 3.20.2
matplotlib 3.8.0
matplotlib-inline 0.1.6
mccabe 0.7.0
mdurl 0.1.2
megatron-core 0.3.0
mir-eval 0.7
more-itertools 10.1.0
mpmath 1.3.0
multidict 6.0.4
multiprocess 0.70.15
mutagen 1.47.0
nemo-text-processing 0.1.8rc0
nemo-toolkit 1.21.0
nest-asyncio 1.5.8
netCDF4 1.6.4
networkx 3.1
nltk 3.8.1
noisereduce 3.0.0
numba 0.57.1
numcodecs 0.12.1
numpy 1.23.5
nvidia-cublas-cu11 11.10.3.66
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu11 11.7.101
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu11 11.7.99
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu11 8.5.0.96
nvidia-cudnn-cu12 8.9.2.26
nvidia-cufft-cu11 10.9.0.58
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu11 10.2.10.91
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu11 11.4.0.1
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu11 11.7.4.91
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu11 2.14.3
nvidia-nccl-cu12 2.18.1
nvidia-nvjitlink-cu12 12.2.140
nvidia-nvtx-cu11 11.7.91
nvidia-nvtx-cu12 12.1.105
oauthlib 3.2.2
omegaconf 2.3.0
onnx 1.15.0
openai 1.5.0
openai-whisper 20231117
OpenCC 1.1.6
opt-einsum 3.3.0
optuna 3.5.0
ordered-set 4.1.0
overrides 7.7.0
packaging 23.1
pandas 2.1.0
pangu 4.0.6.1
parameterized 0.9.0
parso 0.8.3
pathspec 0.12.1
pathtools 0.1.2
pb-bss-eval 0.0.2
peft 0.11.1
pesq 0.0.4
pexpect 4.8.0
pfzy 0.3.4
pickleshare 0.7.5
Pillow 10.0.1
pip 23.2.1
plac 1.4.2
platformdirs 3.10.0
pluggy 1.3.0
pooch 1.7.0
portalocker 2.8.2
primePy 1.3
progress 1.6
prompt-toolkit 3.0.39
protobuf 4.23.4
psutil 5.9.5
ptflops 0.7.2.2
ptyprocess 0.7.0
pure-eval 0.2.2
pyannote.audio 3.1.1
pyannote.core 5.0.0
pyannote.database 5.0.1
pyannote.metrics 3.2.1
pyannote.pipeline 3.0.1
pyarrow 15.0.0
pyarrow-hotfix 0.6
pyasn1 0.5.1
pyasn1-modules 0.3.0
pybind11 2.11.1
pybtex 0.24.0
pybtex-docutils 1.0.3
pycodestyle 2.11.1
pycparser 2.21
pycryptodomex 3.20.0
pydantic 1.10.14
pydantic_core 2.4.0
pyDeprecate 0.3.2
pydub 0.25.1
pyee 11.1.0
pyflakes 3.2.0
Pygments 2.16.1
PyJWT 2.8.0
pyloudnorm 0.1.1
pynini 2.1.5
pyparsing 3.1.1
pypinyin 0.44.0
pypinyin-dict 0.7.0
pyroomacoustics 0.7.3
PySocks 1.7.1
pystoi 0.3.3
pytest 7.4.4
pytest-cov 4.1.0
pytest-flake8 1.1.1
pytest-runner 6.0.1
python-dateutil 2.8.2
python-editor 1.0.4
python-multipart 0.0.6
python-sofa 0.2.0
pytorch-lightning 2.0.7
pytorch-metric-learning 2.4.1
pytorch-ranger 0.1.1
pytorch-wpe 0.0.1
pytz 2023.3.post1
pyworld 0.3.4
PyYAML 6.0.1
pyzmq 25.1.1
rapidfuzz 3.6.2
readchar 4.0.5
referencing 0.30.2
regex 2023.8.8
replicate 0.22.0
requests 2.31.0
requests-oauthlib 1.3.1
resampy 0.4.2
Resemblyzer 0.1.4
rich 13.5.3
rouge-score 0.1.2
rpds-py 0.10.3
rsa 4.9
ruamel.yaml 0.17.28
ruamel.yaml.clib 0.2.7
s3transfer 0.10.0
sacrebleu 2.3.1
sacremoses 0.1.1
safetensors 0.4.2
scaper 1.6.5
scikit-learn 1.3.0
scipy 1.11.2
seaborn 0.13.0
semver 3.0.2
sentence-transformers 2.2.2
sentencepiece 0.1.97
sentry-sdk 1.31.0
setproctitle 1.3.2
setuptools 65.5.1
shellingham 1.5.4
simuleval 1.1.4
six 1.16.0
smmap 5.0.1
sniffio 1.3.0
snowballstemmer 2.2.0
sonar-space 0.2.1
sortedcontainers 2.4.0
soundfile 0.12.1
soupsieve 2.5
sox 1.4.0
soxbindings 1.2.3
soxr 0.3.7
speechbrain 1.0.0
Sphinx 7.2.6
sphinxcontrib-applehelp 1.0.8
sphinxcontrib-bibtex 2.6.2
sphinxcontrib-devhelp 1.0.6
sphinxcontrib-htmlhelp 2.0.5
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-qthelp 1.0.7
sphinxcontrib-serializinghtml 1.1.10
SQLAlchemy 2.0.25
stable-ts 2.13.7
stack-data 0.6.2
starlette 0.27.0
starsessions 1.3.0
sympy 1.12
tabulate 0.9.0
tbb 2021.11.0
tensorboard 2.15.1
tensorboard-data-server 0.7.2
tensorboardX 2.6.2.2
tensorstore 0.1.45
termcolor 2.4.0
text-unidecode 1.3
textdistance 4.6.1
texterrors 0.4.4
TextGrid 1.6.1
thop 0.1.1.post2209072238
threadpoolctl 3.2.0
tiktoken 0.4.0
tokenizers 0.15.2
toml 0.10.2
tomli 2.0.1
torch 2.1.1
torch-audiomentations 0.11.0
torch-complex 0.4.3
torch-optimizer 0.1.0
torch-pitch-shift 1.2.4
torch-stoi 0.1.2
torchaudio 2.1.1
torcheval 0.0.7
torchmetrics 1.3.0.post0
torchvision 0.16.0
tornado 6.3.3
tqdm 4.64.1
traitlets 5.10.0
transformers 4.38.2
triton 2.1.0
typed-ast 1.5.5
typeguard 2.13.3
typer 0.9.0
typing 3.7.4.3
typing_extensions 4.9.0
tzdata 2023.3
Unidecode 1.3.6
urllib3 1.26.18
uvicorn 0.23.2
wandb 0.15.10
wcwidth 0.2.13
webdataset 0.1.62
webrtcvad 2.0.10
websocket-client 1.6.3
websockets 12.0
Werkzeug 3.0.1
wget 3.2
wheel 0.38.4
widgetsnbextension 4.0.9
wrapt 1.16.0
xxhash 3.4.1
yarl 1.9.2
youtokentome 1.0.6
youtube-dl 2021.12.17
yt-dlp 2023.12.30
zarr 2.16.1
zipp 3.17.0
zope.interface 6.0

Relevant Log Output

No response

Additional Context

No response

Start with recipes/CVSS/S2ST at the referenced commit and rerun the CVSS fr-en instructions using the reported environment, checking the loss near epoch 18. Compare the observed behavior with the recipe's expected training behavior; done means documenting a reproducible cause or identifying the setup difference responsible for the high loss.