speechbrain / speechbrain/speechbrain

SepformerSeparation can not load audio.

Open
#2,891 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
11.8k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

Describe the bug

Code:

from speechbrain.inference import SepformerSeparation as separator
import torchaudio

model = separator.from_hparams(
    source="speechbrain/sepformer-wham16k-enhancement",
    # savedir="pretrained_models/sepformer-wham16k-enhancement",
)

# for custom file, change path
import os
assert os.path.exists("../trajectory/uot3d_mel_t=0.50.wav")
est_sources = model.separate_file(
    path="../trajectory/uot3d_mel_t=0.50.wav"
)

Error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[99], [line 12](vscode-notebook-cell:?execution_count=99&line=12)
     [10](vscode-notebook-cell:?execution_count=99&line=10) import os
     [11](vscode-notebook-cell:?execution_count=99&line=11) assert os.path.exists("../trajectory/uot3d_mel_t=0.50.wav")
---> [12](vscode-notebook-cell:?execution_count=99&line=12) est_sources = model.separate_file(
     [13](vscode-notebook-cell:?execution_count=99&line=13)     path="../trajectory/uot3d_mel_t=0.50.wav"
     [14](vscode-notebook-cell:?execution_count=99&line=14) )
     [16](vscode-notebook-cell:?execution_count=99&line=16) # torchaudio.save("enhanced_wham16k.wav", est_sources[:, :, 0].detach().cpu(), 16000)

File /mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/speechbrain/inference/separation.py:107, in SepformerSeparation.separate_file(self, path, savedir)
     [99](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/speechbrain/inference/separation.py:99) source, fl = split_path(path)
    [100](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/speechbrain/inference/separation.py:100) path = fetch(
    [101](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/speechbrain/inference/separation.py:101)     fl,
    [102](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/speechbrain/inference/separation.py:102)     source=source,
    [103](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/speechbrain/inference/separation.py:103)     savedir=savedir,
    [104](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/speechbrain/inference/separation.py:104)     local_strategy=LocalStrategy.SYMLINK,
    [105](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/speechbrain/inference/separation.py:105) )
--> [107](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/speechbrain/inference/separation.py:107) batch, fs_file = torchaudio.load(path)
    [108](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/speechbrain/inference/separation.py:108) batch = batch.to(self.device)
    [109](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/speechbrain/inference/separation.py:109) fs_model = self.hparams.sample_rate

File /mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torchaudio/_backend/utils.py:205, in get_load_func.<locals>.load(uri, frame_offset, num_frames, normalize, channels_first, format, buffer_size, backend)
    [128](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torchaudio/_backend/utils.py:128) """Load audio data from source.
    [129](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torchaudio/_backend/utils.py:129) 
    [130](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torchaudio/_backend/utils.py:130) By default (``normalize=True``, ``channels_first=True``), this function returns Tensor with
   (...)
    [202](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torchaudio/_backend/utils.py:202)         `[channel, time]` else `[time, channel]`.
    [203](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torchaudio/_backend/utils.py:203) """
    [204](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torchaudio/_backend/utils.py:204) backend = dispatcher(uri, format, backend)
--> [205](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torchaudio/_backend/utils.py:205) return backend.load(uri, frame_offset, num_frames, normalize, channels_first, format, buffer_size)

File /mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torchaudio/_backend/sox.py:44, in SoXBackend.load(uri, frame_offset, num_frames, normalize, channels_first, format, buffer_size)
     [39](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torchaudio/_backend/sox.py:39)     raise ValueError(
     [40](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torchaudio/_backend/sox.py:40)         "SoX backend does not support loading from file-like objects. ",
     [41](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torchaudio/_backend/sox.py:41)         "Please use an alternative backend that does support loading from file-like objects, e.g. FFmpeg.",
     [42](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torchaudio/_backend/sox.py:42)     )
     [43](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torchaudio/_backend/sox.py:43) else:
---> [44](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torchaudio/_backend/sox.py:44)     ret = sox_ext.load_audio_file(uri, frame_offset, num_frames, normalize, channels_first, format)
     [45](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torchaudio/_backend/sox.py:45)     if not ret:
     [46](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torchaudio/_backend/sox.py:46)         raise RuntimeError(f"Failed to load audio from {uri}.")

File /mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torch/_ops.py:854, in OpOverloadPacket.__call__(self_, *args, **kwargs)
    [846](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torch/_ops.py:846) def __call__(self_, *args, **kwargs):  # noqa: B902
    [847](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torch/_ops.py:847)     # use `self_` to avoid naming collide with aten ops arguments that
    [848](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torch/_ops.py:848)     # named "self". This way, all the aten ops can be called by kwargs.
   (...)
    [852](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torch/_ops.py:852)     # We save the function ptr as the `op` attribute on
    [853](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torch/_ops.py:853)     # OpOverloadPacket to access it here.
--> [854](https://vscode-remote+ssh-002dremote-002b10-002e120-002e17-002e57.vscode-resource.vscode-cdn.net/mnt/nvme0n1/xietianxin/miniconda3/envs/f5-tts/lib/python3.10/site-packages/torch/_ops.py:854)     return self_._op(*args, **(kwargs or {}))

RuntimeError: torchaudio_sox::load_audio_file() Expected a value of type 'str' for argument '_0' but instead found type 'PosixPath'.
Position: 0
Value: PosixPath('/mnt/nvme0n1/xietianxin/code/github/F5-TTS/notebooks/../trajectory/uot3d_mel_t=0.50.wav')
Declaration: torchaudio_sox::load_audio_file(str _0, int? _1, int? _2, bool? _3, bool? _4, str? _5) -> (Tensor _0, int _1)
Cast error details: Unable to cast Python instance of type <class 'pathlib.PosixPath'> to C++ type '?' (#2650 ```


### Expected behaviour

It should load and separate the audio file.

### To Reproduce

_No response_

### Environment Details

Python env:

```text
accelerate==1.3.0
aiofiles==23.2.1
aiohappyeyeballs==2.4.6
aiohttp==3.11.12
aiosignal==1.3.2
aliyun-python-sdk-core==2.16.0
aliyun-python-sdk-kms==2.16.5
annotated-types==0.7.0
antlr4-python3-runtime==4.9.3
anyio==4.8.0
asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1733250440834/work
async-timeout==5.0.1
attrs==25.1.0
audeer==2.2.1
audioread==3.0.1
audobject==0.7.11
audonnx==0.7.0
bitsandbytes==0.45.2
blessed==1.20.0
boto3==1.36.20
botocore==1.36.20
cached_path==1.6.7
cachetools==5.5.1
certifi==2025.1.31
cffi==1.17.1
charset-normalizer==3.4.1
click==8.1.8
coloredlogs==15.0.1
comm @ file:///home/conda/feedstock_root/build_artifacts/comm_1733502965406/work
contourpy==1.3.1
crcmod==1.7
cryptography==44.0.2
cycler==0.12.1
datasets==3.3.0
debugpy @ file:///croot/debugpy_1736267418885/work
decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1740384970518/work
dill==0.3.8
docker-pycreds==0.4.0
editdistance==0.8.1
einops==0.8.1
einx==0.3.0
ema-pytorch==0.7.7
encodec==0.1.1
entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1733327148154/work
exceptiongroup @ file:///home/conda/feedstock_root/build_artifacts/exceptiongroup_1733208806608/work
executing @ file:///home/conda/feedstock_root/build_artifacts/executing_1733569351617/work
-e git+https://github.com/SWivid/F5-TTS.git@4b4359bc39ec9a3dfb6ff412345030792f5c37fb#egg=f5_tts
fastapi==0.115.8
fastdtw==0.3.4
fastjsonschema==2.21.1
ffmpy==0.5.0
filelock==3.17.0
flatbuffers==25.2.10
fonttools==4.56.0
frozendict==2.4.6
frozenlist==1.5.0
fsspec==2024.12.0
funasr==1.0.27
gitdb==4.0.12
GitPython==3.1.44
google-api-core==2.24.1
google-auth==2.38.0
google-cloud-core==2.4.1
google-cloud-storage==2.19.0
google-crc32c==1.6.0
google-resumable-media==2.7.2
googleapis-common-protos==1.67.0
gpustat==1.1.1
gradio==5.14.0
gradio_client==1.7.0
h11==0.14.0
httpcore==1.0.7
httpx==0.28.1
huggingface-hub==0.27.1
humanfriendly==10.0
hydra-core==1.3.2
HyperPyYAML==1.2.2
idna==3.10
importlib_metadata==8.6.1
ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1719845459717/work
ipython @ file:///home/conda/feedstock_root/build_artifacts/bld/rattler-build_ipython_1744034854/work
ipywidgets==8.1.6
jaconv==0.4.0
jamo==0.4.1
jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1733300866624/work
jieba==0.42.1
Jinja2==3.1.5
jmespath==0.10.0
joblib==1.4.2
jsonschema==4.23.0
jsonschema-specifications==2024.10.1
jupyter-client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1654730843242/work
jupyter_core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1727163409502/work
jupyterlab_widgets==3.0.14
kaldiio==2.18.1
kiwisolver==1.4.8
lazy_loader==0.4
librosa==0.11.0
llvmlite==0.44.0
loguru==0.7.3
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib==3.10.0
matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1733416936468/work
mdurl==0.1.2
modelscope==1.25.0
more-itertools==10.6.0
mpmath==1.3.0
msgpack==1.1.0
multidict==6.1.0
multiprocess==0.70.16
narwhals==1.36.0
nbformat==5.10.4
nest_asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1733325553580/work
networkx==3.4.2
numba==0.61.2
numpy==2.2.5
nvidia-cublas-cu11==11.11.3.6
nvidia-cuda-cupti-cu11==11.8.87
nvidia-cuda-nvrtc-cu11==11.8.89
nvidia-cuda-runtime-cu11==11.8.89
nvidia-cudnn-cu11==8.7.0.84
nvidia-cufft-cu11==10.9.0.58
nvidia-curand-cu11==10.3.0.86
nvidia-cusolver-cu11==11.4.1.48
nvidia-cusparse-cu11==11.7.5.86
nvidia-ml-py==12.570.86
nvidia-nccl-cu11==2.20.5
nvidia-nvtx-cu11==11.8.86
omegaconf==2.3.0
onnx==1.17.0
onnxruntime==1.21.1
openai-whisper==20240930
opencv-python==4.11.0.86
orjson==3.10.15
oss2==2.19.1
oyaml==1.0
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1745075690131/work
pandas==2.2.3
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1733271261340/work
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1733301927746/work
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1733327343728/work
pillow==11.1.0
platformdirs @ file:///home/conda/feedstock_root/build_artifacts/bld/rattler-build_platformdirs_1742485085/work
plotly==6.0.1
pooch==1.8.2
POT==0.9.5
prompt_toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1744724089886/work
propcache==0.2.1
proto-plus==1.26.0
protobuf==5.29.4
psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1653089181607/work
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1733302279685/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl#sha256=92c32ff62b5fd8cf325bec5ab90d7be3d2a8ca8c8a3813ff487a8d2002630d1f
pure_eval @ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1733569405015/work
pyarrow==19.0.0
pyasn1==0.6.1
pyasn1_modules==0.4.1
pycparser==2.22
pycryptodome==3.22.0
pydantic==2.10.6
pydantic_core==2.27.2
pydub==0.25.1
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1736243443484/work
pynndescent==0.5.13
pyparsing==3.2.1
pypinyin==0.53.0
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1733215673016/work
python-multipart==0.0.20
pytorch-wpe==0.0.1
pytz==2025.1
PyYAML==6.0.2
pyzmq @ file:///croot/pyzmq_1734687138743/work
referencing==0.36.2
regex==2024.11.6
requests==2.32.3
rich==13.9.4
rotary-embedding-torch==0.8.6
rpds-py==0.24.0
rsa==4.9
ruamel.yaml==0.18.10
ruamel.yaml.clib==0.2.12
ruff==0.9.6
s3transfer==0.11.2
safehttpx==0.1.6
safetensors==0.5.2
scikit-learn==1.6.1
scipy==1.15.2
semantic-version==2.10.0
sentencepiece==0.2.0
sentry-sdk==2.21.0
setproctitle==1.3.4
shellingham==1.5.4
six @ file:///home/conda/feedstock_root/build_artifacts/six_1733380938961/work
smmap==5.0.2
sniffio==1.3.1
socksio==1.0.0
soundfile==0.13.1
soxr==0.5.0.post1
speechbrain==1.0.3
stack_data @ file:///home/conda/feedstock_root/build_artifacts/stack_data_1733569443808/work
starlette==0.45.3
sympy==1.13.3
tensorboardX==2.6.2.2
threadpoolctl==3.6.0
tiktoken==0.9.0
tokenizers==0.21.0
tomli==2.2.1
tomlkit==0.13.2
torch==2.3.0+cu118
torch-complex==0.4.4
torchaudio==2.3.0+cu118
torchdiffeq==0.2.5
tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1648827254365/work
tqdm==4.67.1
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1733367359838/work
transformers==4.48.3
transformers-stream-generator==0.0.5
triton==2.3.0
typer==0.15.1
typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/bld/rattler-build_typing_extensions_1744302253/work
tzdata==2025.1
umap-learn==0.5.7
urllib3==2.4.0
uvicorn==0.34.0
vocos==0.1.0
wandb==0.19.9
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1733231326287/work
websockets==14.2
widgetsnbextension==4.0.14
x-transformers==2.0.2
xxhash==3.5.0
yarl==1.18.3
zipp==3.21.0
Relevant Log Output

Additional Context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with speechbrain/inference/separation.py, especially SepformerSeparation.separate_file and the torchaudio.load call shown in the traceback. Reproduce the failure with the provided Path-based audio example, then verify that the method loads and separates the file successfully without the reported type error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
audio-video-rtc, machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.