asteroid-team / asteroid-team/asteroid

Use Pretrained Model for multi-channel input

Open
#703 1 comment 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
2.6k
Forks
450
PR merge metrics
No merged PRs in 30d

Description

Hello Asteroid-Team/Community,
I have a task where i need to separate 2-3 sound-sources from a noisy two channeled (binaural-audio) input. I am using pretrained asteroid models (like https://huggingface.co/JorisCos/ConvTasNet_Libri3Mix_sepnoisy_16k). But when i try to separate the 2 channels together i get an error. If i iterate over the channels and separate each separately it works but i want to handle them together to get better results.
I looked up into the class ` BaseModel` where it looks like the in_channels are fixed to 1 because the in_channels argument isnt called from the init function from the inherited class. Can anyone explain me if it is possible to use mutli-channel input for pretrained models and how this is possible?

```
class Base(torch.nn.Module):
"""Base class for serializable models.

Defines saving/loading procedures, and separation interface to `separate`.
Need to overwrite the `forward` and `get_model_args` methods.

Models inheriting from `BaseModel` can be used by :mod:`asteroid.separate`
and by the `asteroid-infer` CLI. For models whose `forward` doesn't go from
waveform to waveform tensors, overwrite `forward_wav` to return
waveform tensors.

Args:
sample_rate (float): Operating sample rate of the model.
in_channels: Number of input channels in the signal.
If None, no checks will be performed.
"""

def __init__(self, sample_rate: float, in_channels: Optional[int] = 1):
super().__init__()

```

Contributor guide

Open the contributing guide

Research direction

Start with the Base class shown in the issue and inspect the linked pretrained ConvTasNet model's expected input shape. Trace how multi-channel input reaches asteroid.separate and determine whether pretrained models can support it; done means documenting the supported input shape and the required scope for multi-channel support.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
audio-video-rtc, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.