asteroid-team / asteroid-team/asteroid
Use Pretrained Model for multi-channel input
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 450
- PR merge metrics
- No merged PRs in 30d
Description
Hello Asteroid-Team/Community,
I have a task where i need to separate 2-3 sound-sources from a noisy two channeled (binaural-audio) input. I am using pretrained asteroid models (like https://huggingface.co/JorisCos/ConvTasNet_Libri3Mix_sepnoisy_16k). But when i try to separate the 2 channels together i get an error. If i iterate over the channels and separate each separately it works but i want to handle them together to get better results.
I looked up into the class ` BaseModel` where it looks like the in_channels are fixed to 1 because the in_channels argument isnt called from the init function from the inherited class. Can anyone explain me if it is possible to use mutli-channel input for pretrained models and how this is possible?
```
class Base(torch.nn.Module):
"""Base class for serializable models.
Defines saving/loading procedures, and separation interface to `separate`.
Need to overwrite the `forward` and `get_model_args` methods.
Models inheriting from `BaseModel` can be used by :mod:`asteroid.separate`
and by the `asteroid-infer` CLI. For models whose `forward` doesn't go from
waveform to waveform tensors, overwrite `forward_wav` to return
waveform tensors.
Args:
sample_rate (float): Operating sample rate of the model.
in_channels: Number of input channels in the signal.
If None, no checks will be performed.
"""
def __init__(self, sample_rate: float, in_channels: Optional[int] = 1):
super().__init__()
```
Contributor guide
Research direction
Start with the Base class shown in the issue and inspect the linked pretrained ConvTasNet model's expected input shape. Trace how multi-channel input reaches asteroid.separate and determine whether pretrained models can support it; done means documenting the supported input shape and the required scope for multi-channel support.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- audio-video-rtc, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100