facebookresearch / facebookresearch/fairseq2
`WaveformToFbankConverter` does not work as expected for `channels > 1`
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 144
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 1
Description
**Describe the bug:**
While the `AudioDecoder` can read a stereo wav file, the output of `WaveformToFbankConverter` is always `num_frames x num_mel_bins`.
**Describe how to reproduce:**
```python
audio_wav_path, device, dtype = ...
audio_decoder = AudioDecoder(dtype=torch.float32, device=device)
fbank_converter = WaveformToFbankConverter(
num_mel_bins=80,
waveform_scale=2**15,
channel_last=True,
standardize=True,
device=device,
dtype=dtype,
)
with Path(audio_wav_path).open("rb") as fb:
block = MemoryBlock(fb.read())
decoded_audio = audio_decoder(block) # for stereo waveform shape is (num_samples x num_channels=2)
fbank_features = fbank_converter(decoded_audio)['fbank']
# shape is always (num_frames x num_mel_bins) irrespective of num_channels
```
**Describe the expected behavior:**
`WaveformToFbankConverter` should handle stereo/multi-channel audio and return `(num_channels x num_frames x num_mel_bins)`
**Environment:**
Installed with following command:
`pip install fairseq2 --pre --extra-index-url https://fair.pkg.atmeta.com/fairseq2/whl/nightly/pt2.2.0/cu118`
Contributor guide
Assessment
This issue has not been assessed yet.