pytorch / pytorch/audio

New interface for MVDR beamforming

Open
#2,158 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.9k
Forks
799
Avg merge
58m
Merged PRs (30d)
3

Description

🚀 The feature

It would be very helpful to provide the following interface for the beamforming module (torchaudio.transforms.MVDR):

forward(specgram: torch.Tensor, psd_s: torch.Tensor, psd_n: torch.Tensor) → torch.Tensor

and maybe add some high-level glue functions that takes the masks as input, but has only a few lines of code.

Motivation, pitch

The current forward method of torchaudio.transforms.MVDR only accepts spectrogram and masks as input, and calculates the PSD matrices internally.

The current design is easy to use mainly for mask-based beamforming, but it may lose the flexibility for the users to:

  • use their own PSD estimates (e.g. mask-based PSDs, neural-network-based PSD estimates in ADL-MVDR and NICE-BEAM)
  • adjust PSD estimates with some additional regularization/stabilization techniques (e.g. PSD diagonal loading and PSD eigenvalue flooring)

    Because in different setups, a different stabilization may be ideal or preferred. It would be better to allow the user to make the choice.

  • combine with conventional blind beamforming methods
  • etc.
Alternatives

No response

Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the torchaudio.transforms.MVDR.forward entry point and compare its current spectrogram-and-mask interface with the proposed specgram, psd_s, and psd_n tensors. Confirm that externally supplied PSD estimates and the existing mask-based workflow are both covered; any high-level mask helpers are optional in the request.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
audio-video-rtc
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.