pytorch / pytorch/audio

`F.vad` batch consistency behaviour

Open
#1,348 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.9k
Forks
799
Avg merge
58m
Merged PRs (30d)
3

Description

I isolated a potential issue with batch consistency and F.vad in #1341:

[Describing the change] Tweak the VAD consistency tests by using longer samples for the random test, and using lower thresholds for the file test. This now causes many of the VAD tests to fail.

This has me thinking - if the purpose of F.vad is to outright crop the input tensor (not, for example, zero it), shouldn't we expect it to produce different results on a sample vs. a batch? I believe my tweaks just contain the right conditions for that to happen, but I might be misreading. If this is the case, should F.vad be removed from the batch consistency tests?

@mthrok replied:

I see. I did not review the design of F.vad / T.Vad in detail, but your finding sounds like that the design of T.Vad is ill-formed. (I thought that F.vad would zero-out the non-vocal part). To me it does not make sense to batch the results that could be cropped in any way. We should check the behavior of the sox command, but @astaff do you have an idea of how it is supposed to behave?

Opening a dedicated issue so the discussion can continue here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the F.vad and T.Vad behavior and the VAD consistency tests discussed in this issue. Check the behavior of the sox command, then determine and test the expected relationship between sample and batch results, including whether cropping is compatible with batch consistency.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
audio-video-rtc
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.