unit tests Windows GPU failing

🐛 Describe the bug

Following test are failing:
https://app.circleci.com/pipelines/github/pytorch/audio/16491/workflows/e9d2d0de-56ba-42f8-804b-77bf26fa291f/jobs/1213093

unittest_windows_gpu_py3.8:

FAILED torchaudio_unittest\io\stream_reader_test.py::FilterGraphWithCudaAccel::test_scale_cuda_format - RuntimeError: Failed to create the filter from "scale_cuda=format=yuv444p" (Invalid argument.)
FAILED torchaudio_unittest\io\stream_reader_test.py::FilterGraphWithCudaAccel::test_sclae_cuda_change_size - RuntimeError: Failed to create the filter from "scale_cuda=iw/2:ih/2" (Invalid argument.)
FAILED torchaudio_unittest\models\wav2vec2\huggingface_intergration_test.py::TestHFIntegration::test_import_finetune_2_wav2vec2_large_lv60k - AssertionError: Tensor-likes are not close!

Mismatched elements: 49150 / 49152 (100.0%)
Greatest absolute difference: 8.202713966369629 at index (10, 0, 812) (up to 0.0001 allowed)
Greatest relative difference: 970.2109375 at index (12, 2, 562) (up to 0.001 allowed)
FAILED torchaudio_unittest\models\wav2vec2\huggingface_intergration_test.py::TestHFIntegration::test_import_finetune_3_wav2vec2_large_lv60k - AssertionError: Tensor-likes are not close!

Mismatched elements: 49150 / 49152 (100.0%)
Greatest absolute difference: 8.202713966369629 at index (10, 0, 812) (up to 0.0001 allowed)
Greatest relative difference: 970.2109375 at index (12, 2, 562) (up to 0.001 allowed)
FAILED torchaudio_unittest\models\wav2vec2\huggingface_intergration_test.py::TestHFIntegration::test_import_finetune_4_wav2vec2_large_lv60k - AssertionError: Tensor-likes are not close!

Mismatched elements: 49151 / 49152 (100.0%)
Greatest absolute difference: 8.277942657470703 at index (3, 1, 336) (up to 0.0001 allowed)
Greatest relative difference: 590.3825073242188 at index (12, 2, 377) (up to 0.001 allowed)
FAILED torchaudio_unittest\models\wav2vec2\huggingface_intergration_test.py::TestHFIntegration::test_import_pretrain_2_wav2vec2_large_lv60k - AssertionError: Tensor-likes are not close!

Mismatched elements: 49148 / 49152 (100.0%)
Greatest absolute difference: 8.543177604675293 at index (14, 1, 662) (up to 0.0001 allowed)
Greatest relative difference: 1439.3043212890625 at index (1, 2, 958) (up to 0.001 allowed)
FAILED torchaudio_unittest\models\wav2vec2\huggingface_intergration_test.py::TestHFIntegration::test_import_pretrain_3_wav2vec2_large_lv60k - AssertionError: Tensor-likes are not close!

Mismatched elements: 49148 / 49152 (100.0%)
Greatest absolute difference: 8.543177604675293 at index (14, 1, 662) (up to 0.0001 allowed)
Greatest relative difference: 1439.3043212890625 at index (1, 2, 958) (up to 0.001 allowed)
FAILED torchaudio_unittest\models\wav2vec2\huggingface_intergration_test.py::TestHFIntegration::test_import_pretrain_wavlm_1_wavlm_large - AssertionError: Tensor-likes are not close!

Mismatched elements: 49150 / 49152 (100.0%)
Greatest absolute difference: 8.820459365844727 at index (5, 2, 486) (up to 0.0001 allowed)
Greatest relative difference: 3418.4130859375 at index (8, 2, 865) (up to 0.001 allowed)
FAILED torchaudio_unittest\prototype\conformer_wav2vec2_test.py::TestConformerWav2Vec2::test_pretrain_cuda_smoke_test__function_conformer_wav2vec2_pretrain_base_at_0x00000162D4F56E50__torch_float32 - AssertionError
FAILED torchaudio_unittest\prototype\conformer_wav2vec2_test.py::TestConformerWav2Vec2::test_pretrain_cuda_smoke_test__function_conformer_wav2vec2_pretrain_base_at_0x00000162D4F56E50__torch_float64 - AssertionError
FAILED torchaudio_unittest\prototype\ssl_model_test.py::TestSSLModel::test_cuda_smoke_test__function_conformer_wav2vec2_pretrain_base_at_0x00000162D4F56E50__64_torch_float32 - AssertionError
FAILED torchaudio_unittest\prototype\ssl_model_test.py::TestSSLModel::test_cuda_smoke_test__function_conformer_wav2vec2_pretrain_base_at_0x00000162D4F56E50__64_torch_float64 - AssertionError
= 12 failed, 7448 passed, 4737 skipped, 4 xfailed, 10402 warnings in 6572.91s (1:49:32) =
 000001639764e1c0] 264 - core 159 - H.264/MPEG-4 AVC codec - Copyleft 2003-2019 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=1 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
[libx264 @ 000001639764e1c0] frame I:41    Avg QP: 7.07  size:    19
[libx264 @ 000001639764e1c0] frame P:2523  Avg QP:10.07  size:    18
[libx264 @ 000001639764e1c0] frame B:7446  Avg QP:13.67  size:    13
[libx264 @ 000001639764e1c0] consecutive B-frames:  0.8%  0.0%  0.0% 99.2%
[libx264 @ 000001639764e1c0] mb I  I16..4: 100.0%  0.0%  0.0%
[libx264 @ 000001639764e1c0] mb P  I16..4:  0.0%  0.0%  0.0%  P16..4:  0.0%  0.0%  0.0%  0.0%  0.0%    skip:100.0%
[libx264 @ 000001639764e1c0] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8:  0.0%  0.0%  0.0%  direct: 0.0%  skip:100.0%
[libx264 @ 000001639764e1c0] 8x8 transform intra:0.0%
[libx264 @ 000001639764e1c0] coded y,uvDC,uvAC intra: 0.0% 0.0% 0.0% inter: 0.0% 0.0% 0.0%
[libx264 @ 000001639764e1c0] i16 v,h,dc,p:  0%  0% 100%  0%
[libx264 @ 000001639764e1c0] i8c dc,h,v,p: 100%  0%  0%  0%
[libx264 @ 000001639764e1c0] Weighted P-Frames: Y:0.0% UV:0.

unittest_windows_cpu_py3.8

=========================== short test summary info ===========================
FAILED torchaudio_unittest\models\wav2vec2\huggingface_intergration_test.py::TestHFIntegration::test_import_finetune_2_wav2vec2_large_lv60k - AssertionError: Tensor-likes are not close!

Mismatched elements: 49149 / 49152 (100.0%)
Greatest absolute difference: 8.202720642089844 at index (10, 0, 812) (up to 0.0001 allowed)
Greatest relative difference: 941.2529907226562 at index (12, 2, 562) (up to 0.001 allowed)
FAILED torchaudio_unittest\models\wav2vec2\huggingface_intergration_test.py::TestHFIntegration::test_import_finetune_3_wav2vec2_large_lv60k - AssertionError: Tensor-likes are not close!

Mismatched elements: 49149 / 49152 (100.0%)
Greatest absolute difference: 8.202720642089844 at index (10, 0, 812) (up to 0.0001 allowed)
Greatest relative difference: 941.2529907226562 at index (12, 2, 562) (up to 0.001 allowed)
FAILED torchaudio_unittest\models\wav2vec2\huggingface_intergration_test.py::TestHFIntegration::test_import_finetune_4_wav2vec2_large_lv60k - AssertionError: Tensor-likes are not close!

Mismatched elements: 49151 / 49152 (100.0%)
Greatest absolute difference: 8.277942657470703 at index (3, 1, 336) (up to 0.0001 allowed)
Greatest relative difference: 565.6646728515625 at index (12, 2, 377) (up to 0.001 allowed)
FAILED torchaudio_unittest\models\wav2vec2\huggingface_intergration_test.py::TestHFIntegration::test_import_pretrain_2_wav2vec2_large_lv60k - AssertionError: Tensor-likes are not close!

Mismatched elements: 49148 / 49152 (100.0%)
Greatest absolute difference: 8.543174743652344 at index (14, 1, 662) (up to 0.0001 allowed)
Greatest relative difference: 2450.420654296875 at index (1, 2, 958) (up to 0.001 allowed)
FAILED torchaudio_unittest\models\wav2vec2\huggingface_intergration_test.py::TestHFIntegration::test_import_pretrain_3_wav2vec2_large_lv60k - AssertionError: Tensor-likes are not close!

Mismatched elements: 49148 / 49152 (100.0%)
Greatest absolute difference: 8.543174743652344 at index (14, 1, 662) (up to 0.0001 allowed)
Greatest relative difference: 2450.420654296875 at index (1, 2, 958) (up to 0.001 allowed)
FAILED torchaudio_unittest\models\wav2vec2\huggingface_intergration_test.py::TestHFIntegration::test_import_pretrain_wavlm_1_wavlm_large - AssertionError: Tensor-likes are not close!

Mismatched elements: 49150 / 49152 (100.0%)
Greatest absolute difference: 8.820455551147461 at index (5, 2, 486) (up to 0.0001 allowed)
Greatest relative difference: 2746.401123046875 at index (8, 2, 865) (up to 0.001 allowed)
FAILED torchaudio_unittest\models\wav2vec2\huggingface_intergration_test.py::TestHFIntegration::test_import_xlsr_pretrain_0_wav2vec2_xlsr_300m - AssertionError: Tensor-likes are not close!

Mismatched elements: 49148 / 49152 (100.0%)
Greatest absolute difference: 8.543174743652344 at index (14, 1, 662) (up to 0.0001 allowed)
Greatest relative difference: 2450.420654296875 at index (1, 2, 958) (up to 0.001 allowed)
FAILED torchaudio_unittest\models\wav2vec2\huggingface_intergration_test.py::TestHFIntegration::test_import_xlsr_pretrain_1_wav2vec2_xlsr_1b - AssertionError: Tensor-likes are not close!

Mismatched elements: 61439 / 61440 (100.0%)
Greatest absolute difference: 20.117582321166992 at index (11, 2, 311) (up to 0.0001 allowed)
Greatest relative difference: 336.73870849609375 at index (5, 2, 1024) (up to 0.001 allowed)
FAILED torchaudio_unittest\models\wav2vec2\huggingface_intergration_test.py::TestHFIntegration::test_import_xlsr_pretrain_2_wav2vec2_xlsr_2b - AssertionError: Tensor-likes are not close!

Mismatched elements: 92159 / 92160 (100.0%)
Greatest absolute difference: 32.9678955078125 at index (9, 0, 1329) (up to 0.0001 allowed)
Greatest relative difference: 93.1497802734375 at index (14, 2, 365) (up to 0.001 allowed)
= 9 failed, 5502 passed, 6688 skipped, 2 xfailed, 6316 warnings in 4433.17s (1:13:53) =
8c dc,h,v,p: 100%  0%  0%  0%
[libx264 @ 000001408941e6c0] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 000001408941e6c0] kb/s:6.72
[libx264 @ 000001408941f480] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
[libx264 @ 000001408941f480] profile High, level 1.0, 4:2:0, 8-bit
[libx264 @ 000001408941f480] 264 - core 159 - H.264/MPEG-4 AVC codec - Copyleft 2003-2019 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=1 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
[libx264 @ 000001408941f480] frame I:41    Avg QP: 7.07  size:    19
[libx264 @ 000001408941f480] frame P:2523  Avg QP:10.07  size:    18
[libx264 @ 000001408941f480] frame B:7446  Avg QP:13.67  size:    13
[libx264 @ 000001408941f480] consecutive B-frames:  0.8%  0.0%  0.0% 99.2%
[libx264 @ 000001408941f480] mb I  I16..4: 100.0%  0.0%  0.0%
[libx264 @ 000001408941f480] mb P  I16..4:  0.0%  0.0%  0.0%  P16..4:  0.0%  0.0%  0.0%  0.0%  0.0%    skip:100.0%
[libx264 @ 000001408941f480] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8:  0.0%  0.0%  0.0%  direct: 0.0%  skip:100.0%
[libx264 @ 000001408941f480] 8x8 transform intra:0.0%
[libx264 @ 000001408941f480] coded y,uvDC,uvAC intra: 0.0% 0.0% 0.0% inter: 0.0% 0.0% 0.0%
[libx264 @ 000001408941f480] i16 v,h,dc,p:  0%  0% 100%  0%
[libx264 @ 000001408941f480] i8c dc,h,v,p: 100%  0%  0%  0%
[libx264 @ 000001408941f480] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 000001408941f480] kb/s:3.36

Exited with code exit status 1
CircleCI received exit code 1

Versions

2.1.0 nightly

Start with the linked CircleCI Windows jobs and reproduce the failures in torchaudio_unittest\io\stream_reader_test.py, models\wav2vec2\huggingface_intergration_test.py, prototype\conformer_wav2vec2_test.py, and prototype\ssl_model_test.py. Compare the CPU and GPU results and inspect the reported assertions and filter errors. Done means the Windows test jobs pass without these failures.