Wav2letter model outputs incorrect on Vulkan

🐛 Describe the bug

When running the Wav2letter model on Vulkan, the outputs appear to significantly diverge from eager mode. As this is output tolerance related, feel free to close this issue if it's working as intended. The model does meet the atol=0.1, rtol=0.04 metric on other backends, so the error appears to be significant.

Output stats:

Output 0 does not match reference output.
Given atol: 0.1, rtol: 0.04.
Output tensor shape: torch.Size([1, 40, 4]), dtype: torch.float32
Difference: max: 1.3840556144714355, abs: 8.47005558013916, ... 160
Median: -12.151180267333984, -3.687800407409668
Mean: -9.692764343321324, -3.688911700248718
Max: -2.2928080558776855, -3.6739728450775146
Min: -12.16968059539795, -3.699754238128662

Lowering repro:

import torch
import torchaudio
from executorch.exir import to_edge_transform_and_lower

from executorch.backends.vulkan.partitioner.vulkan_partitioner import VulkanPartitioner

model = torchaudio.models.Wav2Letter()
inputs = (torch.randn(1, 1, 1024),)

ep = torch.export.export(model, inputs)

model = to_edge_transform_and_lower(
    torch.export.export(model, inputs),
    partitioner=[VulkanPartitioner()],
).to_executorch()

The output stats are from a CI run using Swiftshader: https://github.com/pytorch/executorch/actions/runs/17505373586. See the "vulkan, models" job.

Versions

Commit https://github.com/pytorch/executorch/commit/fbda3a9545de747329577bd910086072ec5c7ad1, output stats are from CI on https://github.com/pytorch/executorch/actions/runs/17505373586

cc @SS-JIA @manuelcandales @cbilgin

Start with the Wav2Letter export and lowering repro using VulkanPartitioner, then inspect the Vulkan, models CI job and compare its output with eager mode under the stated tolerances. Done means determining why the Vulkan result diverges and confirming corrected outputs meet the atol=0.1, rtol=0.04 metric or documenting that the divergence is expected.