Add Positional Encoding in Conformer Implementation

Missing Relative Positional Encoding in Conformer Implementation

Issue Description

The current Conformer implementation in Torchaudio is missing the relative sinusoidal positional encoding scheme that is a key component of the original Conformer architecture as described in the paper "Conformer: Convolution-augmented Transformer for Speech Recognition".

Details

In the original paper, section 2.1 "Multi-Headed Self-Attention Module" specifically states:

"We employ multi-headed self-attention (MHSA) while integrating an important technique from Transformer-XL [20], the relative sinusoidal positional encoding scheme. The relative positional encoding allows the self-attention module to generalize better on different input length and the resulting encoder is more robust to the variance of the utterance length."

However, the current implementation in conformer.py uses standard PyTorch MultiheadAttention without implementing the relative positional encoding:

self.self_attn = torch.nn.MultiheadAttention(input_dim, num_attention_heads, dropout=dropout)

Reference Implementation

For reference, NVIDIA's NeMo library does properly implement the positional encoding in their Conformer implementation: https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/modules/conformer_encoder.py

Start in conformer.py by reading the current torch.nn.MultiheadAttention setup and compare its behavior with the relative sinusoidal positional encoding described in section 2.1 of the Conformer paper. Use NVIDIA NeMo's conformer_encoder.py as the reference implementation. Done means the Conformer implementation includes the missing relative positional encoding scheme and preserves the intended attention behavior.