facebookresearch / facebookresearch/fairseq2
Check state reorder implementation in cross attention
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 144
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 1
Description
We have a performance optimization in `StaticAttentionState` where we avoid reordering the KV cache if the number of beams has not changed (see [here](https://github.com/facebookresearch/fairseq2/blob/f2be5c0196ac9eb566a1d2bc5ef3544f59220a83/src/fairseq2/nn/transformer/multihead_attention.py#L888)). However in our beam search implementation, it is theoretically possible to have the same number of sequences in a batch even when the number of beams changes. Although a very edge case, it would be nice to ensure that we handle it correctly. Check whether we can address it without introducing too much complexity.
Contributor guide
Assessment
This issue has not been assessed yet.