facebookresearch / facebookresearch/fairseq2

Check state reorder implementation in cross attention

Open
#307 0 comments 0 reactions 0 assignees View on GitHub
bug generation
Dominant language
Python
Stars
1.1k
Forks
144
Avg merge
4d 1h
Merged PRs (30d)
1

Description

We have a performance optimization in `StaticAttentionState` where we avoid reordering the KV cache if the number of beams has not changed (see [here](https://github.com/facebookresearch/fairseq2/blob/f2be5c0196ac9eb566a1d2bc5ef3544f59220a83/src/fairseq2/nn/transformer/multihead_attention.py#L888)). However in our beam search implementation, it is theoretically possible to have the same number of sequences in a batch even when the number of beams changes. Although a very edge case, it would be nice to ensure that we handle it correctly. Check whether we can address it without introducing too much complexity.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.