NVIDIA / NVIDIA/Megatron-LM

[QUESTION] Why is Grouped-Query Attention (GQA) not supported in CrossAttention?

Open
#1,814 1 comment 0 reactions 0 assignees View on GitHub
community-request question waiting-on-customer
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

## Question
When trying to enable Grouped-Query Attention (GQA) in the cross-attention module, the following check in the current `CrossAttention` class prevents it:

```python
if self.config.num_query_groups != self.config.num_attention_heads:
raise ValueError("Group query attention is not currently supported in cross attention.")
assert self.query_projection_size == self.kv_projection_size
```

This effectively disables GQA whenever num_query_groups < num_attention_heads, even though GQA is supported in self-attention.

I would like to understand the reasoning behind this limitation:

- Is there a technical or architectural constraint that makes GQA incompatible with cross-attention?

- Are there known issues with masking, KV-cache handling, or parallelism that block support?

From a user perspective, GQA could significantly reduce memory usage and KV-cache size for encoder–decoder or retrieval-augmented models where cross-attention dominates cost.

It would be wonderful if future versions could support this feature.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.