awslabs / awslabs/keys_values

Can KV reordering be avoided with `flex_attention`?

Open
#39 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
18
Forks
8
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**

In the `flex_attention` integration, we take `token_positions` into account by reordering `key, value` for every call. This adds time and especially memory requirements.

It also does not work with `sliding_window_size` given, even though this is easily supported in `flex_attention` if `token_positions` is not given.

It would be much simpler to use a `mask_mod` of the form
`q_idx + input_pos >= token_positions[kv_idx]`

Here, `input_pos, token_positions` are captured tensors, and `token_positions.ndim == 1`. We cannot use an overall fix `token_positions`, because it changes with every call. On the other hand, `input_pos` could be folded into `token_positions`.

**Describe the solution you'd like**

We need a solution where `token_positions` becomes an input to the kernel or its `BlockMask`. This is not doable as in the simple examples, where captured tensors must be scalar (so `ndim==0`). But it may work with what they call BlockMask conversion, where the BlockMask can be changed without kernel re-compilation.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the flex_attention integration and how token_positions currently triggers key/value reordering. Read the mask_mod examples and the BlockMask conversion path, then determine whether token_positions can be passed as a kernel or BlockMask input without recompilation. Done means avoiding per-call KV reordering while preserving token-position behavior and supporting sliding_window_size.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.