awslabs / awslabs/keys_values

Implement FastGen as meta policy

Open
#17 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
18
Forks
8
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**

BIGGER PROJECT

FastGen (https://openreview.net/forum?id=uNrFpDPMyo) is a very powerful meta-policy. It is implemented in `MInference`. It is a good solution to the problem of what policy (and which parameters) to use for each inference task.

**Describe the solution you'd like**

Implement `FastGen` as meta policy. Figure out how to compute outputs for simple policy just from full attention weights!

* Different scores for approx quality on initial part
* Ideally: Compute SDPA only once! So that a large number of policies can be chosen from
* Fast inference for fixed policy setup
* Try to find solution which bundles different `AttnWeigthsKVCache` policies

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked FastGen paper and the existing implementation in MInference. Map how policies and AttnWeigthsKVCache are represented, then define how the meta policy should compute outputs and select policies. Done means FastGen is implemented with the requested policy selection and inference behavior, validated against the stated goals.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.