Implement FastGen as meta policy
- Dominant language
- Python
- Stars
- 18
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
BIGGER PROJECT
FastGen (https://openreview.net/forum?id=uNrFpDPMyo) is a very powerful meta-policy. It is implemented in `MInference`. It is a good solution to the problem of what policy (and which parameters) to use for each inference task.
**Describe the solution you'd like**
Implement `FastGen` as meta policy. Figure out how to compute outputs for simple policy just from full attention weights!
* Different scores for approx quality on initial part
* Ideally: Compute SDPA only once! So that a large number of policies can be chosen from
* Fast inference for fixed policy setup
* Try to find solution which bundles different `AttnWeigthsKVCache` policies
Contributor guide
Research direction
Start by reading the linked FastGen paper and the existing implementation in MInference. Map how policies and AttnWeigthsKVCache are represented, then define how the meta policy should compute outputs and select policies. Done means FastGen is implemented with the requested policy selection and inference behavior, validated against the stated goals.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100