facebookresearch / facebookresearch/LayerSkip

Use HuggingFace's new KV Cache Implementation

Open
#9 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
380
Forks
47
PR merge metrics
No merged PRs in 30d

Description

In order to enable Llama3.2 1B (see #8 ), we had to upgrade from `transformers` v4.34.1 to v4.45.2.

This new version of `transformers` had refactored the KV cache implementation to a more efficient implementation that would have required us to refactor `forward_early(...)` and `forward_remainder(...)` in `self_speculation/llama_model_utils.py`. Instead, we opted to use the less efficient legacy KV cache.

In order to ensure apples-to-apples comparison, in 62debc0db07f26e6652ac176bae303d63b03873b, we changed autoregressive decoding to use legacy cache.

Ideally, we should ensure `forward_early(...)` and `forward_remainder(...)` to use `transformers` new more efficient KV cache implementation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.