facebookresearch / facebookresearch/blt

KV Cache in BLT Generation.

Open
#67 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2.1k
Forks
196
PR merge metrics
No merged PRs in 30d

Description

Thank you for your insight work! I find kv cache in your generate.py file, but it can't run on BLT generation in:

self.setup_prefilling(lengths=lengths)
prefill_out = self.model.forward(
tokens,
tok_idx=self.prefill_tok_id,
mask=self.prefill_mask,
attn_impl="flex_attention",
)

reporting:

TypeError: ByteLatentTransformer.forward() got an unexpected keyword argument 'mask'

TypeError: ByteLatentTransformer.forward() got an unexpected keyword argument 'tok_idx'

How can I use kv cache in blt generation?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.