Remove "--max_seq_length" option from export_llama
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5k
- Forks
- 1.2k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 581
Description
🚀 The feature, motivation and pitch
Remove "--max_seq_length" option from export_llama and make it either (1) runtime option or (2) model definition
Currently, when we export llama model, we have to specify max_seq_length
The reason is that it is used for initializing kv_cache.
In theory, --max_seq_length shouldn't be an export time config
It should be either be set during (1) initializing the model in eager mode and/or (2) setting during runtime
Implementation-wise, we need to consider both cases where kv_cache is an attribute vs io.
Cc @iseeyuan @kimishpatel
Alternatives
No response
Additional context
No response
RFC (Optional)
No response
cc @cccclai @helunwencser @dvorjackz
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the export_llama entry point and trace how --max_seq_length initializes kv_cache. Decide whether it belongs in the model definition or runtime, then account for both kv_cache-as-attribute and kv_cache-as-IO cases; done means export_llama no longer requires --max_seq_length.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- ai, machine-learning
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100