[etLLM] Explore the way to load a new LLM to llama_transformer
Open
@jackzhxng is already working on this.
Since Feb 5, 2025.
module: llm
triaged
- Dominant language
- Python
- Stars
- 5k
- Forks
- 1.2k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 581
Description
It's related to the first part (eager mode definition of this RFC.
There's llama_transformer that used to be Llama specific. A lot of infra are built around of this model, like source transforms, quantization, export, lowering to each backend, etc.
In open source a lot of models share the same architecture. Can we reuse llama_transformer and all the existing infra, to quickly enable a new model with optimized performance in multiple Edge backends (CPU, NPUs, CoreML, etc.)?
Items to explore:
- Utils to quickly convert different checkpoints to Llama compliant. An example is to leverage the torchtune utils and convert a Hugging Face checkpoint.
- Build the model quickly with proper configs.
- Any interfaces are missing to streamline this flow?
- Hide the technical details (users may not understand operators or a delegate partitioner), and expose necessary APIs for user to config, to meet their KPIs. For example, based on the accuracy and memory, provide supported quantization bit width for users to config.
- Quick return on results (perplexity, performance numbers, etc.)
cc @mergennachin @cccclai @helunwencser @dvorjackz
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.