pytorch / pytorch/executorch

[etLLM] Explore the way to load a new LLM to llama_transformer

Open
#8,231 1 comment 0 reactions 1 assignee View on GitHub

@jackzhxng is already working on this.

Since Feb 5, 2025.

module: llm triaged
Dominant language
Python
Stars
5k
Forks
1.2k
Avg merge
2d 10h
Merged PRs (30d)
581

Description

It's related to the first part (eager mode definition of this RFC.

There's llama_transformer that used to be Llama specific. A lot of infra are built around of this model, like source transforms, quantization, export, lowering to each backend, etc.

In open source a lot of models share the same architecture. Can we reuse llama_transformer and all the existing infra, to quickly enable a new model with optimized performance in multiple Edge backends (CPU, NPUs, CoreML, etc.)?

Items to explore:

  • Utils to quickly convert different checkpoints to Llama compliant. An example is to leverage the torchtune utils and convert a Hugging Face checkpoint.
  • Build the model quickly with proper configs.
  • Any interfaces are missing to streamline this flow?
  • Hide the technical details (users may not understand operators or a delegate partitioner), and expose necessary APIs for user to config, to meet their KPIs. For example, based on the accuracy and memory, provide supported quantization bit width for users to config.
  • Quick return on results (perplexity, performance numbers, etc.)

cc @mergennachin @cccclai @helunwencser @dvorjackz

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.