OpenPipe / OpenPipe/ART

Decouple vLLM & Unsloth Trainer

Open
#210 5 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
10.8k
Forks
989
Avg merge
6h 29m
Merged PRs (30d)
85

Description

Unsloth does not yet support the vLLM V1 engine or multi-device training. A realistic solution is to decouple vLLM for inference and the Unsloth model for training so that we can update them independently.

As a first step I propose creating a new ModelService protocol compliant service class, DecoupledUnslothService in a new file /src/art/unsloth/decoupled-service.py. It will mimic the approach taken with the TorchtuneService in /src/art/torchtune/service.py with a division between the vLLM (V1) engine and the trainer. Some logical internal model config setting will act as a sentinel to indicate that the new DecoupledUnslothService should be used instead of the UnslothService in /src/art/local/backend.py. In the same way as with the torchtune service, we will offload vLLM's model and KV cache during training and load the Unsloth model and optimizer. After training is finished, we will offload the Unsloth model and optimizer and save the LoRA checkpoint to disk. We will then reload vLLM's model and KV cache and then load the latest LoRA checkpoint so that the model's name points to the appropriate LoRA, same as we do now in the UnslothService.

We want to do this with the absolute minimum number of changes to the existing code to minimize disruption and avoid any breaking changes to usage of the existing Unsloth and torchtune model services.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read /src/art/torchtune/service.py and /src/art/local/backend.py first, then trace the existing UnslothService flow. The proposed work is centered on a new /src/art/unsloth/decoupled-service.py implementing the ModelService protocol. Done means vLLM inference and Unsloth training can be updated independently while existing Unsloth and torchtune services remain compatible.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.