containers / containers/ramalama

Enabling Support for Model Hot-Swapping

Open
#2,201 1 comment 1 reaction 1 assignee Claimed by @engelmi View on GitHub
enhancement stale-issue
Dominant language
Python
Stars
3k
Forks
364
Avg merge
2d 11h
Merged PRs (30d)
19

Description

### Feature request description

We have been exploring a new method for model hot-swapping that allows to swap between LLM models by transparently checkpointing the GPU state of an inference engine into host memory and pausing/resuming the running container.

We have implemented [SwapServeLLM](https://github.com/rst0git/SwapServeLLM) as a prototype of this functionality using Podman's Go bindings to manage containers, an OpenAI request router to handle inference requests, and the [cuda-checkpoint](http://github.com/nvidia/cuda-checkpoint) utility to transparently checkpoint/restore the GPU state. The following paper and presentation provide more information:
- Presentation: [slides](https://radostin.io/files/Engine-Agnostic-Model-Hot-Swapping-for-Cost-Effective-LLM-Inference-CANOPIE-HPC-2025.pdf), [demo](https://youtu.be/oIQNPCAzrH8)
- Research paper: https://dl.acm.org/doi/10.1145/3731599.3767354

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.