containers / containers/ramalama
Enabling Support for Model Hot-Swapping
- Dominant language
- Python
- Stars
- 3k
- Forks
- 364
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 19
Description
### Feature request description
We have been exploring a new method for model hot-swapping that allows to swap between LLM models by transparently checkpointing the GPU state of an inference engine into host memory and pausing/resuming the running container.
We have implemented [SwapServeLLM](https://github.com/rst0git/SwapServeLLM) as a prototype of this functionality using Podman's Go bindings to manage containers, an OpenAI request router to handle inference requests, and the [cuda-checkpoint](http://github.com/nvidia/cuda-checkpoint) utility to transparently checkpoint/restore the GPU state. The following paper and presentation provide more information:
- Presentation: [slides](https://radostin.io/files/Engine-Agnostic-Model-Hot-Swapping-for-Cost-Effective-LLM-Inference-CANOPIE-HPC-2025.pdf), [demo](https://youtu.be/oIQNPCAzrH8)
- Research paper: https://dl.acm.org/doi/10.1145/3731599.3767354
Contributor guide
Assessment
This issue has not been assessed yet.