docker / docker/model-runner

automatic unloading mechanism in Docker Model, similar to the one in Ollama

Open
#1,014 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
651
Forks
155
PR merge metrics
No merged PRs in 30d

Description

I am migrating from Ollama to Docker Model, and I’ve noticed that DMR does not seem to automatically unload models when another model is requested, whereas Ollama handled this automatically. This can lead to errors when requesting new models, especially if there are memory or resource limitations in the environment where Docker Model is running.

Segue o exemplo abaixo para ilustrar o problema:

docker model run gpt-oss:20B "Hi"
Thinking:
We need to respond to a simple "Hi". Probably friendly.
docker model run gemma4:12B "Hi"
Failed to generate a response: error response: status=500 body=unable to load runner: error waiting for runner to be ready: llama.cpp terminated unexpectedly: llama.cpp failed: not enough GPU memory to load the model (CUDA)

Verbose output:
0.01.795.624 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 7010.14 MiB on device 0: cudaMalloc failed: out of memory
0.01.795.630 E alloc_tensor_range: failed to allocate CUDA0 buffer of size 7350663424
0.02.003.518 E llama_model_load: error loading model: unable to allocate CUDA0 buffer
0.02.003.524 E llama_model_load_from_file_impl: failed to load model
0.02.003.529 E cmn  common_init_: failed to load model '/models/bundles/sha256/a3d3a4bff696915699db32fbbb0a29754df3d814d0d577610450a97cb279746a/model/gemma-4-12b-it-UD-Q4_K_XL.gguf'
0.02.003.534 E srv    load_model: failed to load model, '/models/bundles/sha256/a3d3a4bff696915699db32fbbb0a29754df3d814d0d577610450a97cb279746a/model/gemma-4-12b-it-UD-Q4_K_XL.gguf'
0.02.003.536 I srv    operator(): operator(): cleaning up before exit...
0.02.004.449 E srv  llama_server: exiting due to model loading error
bug in the model. its type will be overridden
0.01.426.561 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list

It is necessary to use the docker model unload command to manually release the previous model before loading a new one, which is not as convenient as Ollama's automatic behavior. There is an opportunity to implement an automatic unloading mechanism in Docker Model—similar to Ollama's—to improve the user experience and prevent memory issues??

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the docker model run and docker model unload entry points, then reproduce the two-model sequence under GPU memory pressure described in the issue. The change is done when requesting a new model automatically releases the previous model as needed and the second command succeeds without manual unloading.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
backend, cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.