automatic unloading mechanism in Docker Model, similar to the one in Ollama
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 651
- Forks
- 155
- PR merge metrics
- No merged PRs in 30d
Description
I am migrating from Ollama to Docker Model, and I’ve noticed that DMR does not seem to automatically unload models when another model is requested, whereas Ollama handled this automatically. This can lead to errors when requesting new models, especially if there are memory or resource limitations in the environment where Docker Model is running.
Segue o exemplo abaixo para ilustrar o problema:
docker model run gpt-oss:20B "Hi"
Thinking:
We need to respond to a simple "Hi". Probably friendly.
docker model run gemma4:12B "Hi"
Failed to generate a response: error response: status=500 body=unable to load runner: error waiting for runner to be ready: llama.cpp terminated unexpectedly: llama.cpp failed: not enough GPU memory to load the model (CUDA)
Verbose output:
0.01.795.624 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 7010.14 MiB on device 0: cudaMalloc failed: out of memory
0.01.795.630 E alloc_tensor_range: failed to allocate CUDA0 buffer of size 7350663424
0.02.003.518 E llama_model_load: error loading model: unable to allocate CUDA0 buffer
0.02.003.524 E llama_model_load_from_file_impl: failed to load model
0.02.003.529 E cmn common_init_: failed to load model '/models/bundles/sha256/a3d3a4bff696915699db32fbbb0a29754df3d814d0d577610450a97cb279746a/model/gemma-4-12b-it-UD-Q4_K_XL.gguf'
0.02.003.534 E srv load_model: failed to load model, '/models/bundles/sha256/a3d3a4bff696915699db32fbbb0a29754df3d814d0d577610450a97cb279746a/model/gemma-4-12b-it-UD-Q4_K_XL.gguf'
0.02.003.536 I srv operator(): operator(): cleaning up before exit...
0.02.004.449 E srv llama_server: exiting due to model loading error
bug in the model. its type will be overridden
0.01.426.561 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
It is necessary to use the docker model unload command to manually release the previous model before loading a new one, which is not as convenient as Ollama's automatic behavior. There is an opportunity to implement an automatic unloading mechanism in Docker Model—similar to Ollama's—to improve the user experience and prevent memory issues??
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the docker model run and docker model unload entry points, then reproduce the two-model sequence under GPU memory pressure described in the issue. The change is done when requesting a new model automatically releases the previous model as needed and the second command succeeds without manual unloading.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- backend, cli
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100