Using CPU inference there is no way to control how much ram of the host is used
Open
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 651
- Forks
- 155
- PR merge metrics
- No merged PRs in 30d
Description
We are using tiny models like granite 1b and no matter what settings pass it eventually eats all the server ram.
example
cat /proc/$(pgrep -f llama-server | head -n1)/cmdline | xargs -0 echo
/app/llama-server -ngl 999 --metrics --model /models/bundles/sha256/91eb206d6605bbeb033ea32d68e9ebb180539f2dd6bdff0d6e4e967ea15f5ace/model/model.gguf --host inference-runner-0.sock --ctx-size 2048 --parallel 2 --threads 5 --mlock --jinja
settings example
docker model configure --context-size 2048 granite-4.0-h-nano:1B -- --parallel 2 --threads 6 --mlock
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the host-RAM growth with the shown llama-server command and the docker model configure settings for context size, parallelism, threads, and mlock. Trace how those options reach CPU inference and identify the project entry point that controls memory usage. Done means a documented setting or behavior reliably prevents CPU inference from consuming all host RAM, with verification using the same process inspection command.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- ai, cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100