mudler / mudler/LocalAI

/api/ps and /api/tags report size and size_vram as a hardcoded 0

Open
#11,969 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/api bug confirmed
Dominant language
Go
Stars
49.2k
Forks
4.5k
Avg merge
1d 3m
Merged PRs (30d)
239

Description

LocalAI version:
v4.9.0 (f7ad3f70eb5d8a0ddf80e08557f0d7df28cf032e), image localai/localai:v4.9.0-gpu-nvidia-cuda-12

Environment, CPU architecture, OS, and Version:

x86_64, RTX 3060 12GB, Docker on WSL2

Describe the bug
GET /api/ps reports "size": 0 and "size_vram": 0 for every loaded model. The
values are literals rather than unpopulated fields:

https://github.com/mudler/LocalAI/blob/v4.9.0/core/http/endpoints/ollama/models.go#L106-L110

GET /api/tags has the same literal Size: 0 at line 40, and ExpiresAt on the ps
entry is synthesised as time.Now().Add(24 * time.Hour).

This is worse than omitting the fields. /api/ps is an Ollama-compatibility surface,
and Ollama reports real byte counts there, so a client written against Ollama reads
0 as authoritative and concludes the models cost nothing. Anything scheduling GPU
work from that will over-commit the card. An absent field or a null is safe, because
a consumer can distinguish "unknown" from "nothing"; a 0 cannot be distinguished.

To Reproduce

  1. Load any model (granite-4.1-8b, llama-cpp backend) so it is resident.
  2. curl localhost:8080/api/ps
  3. Compare against nvidia-smi.

Expected behavior
Either the real resident sizes, or the fields omitted/null when the backend cannot
report them. Not 0.

Logs

{"models":[{"name":"granite-4.1-8b:latest","size":0,"size_vram":0,
  "details":{"format":"gguf","family":"llama-cpp","parameter_size":"8B",
  "quantization_level":"Q4_K_M"}}]}

Card at the same moment: 10849 / 12288 MiB used, two models resident (second row
elided, also all-zero).

Additional context
I gave up on /api/ps for this engine entirely and treat the size as unknown unconditionally, because I can't tell its zeros from real ones.

Filed separately as a feature request: there is no endpoint that reports per-model memory at all.

I'm not in a position to take the PR, but I'm happy to test a fix against this setup.

Written by my beloved Claude Code

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in core/http/endpoints/ollama/models.go at the /api/ps and /api/tags mappings referenced by the issue. Trace how loaded llama-cpp models are represented and whether resident sizes are available; then verify both endpoints with a loaded model. Done means the responses no longer present unavailable size values as authoritative zeroes, while retaining real byte counts when the backend can report them.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.