Model Gallery sizing ("YOUR HOST") ignores registered Distributed Mode nodes
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 49.2k
- Forks
- 4.5k
- Avg merge
- 1d 3m
- Merged PRs (30d)
- 239
Description
Description
In Distributed Mode, the Model Gallery's "will this model fit" sizing check (the "YOUR HOST" panel and per-model fit indicators on /app/models) reads from /api/resources, which only reports the frontend's own local resources — not the aggregated/best-fit capacity of registered worker nodes from /api/nodes. This makes the Gallery sizing feature actively misleading (and in our case, useless) for any deployment where the frontend itself is a lightweight/non-compute host and real compute lives on registered distributed nodes.
Setup
- v4.9.0, Distributed Mode enabled (
LOCALAI_DISTRIBUTED=true, NATS + Postgres backing store, two nodes registered vialocal-ai worker --register-to) - Frontend runs on a small LXC container with 2GB RAM and no GPU — intentionally, it's meant to be a thin gateway
- Two healthy compute nodes registered: one with an NVIDIA GPU (12GB VRAM), one Apple Silicon (16GB unified memory,
capability: metal)
Evidence
GET /api/nodes correctly reports both real nodes:
[
{"name":"AMXELA-Main","status":"healthy","gpu_vendor":"nvidia","total_vram":12884901888,"total_ram":67208949760,...},
{"name":"MacBookAir.localdomain","status":"healthy","gpu_vendor":"apple","capability":"metal","total_vram":17179869184,"total_ram":17179869184,...}
]
But GET /api/resources (what the Gallery's "YOUR HOST" panel and sizing checks use) only ever reports the frontend's own tiny footprint, oblivious to any registered node:
{"aggregate":{"total_memory":2147483648,"used_memory":176013312,"free_memory":1971470336,"usage_percent":8.2,"gpu_count":0},"gpus":null,"ram":{"total":2147483648,...},"type":"ram",...}
Actual model scheduling/placement (via /api/node/register, NATS backend.install, and LOCALAI_MODEL_SCHEDULING node selectors) correctly uses the real node capacity from /api/nodes — we confirmed models schedule and run correctly on the appropriate node based on real hardware. So the underlying distributed capability-awareness works; it's specifically the Gallery UI's "will this fit" sizing feature that's still wired to the pre-distributed-mode single-host /api/resources data source and was never updated to consider the cluster.
Expected behavior
When Distributed Mode is enabled and nodes are registered, the Gallery's sizing/fit indicators should evaluate against the best-fit registered node's capacity (or an aggregate/max across nodes), not just the frontend host's own resources. As-is, on a thin-gateway deployment, the Gallery reports every model as basically not fitting anywhere, regardless of what real compute is actually available on the cluster.
Also observed (may be a separate/related issue)
The Gallery's model list gets stuck on "Checking availability..." indefinitely once 2 nodes are registered (worked fine, if inaccurately, with a single implicit host). Unconfirmed whether this is a performance issue from checking many gallery models against multiple nodes, or a separate bug — noting it here in case it's related to the same code path.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace the Model Gallery sizing checks on /app/models and identify how they consume /api/resources. Compare that path with /api/nodes and the existing distributed scheduling behavior; done means the YOUR HOST panel and per-model fit indicators use registered node capacity in Distributed Mode, while availability checks no longer remain stuck with multiple nodes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, postgres
- Domain
- api, distributed-systems, frontend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100