mudler / mudler/LocalAI

Model Gallery sizing ("YOUR HOST") ignores registered Distributed Mode nodes

Open
#11,814 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/p2p area/webui bug need-more-information waiting-from-reporter
Dominant language
Go
Stars
49.2k
Forks
4.5k
Avg merge
1d 3m
Merged PRs (30d)
239

Description

Description

In Distributed Mode, the Model Gallery's "will this model fit" sizing check (the "YOUR HOST" panel and per-model fit indicators on /app/models) reads from /api/resources, which only reports the frontend's own local resources — not the aggregated/best-fit capacity of registered worker nodes from /api/nodes. This makes the Gallery sizing feature actively misleading (and in our case, useless) for any deployment where the frontend itself is a lightweight/non-compute host and real compute lives on registered distributed nodes.

Setup

  • v4.9.0, Distributed Mode enabled (LOCALAI_DISTRIBUTED=true, NATS + Postgres backing store, two nodes registered via local-ai worker --register-to)
  • Frontend runs on a small LXC container with 2GB RAM and no GPU — intentionally, it's meant to be a thin gateway
  • Two healthy compute nodes registered: one with an NVIDIA GPU (12GB VRAM), one Apple Silicon (16GB unified memory, capability: metal)

Evidence

GET /api/nodes correctly reports both real nodes:

[
  {"name":"AMXELA-Main","status":"healthy","gpu_vendor":"nvidia","total_vram":12884901888,"total_ram":67208949760,...},
  {"name":"MacBookAir.localdomain","status":"healthy","gpu_vendor":"apple","capability":"metal","total_vram":17179869184,"total_ram":17179869184,...}
]

But GET /api/resources (what the Gallery's "YOUR HOST" panel and sizing checks use) only ever reports the frontend's own tiny footprint, oblivious to any registered node:

{"aggregate":{"total_memory":2147483648,"used_memory":176013312,"free_memory":1971470336,"usage_percent":8.2,"gpu_count":0},"gpus":null,"ram":{"total":2147483648,...},"type":"ram",...}

Actual model scheduling/placement (via /api/node/register, NATS backend.install, and LOCALAI_MODEL_SCHEDULING node selectors) correctly uses the real node capacity from /api/nodes — we confirmed models schedule and run correctly on the appropriate node based on real hardware. So the underlying distributed capability-awareness works; it's specifically the Gallery UI's "will this fit" sizing feature that's still wired to the pre-distributed-mode single-host /api/resources data source and was never updated to consider the cluster.

Expected behavior

When Distributed Mode is enabled and nodes are registered, the Gallery's sizing/fit indicators should evaluate against the best-fit registered node's capacity (or an aggregate/max across nodes), not just the frontend host's own resources. As-is, on a thin-gateway deployment, the Gallery reports every model as basically not fitting anywhere, regardless of what real compute is actually available on the cluster.

Also observed (may be a separate/related issue)

The Gallery's model list gets stuck on "Checking availability..." indefinitely once 2 nodes are registered (worked fine, if inaccurately, with a single implicit host). Unconfirmed whether this is a performance issue from checking many gallery models against multiple nodes, or a separate bug — noting it here in case it's related to the same code path.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace the Model Gallery sizing checks on /app/models and identify how they consume /api/resources. Compare that path with /api/nodes and the existing distributed scheduling behavior; done means the YOUR HOST panel and per-model fit indicators use registered node capacity in Distributed Mode, while availability checks no longer remain stuck with multiple nodes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgres
Domain
api, distributed-systems, frontend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.