[bug] Image upgrade doesn't detect stale backend venv → cryptic 'Engine core init failed'; install.sh needs uv but nvidia-l4t image has neither uv nor pip

Summary

Two related bugs I hit today on the nvidia-l4t-arm64-cuda-13 variant when upgrading LocalAI from v4.6.0 to v4.6.2 while keeping the persistent /backends/ volume:

1) Silent ABI mismatch between container image and backend venv

After the image swap (keeping -v localai-backends:/backends), model loads fail with:

ERROR Failed to load model modelID="coder"
  error=failed to load model with internal loader: could not load model (no success):
  Unexpected err=RuntimeError('Engine core initialization failed. See root cause above.
  Failed core proc(s): {}'), type(err)=<class 'RuntimeError'>
  backend="cuda13-nvidia-l4t-arm64-vllm"

The backend subprocess exits with exitCode=0 in ~10s. The Failed core proc(s): {} set is empty, no Python traceback surfaces in docker logs, and there is no obvious hint that the backend venv on the persistent volume is incompatible with the newer LocalAI image.

The LocalAI startup log DOES contain:

INFO Backend upgrade available (new build) backend="cuda13-nvidia-l4t-arm64-vllm"

…but that line looks informational (as opposed to causal), sits among unrelated "backend upgrade available" lines for other backends, and has no severity elevation. Without prior knowledge of the mechanism it's very hard to connect this to the load failure.

2) Recovery path via local-ai backends install is broken on the minimal image

The documented recovery — reinstalling the backend so the venv is rebuilt for the new image ABI — fails because the minimal nvidia-l4t-arm64-cuda-13 container has neither uv nor pip:

$ docker exec localai /local-ai backends install cuda13-nvidia-l4t-arm64-vllm
downloading backend cuda13-nvidia-l4t-arm64-vllm 100% |...|  DONE

$ docker exec localai bash /backends/cuda13-nvidia-l4t-arm64-vllm/install.sh
Initializing libbackend for cuda13-nvidia-l4t-arm64-vllm
Using portable Python
./common/libbackend.sh: line 378: uv: command not found

Result: OCI-layer download succeeds, backend.py and requirements-*.txt land in the backend dir, but venv/ never gets built. Backend is permanently broken until I manually restore a pre-upgrade snapshot from outside the container.

Both pip and uv are absent inside the image, so there is no way to bootstrap the missing package manager from within.

Repro

1. Run LocalAI v4.6.0 (nvidia-l4t-arm64-cuda-13), install a backend and load a model successfully.
2. Stop container, docker rm, and start LocalAI v4.6.2 with the same volumes (localai-backends, localai-models, …).
3. POST /v1/chat/completions for any model on that backend → 500 with the empty Failed core proc(s): {} message above.
4. Try to recover with docker exec localai /local-ai backends install cuda13-nvidia-l4t-arm64-vllm → OCI download completes but venv is not rebuilt.
5. Try bash /backends/cuda13-nvidia-l4t-arm64-vllm/install.sh inside the container → uv: command not found.

Expected behavior

At least one, ideally all:

• Diagnose ABI/version mismatch and surface it clearly: e.g. Backend cuda13-nvidia-l4t-arm64-vllm was installed with LocalAI v4.6.0 and may be incompatible with v4.6.2; run 'local-ai backends upgrade' before serving.
• Elevate "Backend upgrade available" to WARN with a "reinstall required" hint when the delta likely breaks ABI.
• Pipe backend-subprocess stderr into docker logs so the real Python-side traceback isn't lost behind the empty Failed core proc(s): {} set. The current wrapper hides the actual error.
• Bundle uv (or fall back to pip) in every nvidia-l4t-arm64-cuda-* image so that local-ai backends install actually completes end-to-end. Alternatively make install.sh bootstrap uv itself when missing.

Right now the documented recovery path is impossible from within the container.

Version

LocalAI v4.6.2 (variant nvidia-l4t-arm64-cuda-13). Reproducible.

Environment

• Host: NVIDIA DGX Spark (GB10, ARM64, 121 GB unified memory)
• Image: localai/localai:v4.6.2-nvidia-l4t-arm64-cuda-13
• Backend venv persistent on localai-backends volume

Start by reproducing the v4.6.0-to-v4.6.2 upgrade with the persistent backend volume and the local-ai backends install path. Inspect install.sh, common/libbackend.sh around line 378, and the backend-subprocess logging and upgrade messages on the nvidia-l4t-arm64-cuda-13 image. Done when the incompatibility and recovery behavior are clearly diagnosed and the backend failure details are surfaced as intended.