Upgrade backends fails if auto-upgrade is enabled

LocalAI version: v4.9.0 (built from commit 98649d77, includes the install-resume fix d7ff4378).

Environment, CPU architecture, OS, and Version:

• OS: openSUSE Tumbleweed
• Architecture: x86_64
• Kernel:

Linux 7.1.8-1-default #1 SMP PREEMPT_DYNAMIC Mon Aug 10 05:03:20 UTC 2026 (f1071af) x86_64 x86_64 x86_64 GNU/Linux

• CPU: AMD EPYC 9124
• RAM: 64 GB
• GPU: AMD Radeon AI PRO R9700 32 GB
• Deployment: Docker / Docker Compose

Relevant runtime settings: "auto_upgrade_backends": true, "single_backend": true, backend_galleries: github:mudler/LocalAI/backend/index.yaml@master
~25 installed gallery backends in /backends, most of them multi-GB ROCm images

Describe the bug

With backend auto-upgrade enabled, manual backend upgrades fail deterministically while fresh installs of the same backends always succeed. The failures happen during OCI image extraction into the .upgrade-tmp staging directory:

failed to download backend: failed to extract image tar: lstat /backends/rocm-diffusers.upgrade-tmp/venv/lib/python3.10/site-packages/torch/lib/hipblaslt/library/TensileLibrary_..._gfx942.dat: no such file or directory
failed to extract image tar: open /backends/rocm-vllm.upgrade-tmp/Makefile: no such file or directory
failed to extract image tar: open /backends/rocm-acestep-cpp-development.upgrade-tmp/acestep-cpp: no such file or directory
failed to extract image tar: mkdir /backends/rocm-llama-cpp-development.upgrade-tmp/lib: no such file or directory

Worse, a failed upgrade deletes the previously installed backend, so the backend has to be reinstalled from scratch to be usable again (after a failed rocm-acestep-cpp-development upgrade, the directory was completely gone from /backends, including the backup).

To Reproduce

Run LocalAI single-node with several multi-GB gallery backends installed and auto-upgrade backends enabled.
From the UI/API run a manual (or "upgrade all") backend upgrade.
When the download finish, the extraction fails.

The manual upgrade fails with the extraction errors above, and UpgradeChecker keeps re-triggering checks after every completed backend op (OnBackendOpCompleted → TriggerCheck), so auto-upgrades run concurrently with the manual worker.

Root cause (proposed): gallery.UpgradeBackend (core/gallery/upgrade.go) is entered concurrently from two independent code paths for different backends in the same session, with no single-node serialization:

• the manual path runs serially in the GalleryService worker (core/services/galleryop/service.go);
• the auto-upgrade path (core/application/upgrade_checker.go, AutoUpgradeBackends: true) runs in its own goroutine and calls gallery.UpgradeBackend directly.
  The duplicate-op guard in backendHandler only exists in distributed mode (if g.galleryStore != nil), which is nil in single-node. Interleaved os.RemoveAll(tmpPath)/os.RemoveAll(backupPath) calls at the start/failure of one UpgradeBackend run then destroy the staging dir of the other run mid-archive.Apply, and the atomic-swap restore (Rename backup → backend) can fail because a different operation deleted the backup — which is why the old backend is lost instead of being rolled back. Installs are unaffected because they stage into different paths (.install-tmp/.install-backup, InstallBackend in core/gallery/backends.go) and are serialized by the worker.

Qwen 3.8 27B believe that the correct fix is: a per-backend locking mechanism (e.g. per-path mutexes) acquired at the top of gallery.UpgradeBackend and gallery.InstallBackend, plus skipping auto-upgrades (or returning a clear "already in progress" error) for backends that already have an upgrade op in flight; optionally also ensuring the extraction destination dir exists (os.MkdirAll at the top of ExtractOCIImageFromTar/extractTarCopyingLinks) so the OCI layer is self-sufficient.

Expected behavior

Manual backend upgrades succeed even with auto-upgrade backends enabled. If an auto-upgrade of the same backend is already in progress, the checker should skip it and/or the two paths should be serialized per backend. A failed upgrade must roll back to the previously installed version and never leave the backend missing.

Logs
failed to download backend: failed to extract image tar: lstat /backends/rocm-diffusers.upgrade-tmp/venv/lib/python3.10/site-packages/torch/lib/hipblaslt/library/TensileLibrary_..._gfx942.dat: no such file or directory
failed to extract image tar: open /backends/rocm-vllm.upgrade-tmp/Makefile: no such file or directory
failed to extract image tar: open /backends/rocm-acestep-cpp-development.upgrade-tmp/acestep-cpp: no such file or directory
failed to extract image tar: mkdir /backends/rocm-llama-cpp-development.upgrade-tmp/lib: no such file or directory

Verification that the race is the cause

• With auto-upgrade backends disabled, restarting the container, and running the same two manual upgrades → both succeeded.
• The same backends that failed to upgrade installed cleanly while auto-upgrade was enabled, because the install path stages into different temp dirs (.install-tmp) and is not touched by the auto-upgrade path.

Additional context

• The analysis and the proposed solution was made by Qwen 3.8 27B via LocalAI
• /backends is a plain local bind mount — no network/CIFS filesystem, so the CIFS symlink-fallback path in pkg/oci/image.go is not involved.
• Backend state after a failed upgrade: both rocm-acestep-cpp-development and its backup were gone; no *.upgrade-tmp or *.backup leftovers remained in /backends (consistent with the losing side of the race cleaning up the other side's backup in the swap/rollback sequence).
• Suggested minimal fix for maintainers: single-node per-backend serialization between the GalleryService worker and UpgradeChecker (e.g. per-backend mutexes in the gallery package, plus an in-flight op check so runCheck's auto-upgrades back off when a manual op is active).

Start with core/gallery/upgrade.go and trace calls from core/services/galleryop/service.go and core/application/upgrade_checker.go, then compare the staging and rollback paths in core/gallery/backends.go. Reproduce an upgrade with auto-upgrade enabled and inspect the extraction helpers in pkg/oci/image.go. Done means concurrent operations are serialized per backend and a failed upgrade preserves the previously installed backend.