invoke-ai / invoke-ai/InvokeAI
Multi-file install: a single part error deletes the whole install tmpdir, including completed parts and resumable partials
- Dominant language
- Python
- Stars
- 28.2k
- Forks
- 3k
- Avg merge
- 6d 5h
- Merged PRs (30d)
- 19
Description
## Summary
When any single part of a multi-file install errors, `_download_error_callback` deletes the **entire** install tmpdir (`model_install_default.py:1482-1489` → `_safe_rmtree(install_job._install_tmpdir)`) — including parts that finished completely and partials holding gigabytes of resumable progress. One transient failure (a 5xx on one file, a rename race) throws away everything the resume machinery exists to protect.
Found during the adversarial review of #9432; the rmtree behavior predates that PR, but #9432 added a new way to trip it (the sidecar rename race below).
## Mechanism
Worker thread: any exception other than `DownloadJobCancelledException` in `_do_download` marks the part ERROR (`download_default.py:327-330`). For a part belonging to an install, `_download_error_callback` then pops the install job, sets it errored, cancels the multifile job, and rmtrees the whole `_install_tmpdir`.
Two concrete triggers:
1. **Transient server error on one file.** A single `HTTP 500` on file *k* of an *n*-file install (raised at `download_default.py:456-459`) deletes the completed files 1..k-1 and all partial progress. The next attempt starts the entire install from zero.
2. **Sidecar rename race (new surface from #9432).** The 416-promotion path stats the sidecar (`download_default.py:352`) and renames it (`download_default.py:446`) a full network round-trip apart, with no existence guard. If the sidecar vanishes in between — user-triggered `restart_file()` runs `clear_partials=True` (`model_install_default.py:665`, `:1314`) against an in-flight request; manual deletion; AV/indexer interference on Windows — the rename raises `FileNotFoundError` → part ERROR → whole tmpdir deleted.
## Reproduction (test sketch, deterministic)
Trigger 1 needs only two mounts — one good file, one 500 — through the install service, then assert the tmpdir is gone despite file 1 having completed.
Trigger 2 can be made deterministic at the download-queue level with an adapter that deletes the sidecar during the request, simulating the race:
```python
class SidecarDeletingAdapter(TestAdapter):
def __init__(self, *args, sidecar: Path, **kwargs):
super().__init__(*args, **kwargs)
self._sidecar = sidecar
def send(self, request, **kwargs):
self._sidecar.unlink() # the race: sidecar vanishes mid-round-trip
return super().send(request, **kwargs)
def test_sidecar_vanishing_during_416_roundtrip(tmp_path: Path) -> None:
source = AnyHttpUrl("https://test.com/race.safetensors")
content = b"complete"
destination = tmp_path / "race.safetensors"
sidecar = destination.with_name(destination.name + ".downloading")
sidecar.write_bytes(content)
session = TestSession()
session.mount(
str(source),
SidecarDeletingAdapter(
b"", status=416, headers={"Content-Range": f"bytes */{len(content)}"}, sidecar=sidecar
),
)
queue = DownloadQueueService(requests_session=session)
queue.start()
try:
job = queue.download(source=source, dest=destination)
queue.join()
finally:
queue.stop()
# Today: FileNotFoundError escapes the rename -> job ERRORs; in an install context
# _download_error_callback then rmtrees the entire tmpdir.
assert job.status == DownloadJobStatus.ERROR
```
## Suggested fix
- In `_download_error_callback`, preserve the tmpdir when any part has resumable progress (a `.downloading` file on disk) or has already completed: write the install marker with an errored/paused status instead of rmtree, so the existing restore/`restart_failed` machinery can pick it up. Only rmtree when nothing on disk is worth keeping.
- Guard the 416 promotion rename: if the sidecar is missing at rename time, fall through to the pause/restart path instead of letting `FileNotFoundError` escalate to a part ERROR.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.