invoke-ai / invoke-ai/InvokeAI
Restored install that pauses before its first download-started signal cannot be restarted (restart_failed no-ops, marker loses resume metadata)
- Dominant language
- Python
- Stars
- 28.2k
- Forks
- 3k
- Avg merge
- 6d 5h
- Merged PRs (30d)
- 19
Description
## Summary
An install job restored from an on-disk install marker that pauses **before its first download-started signal** ends up in a PAUSED state that cannot be restarted: `restart_failed()` silently no-ops, and the PAUSED marker rewrite erases the persisted resume metadata. The only escape is cancelling the install and re-importing from scratch.
Found during the adversarial review of #9432 (see the review body there); the bug predates that PR, but #9432's stricter 416 gate made the trigger reachable with ordinary local state.
## Mechanism
- `install_job.download_parts` is populated only by `_download_started_callback` (`model_install_default.py:1453`). An install restored by `_restore_incomplete_installs` (`model_install_default.py:214`) starts with an **empty** `download_parts` — the marker's file metadata goes to `job._resume_metadata` instead.
- The 416 resume-mismatch branch in `_do_download` pauses and raises **before** `_signal_job_started` (`download_default.py:434-453`). So if the *first* part of a resumed install hits it, no started callback ever fires and the install job's `download_parts` stays empty.
- `_download_cancelled_callback` correctly detects `resume_required` — but on the **multifile** job's parts (`model_install_default.py:1498`) — and sets the install PAUSED. So far so good.
- `restart_failed()` however reads the **install** job's `download_parts` (`model_install_default.py:625`) — empty → early return. The UI's restart action does nothing; the job is stuck PAUSED forever.
- Compounding it: the PAUSED marker rewrite (`_write_install_marker`, gate at `model_install_default.py:142`) writes `files: []`, permanently discarding the persisted `etag`/`canonical_url`/`expected_total_bytes`/`download_path` metadata that the next restore/resume would have used.
## Reproduction (manual)
1. Start a multi-file install (e.g. an HF diffusers model). Let the first file download partially, then quit the app. An install marker with per-file resume metadata is written to the install tmpdir.
2. While the app is stopped, append a few bytes of garbage to the first `.downloading` file in the tmpdir so it is **larger** than the remote file.
3. Start the app and resume the install. The server answers the `Range: bytes=N-` request with `416` and a mismatched `Content-Range: bytes */`, so the part pauses before any started signal ("Resume refused by server. Restart required."), and the install goes PAUSED.
4. Trigger the restart action (the route that calls `restart_failed`). Observe: it returns immediately, nothing is enqueued, the job stays PAUSED. The marker on disk now contains `"files": []`.
## Reproduction (test sketch)
The core assertion needs no HTTP at all:
```python
def test_restart_failed_with_empty_install_parts(mm2_installer) -> None:
# Simulate a marker-restored install: parts live only on the multifile job.
install_job = ModelInstallJob(id=1, source=URLModelSource(url=...), config_in=ModelRecordChanges(), local_path=tmpdir)
part = DownloadJob(source=..., dest=tmpdir)
part.resume_required = True
install_job._multifile_job = MultiFileDownloadJob(id=2, dest=tmpdir, download_parts=[part])
assert install_job.download_parts == [] # restored state
mm2_installer.restart_failed(install_job)
assert install_job.status == InstallStatus.PAUSED # FAILS today only in that nothing was enqueued/no state change
```
An end-to-end variant can drive it through `DownloadQueueService` with a `TestAdapter` returning `416` + `Content-Range: bytes */8` against a larger pre-seeded `.downloading` file (see the tests added in #9432 for the pattern), with the install job constructed via the marker-restore path.
## Suggested fix
- `restart_failed()` (and anything else that reads `job.download_parts` for liveness) should fall back to `job._multifile_job.download_parts` when the install job's list is empty.
- `_write_install_marker` should not overwrite a marker's existing `files` metadata with `[]` when the install job has no parts — skip the field or merge with what's on disk.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.