invoke-ai / invoke-ai/InvokeAI

Restored install that pauses before its first download-started signal cannot be restarted (restart_failed no-ops, marker loses resume metadata)

Open
#9,480 1 comment 0 reactions 1 assignee Claimed by @lstein View on GitHub
Dominant language
Python
Stars
28.2k
Forks
3k
Avg merge
6d 5h
Merged PRs (30d)
19

Description

## Summary

An install job restored from an on-disk install marker that pauses **before its first download-started signal** ends up in a PAUSED state that cannot be restarted: `restart_failed()` silently no-ops, and the PAUSED marker rewrite erases the persisted resume metadata. The only escape is cancelling the install and re-importing from scratch.

Found during the adversarial review of #9432 (see the review body there); the bug predates that PR, but #9432's stricter 416 gate made the trigger reachable with ordinary local state.

## Mechanism

- `install_job.download_parts` is populated only by `_download_started_callback` (`model_install_default.py:1453`). An install restored by `_restore_incomplete_installs` (`model_install_default.py:214`) starts with an **empty** `download_parts` — the marker's file metadata goes to `job._resume_metadata` instead.
- The 416 resume-mismatch branch in `_do_download` pauses and raises **before** `_signal_job_started` (`download_default.py:434-453`). So if the *first* part of a resumed install hits it, no started callback ever fires and the install job's `download_parts` stays empty.
- `_download_cancelled_callback` correctly detects `resume_required` — but on the **multifile** job's parts (`model_install_default.py:1498`) — and sets the install PAUSED. So far so good.
- `restart_failed()` however reads the **install** job's `download_parts` (`model_install_default.py:625`) — empty → early return. The UI's restart action does nothing; the job is stuck PAUSED forever.
- Compounding it: the PAUSED marker rewrite (`_write_install_marker`, gate at `model_install_default.py:142`) writes `files: []`, permanently discarding the persisted `etag`/`canonical_url`/`expected_total_bytes`/`download_path` metadata that the next restore/resume would have used.

## Reproduction (manual)

1. Start a multi-file install (e.g. an HF diffusers model). Let the first file download partially, then quit the app. An install marker with per-file resume metadata is written to the install tmpdir.
2. While the app is stopped, append a few bytes of garbage to the first `.downloading` file in the tmpdir so it is **larger** than the remote file.
3. Start the app and resume the install. The server answers the `Range: bytes=N-` request with `416` and a mismatched `Content-Range: bytes */`, so the part pauses before any started signal ("Resume refused by server. Restart required."), and the install goes PAUSED.
4. Trigger the restart action (the route that calls `restart_failed`). Observe: it returns immediately, nothing is enqueued, the job stays PAUSED. The marker on disk now contains `"files": []`.

## Reproduction (test sketch)

The core assertion needs no HTTP at all:

```python
def test_restart_failed_with_empty_install_parts(mm2_installer) -> None:
# Simulate a marker-restored install: parts live only on the multifile job.
install_job = ModelInstallJob(id=1, source=URLModelSource(url=...), config_in=ModelRecordChanges(), local_path=tmpdir)
part = DownloadJob(source=..., dest=tmpdir)
part.resume_required = True
install_job._multifile_job = MultiFileDownloadJob(id=2, dest=tmpdir, download_parts=[part])
assert install_job.download_parts == [] # restored state
mm2_installer.restart_failed(install_job)
assert install_job.status == InstallStatus.PAUSED # FAILS today only in that nothing was enqueued/no state change
```

An end-to-end variant can drive it through `DownloadQueueService` with a `TestAdapter` returning `416` + `Content-Range: bytes */8` against a larger pre-seeded `.downloading` file (see the tests added in #9432 for the pattern), with the install job constructed via the marker-restore path.

## Suggested fix

- `restart_failed()` (and anything else that reads `job.download_parts` for liveness) should fall back to `job._multifile_job.download_parts` when the install job's list is empty.
- `_write_install_marker` should not overwrite a marker's existing `files` metadata with `[]` when the install job has no parts — skip the field or merge with what's on disk.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.