[Bug]: crash + resume (AR_RESUME_DIR, immediate saving) silently duplicates already-quantized modules as stale fp weights in the final artifact
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 175
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 99
Description
### Problem Description
## Evidence (opt-125m e2e)
crash after 2 blocks -> resume -> artifact contains 12 duplicated tensors = exactly the 6 quantized Linears of the 2 skipped blocks (each with both `qweight` and `.weight`). No warning is logged.
Qwen3.8-27B W4A16 artifact came out 47.4 GB instead of ~17.7 GB with 318/407 modules duplicated this way — silently.
## Root cause
`ShardWriter.finalize()`'s remaining-weights pass writes every tensor still present in the model tree whose exact NAME was not saved yet. Resumed-over blocks present plain `nn.Linear` modules (no wrapper), so their packed names (`qweight` etc., saved before the crash) never match the plain names — the name-exact dedup misses and the stale fp weight is written on top of the packed one.
### Reproduction Steps
1. run any quantization with immediate saving + `AR_RESUME_DIR=`
2. kill the process after >= 1 block completes (manifest marks it done)
3. rerun the same command with the same `AR_RESUME_DIR` -> run resumes, skipped blocks are not re-quantized
4. inspect the final artifact: every module of a skipped block exists TWICE — the packed tensors saved before the crash AND a plain fp `.weight` written at save time
### Environment Information
- auto-round: reproduced on a build based on upstream `main` @ `32369a69`; `ShardWriter.finalize()` is unchanged through current `main` @ `9e2650f4`. transformers 5.16.1, torch 2.11.0+cu128, immediate-saving path
### Error Logs
```shell
```
### Additional Context
_No response_
Contributor guide
Research direction
Start at ShardWriter.finalize() and trace its remaining-weights pass, then reproduce with immediate saving and AR_RESUME_DIR by stopping after at least one block and resuming. Inspect the final artifact for skipped-block tensors; done means resumed quantized modules are not accompanied by stale plain fp weights or duplicated tensor names.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100