agentscope-ai / agentscope-ai/TuFT
Checkpoint-seeded runs lose their loaded weights on server restart
- Linguagem predominante
- Python
- Estrelas
- 69
- Forks
- 14
- Merge médio
- 6h 31min
- PRs com merge (30d)
- 13
Descrição
## Scenario
With #139, a new training run can be seeded from another run's checkpoint (`load_weights` with a `tinker:///...` path). The load copies the source checkpoint's weights into the destination adapter **in backend memory only**: nothing is written under the destination run's checkpoint directory, and no `CheckpointRecord` is registered on the destination run. On disk, the checkpoint still belongs solely to the source run.
Server recovery rebuilds each run by loading *that run's own latest checkpoint* (`ServerState._restore_from_checkpoints` → `TrainingController.restore_from_checkpoint`). A run seeded only by `load_weights` has none, so after a restart:
- The destination run's adapter is recreated freshly initialized (with the correct LoRA geometry, since #139); the loaded weights are gone.
- All of the run's futures are marked failed with "No checkpoint found ... Please retry" — including the already-completed `load_weights` future, because `mark_futures_failed_after_checkpoint` is called with `checkpoint_future_id=None`.
The behavior is correct-but-lossy rather than silently wrong: a client that re-drives its failed operations in order re-issues the `load_weights` and recovers. But a client that only retries training ops resumes from fresh init, and any window where the source checkpoint has been deleted or made private makes the re-load impossible.
## Proposal
At `load_checkpoint` time, copy the source checkpoint's files into the destination run's own checkpoint directory and register the resulting `CheckpointRecord` on the destination run (persisted like any saved checkpoint).
This makes the destination run physically own what it loaded, restoring the invariant recovery already relies on — *a run with weights has a checkpoint* — with no special cases in the restore path:
- Restart recovery works unchanged: `restore_from_checkpoint` finds the copied checkpoint and loads it.
- No cross-run lifecycle coupling: deleting or privatizing the source checkpoint later cannot strand the destination run.
- Futures after the seed point are failed relative to a real checkpoint `future_id` instead of "no checkpoint", so completed work before the seed is not spuriously failed.
Cost: disk space and copy time per cross-run load, proportional to checkpoint size.
## Alternative considered
Persist a "seeded-from" pointer on the destination run record and re-load the source checkpoint during restore. Cheaper on disk, but it creates a dangling cross-run reference: `delete_checkpoint` has no idea another run depends on the checkpoint, and visibility changes (`public` flipped off) break the restore path. Rejected in favor of physical ownership.
## Pointers
- `TrainingController.load_checkpoint` — where the copy/registration would happen (`src/tuft/training_controller.py`)
- `TrainingController.restore_from_checkpoint` / `ServerState._restore_from_checkpoints` — the recovery path that would then work unchanged
- `TrainingController.save_checkpoint` — existing registration/persistence flow to mirror (`_save_training_run_with_checkpoint`)
Related: #18 (same-run save → restart → resume), #139 (cross-run load fix that makes this scenario reachable).
Guia de contribuição
Nenhum guia de contribuição indexado para este repositório
Direção de pesquisa
Start with src/tuft/training_controller.py, especially TrainingController.load_checkpoint and the save_checkpoint/_save_training_run_with_checkpoint registration flow it should mirror. Read restore_from_checkpoint and ServerState._restore_from_checkpoints to understand the recovery invariant. Done means a cross-run load creates an owned checkpoint record/files for the destination run, so restart restore finds it without special restore logic.
Escrita pelo modelo de indexação a partir do texto da issue.
Avaliação
- Stack de tecnologia
- python
- Domínio
- backend, machine-learning
- Tipo de issue
- Bug
- Dificuldade
- 4/5
- Tempo estimado
- 3-5 dias
- Status de atividade
- Ativa
- Clareza
- Claramente especificada
- Facilidade para iniciantes
- 58/100