agentscope-ai / agentscope-ai/TuFT

Checkpoint-seeded runs lose their loaded weights on server restart

Aberta
#140 0 comentários 0 reações 0 responsáveis Ver no GitHub
enhancement
Linguagem predominante
Python
Estrelas
69
Forks
14
Merge médio
6h 31min
PRs com merge (30d)
13

Descrição

## Scenario

With #139, a new training run can be seeded from another run's checkpoint (`load_weights` with a `tinker:///...` path). The load copies the source checkpoint's weights into the destination adapter **in backend memory only**: nothing is written under the destination run's checkpoint directory, and no `CheckpointRecord` is registered on the destination run. On disk, the checkpoint still belongs solely to the source run.

Server recovery rebuilds each run by loading *that run's own latest checkpoint* (`ServerState._restore_from_checkpoints` → `TrainingController.restore_from_checkpoint`). A run seeded only by `load_weights` has none, so after a restart:

- The destination run's adapter is recreated freshly initialized (with the correct LoRA geometry, since #139); the loaded weights are gone.
- All of the run's futures are marked failed with "No checkpoint found ... Please retry" — including the already-completed `load_weights` future, because `mark_futures_failed_after_checkpoint` is called with `checkpoint_future_id=None`.

The behavior is correct-but-lossy rather than silently wrong: a client that re-drives its failed operations in order re-issues the `load_weights` and recovers. But a client that only retries training ops resumes from fresh init, and any window where the source checkpoint has been deleted or made private makes the re-load impossible.

## Proposal

At `load_checkpoint` time, copy the source checkpoint's files into the destination run's own checkpoint directory and register the resulting `CheckpointRecord` on the destination run (persisted like any saved checkpoint).

This makes the destination run physically own what it loaded, restoring the invariant recovery already relies on — *a run with weights has a checkpoint* — with no special cases in the restore path:

- Restart recovery works unchanged: `restore_from_checkpoint` finds the copied checkpoint and loads it.
- No cross-run lifecycle coupling: deleting or privatizing the source checkpoint later cannot strand the destination run.
- Futures after the seed point are failed relative to a real checkpoint `future_id` instead of "no checkpoint", so completed work before the seed is not spuriously failed.

Cost: disk space and copy time per cross-run load, proportional to checkpoint size.

## Alternative considered

Persist a "seeded-from" pointer on the destination run record and re-load the source checkpoint during restore. Cheaper on disk, but it creates a dangling cross-run reference: `delete_checkpoint` has no idea another run depends on the checkpoint, and visibility changes (`public` flipped off) break the restore path. Rejected in favor of physical ownership.

## Pointers

- `TrainingController.load_checkpoint` — where the copy/registration would happen (`src/tuft/training_controller.py`)
- `TrainingController.restore_from_checkpoint` / `ServerState._restore_from_checkpoints` — the recovery path that would then work unchanged
- `TrainingController.save_checkpoint` — existing registration/persistence flow to mirror (`_save_training_run_with_checkpoint`)

Related: #18 (same-run save → restart → resume), #139 (cross-run load fix that makes this scenario reachable).

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Direção de pesquisa

Start with src/tuft/training_controller.py, especially TrainingController.load_checkpoint and the save_checkpoint/_save_training_run_with_checkpoint registration flow it should mirror. Read restore_from_checkpoint and ServerState._restore_from_checkpoints to understand the recovery invariant. Done means a cross-run load creates an owned checkpoint record/files for the destination run, so restart restore finds it without special restore logic.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
python
Domínio
backend, machine-learning
Tipo de issue
Bug
Dificuldade
4/5
Tempo estimado
3-5 dias
Status de atividade
Ativa
Clareza
Claramente especificada
Facilidade para iniciantes
58/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.