agentscope-ai / agentscope-ai/TuFT

Checkpoint-seeded runs lose their loaded weights on server restart

Đang mở
#140 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
enhancement
Ngôn ngữ chính
Python
Star
69
Fork
14
Merge trung bình
6 giờ 31 phút
Pull request đã merge (30 ngày)
13

Mô tả

## Scenario

With #139, a new training run can be seeded from another run's checkpoint (`load_weights` with a `tinker:///...` path). The load copies the source checkpoint's weights into the destination adapter **in backend memory only**: nothing is written under the destination run's checkpoint directory, and no `CheckpointRecord` is registered on the destination run. On disk, the checkpoint still belongs solely to the source run.

Server recovery rebuilds each run by loading *that run's own latest checkpoint* (`ServerState._restore_from_checkpoints` → `TrainingController.restore_from_checkpoint`). A run seeded only by `load_weights` has none, so after a restart:

- The destination run's adapter is recreated freshly initialized (with the correct LoRA geometry, since #139); the loaded weights are gone.
- All of the run's futures are marked failed with "No checkpoint found ... Please retry" — including the already-completed `load_weights` future, because `mark_futures_failed_after_checkpoint` is called with `checkpoint_future_id=None`.

The behavior is correct-but-lossy rather than silently wrong: a client that re-drives its failed operations in order re-issues the `load_weights` and recovers. But a client that only retries training ops resumes from fresh init, and any window where the source checkpoint has been deleted or made private makes the re-load impossible.

## Proposal

At `load_checkpoint` time, copy the source checkpoint's files into the destination run's own checkpoint directory and register the resulting `CheckpointRecord` on the destination run (persisted like any saved checkpoint).

This makes the destination run physically own what it loaded, restoring the invariant recovery already relies on — *a run with weights has a checkpoint* — with no special cases in the restore path:

- Restart recovery works unchanged: `restore_from_checkpoint` finds the copied checkpoint and loads it.
- No cross-run lifecycle coupling: deleting or privatizing the source checkpoint later cannot strand the destination run.
- Futures after the seed point are failed relative to a real checkpoint `future_id` instead of "no checkpoint", so completed work before the seed is not spuriously failed.

Cost: disk space and copy time per cross-run load, proportional to checkpoint size.

## Alternative considered

Persist a "seeded-from" pointer on the destination run record and re-load the source checkpoint during restore. Cheaper on disk, but it creates a dangling cross-run reference: `delete_checkpoint` has no idea another run depends on the checkpoint, and visibility changes (`public` flipped off) break the restore path. Rejected in favor of physical ownership.

## Pointers

- `TrainingController.load_checkpoint` — where the copy/registration would happen (`src/tuft/training_controller.py`)
- `TrainingController.restore_from_checkpoint` / `ServerState._restore_from_checkpoints` — the recovery path that would then work unchanged
- `TrainingController.save_checkpoint` — existing registration/persistence flow to mirror (`_save_training_run_with_checkpoint`)

Related: #18 (same-run save → restart → resume), #139 (cross-run load fix that makes this scenario reachable).

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.