THUDM / THUDM/slime

Latest Slime Could Not Continue From Checkpoint

Open
#686 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.5k
Forks
1.3k
Avg merge
5h 36m
Merged PRs (30d)
22

Description

 File "/usr/local/lib/python3.12/dist-packages/torch/serialization.py", line 1529, in load
    raise pickle.UnpicklingError(_get_wo_message(str(e))) from None
_pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, do those steps only if you trust the source of the checkpoint.
        (1) In PyTorch 2.6, we changed the default value of the weights_only argument in torch.load from False to True. Re-running torch.load with weights_only set to False will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source.
        (2) Alternatively, to load with weights_only=True please check the recommended steps in the following error message.
        WeightsUnpickler error: Unsupported global: GLOBAL slime.utils.eval_config.EvalDatasetConfig was not an allowed global by default. Please use torch.serialization.add_safe_globals([slime.utils.eval_config.EvalDatasetConfig]) or the torch.serialization.safe_globals([slime.utils.eval_config.EvalDatasetConfig]) context manager to allowlist this global if you trust this class/function.

Whenever I want to restart from a past checkpoint. According to this error message, this is caused by this PR: https://github.com/THUDM/slime/pull/585.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the checkpoint-loading path that reaches torch.serialization.load and review PR 585 for the change associated with the reported behavior. Reproduce restarting from a past checkpoint with the current setup, then verify that checkpoint loading completes successfully without the reported UnpicklingError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.