huggingface / huggingface/alignment-handbook

Having troubles changing save_steps parameter for a resumed job.

Open
#195 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.7k
Forks
490
Avg merge
2m
Merged PRs (30d)
1

Description

I'm having next issue. Let say I'm starting a job

```
ACCELERATE_LOG_LEVEL=info accelerate launch --config_file recipes/accelerate_configs/deepspeed_zero3.yaml scripts/run_sft.py recipes/zephyr-7b-beta/sft/config_full.yaml
```

In `config_full.yaml` I would have `save_steps: 1000`. At some point I would realize that 1000 is too frequent of a step to save, so I stop the job, edit `config_full.yaml` to have `save_steps: 10000` and restart the job. The resume from checkpoint goes as planned, however I would still have checkpoints saved every 1000 steps (original parameter). What do I do wrong?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.