google / google/flax

Inconsistency in `keep` argument for save_checkpoint and save_checkpoint_multiprocess

Open
#3,192 1 comment 4 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
7.3k
Forks
833
Avg merge
5h 11m
Merged PRs (30d)
5

Description

There seems to be an inconsistency in `keep` argument for save_checkpoint and save_checkpoint_multiprocess

### System information
Internal

### Problem you have encountered:
I am trying to migrate to using the Orbax Checkpointer w flax.save_checkpoint_multiprocess and have noticed an inconsistency in behavior between
1. current and past behavior w flax checkpointing
2. save_checkpoint and save_checkpoint_multiprocess

The method save_checkpoint calls save first for the current checkpoint and then removes excessive checkpoints, but the save_checkpoint_multiprocess calls remove first and then saves the current checkpoint.

### What you expected to happen:
In the past when specifying keep=N for the save method, the number of checkpoints at the end of the save_checkpoint call was equal to N. Currently when calling save_checkpoint_multiprocess.py the number of checkpoints after the save is equal to N+1.

### Logs, error messages, etc:
Internal

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.