If you copy checkpoints from HOME to gcs they can get deleted

because of this line https://github.com/google/orbax/blob/c7a7fd48ff094ac4167d3fb1c10f8b6c8de32b3a/checkpoint/orbax/checkpoint/utils.py#L669

They don't have success file but are in GCS so orbax thinks its tmp and cleans it up.

I would suggest always or never saving COMMIT_SUCCESS file.

This is not blocking me (easy to just write extra commit success files once I found this) but it felt like I should report because it was very unexpected behavior and moving around checkpoints is super common.