google / google/orbax

Custom TypeHandler and "No per-process OCDBT checkpoint subdirs" warning

Open
#1,326 4 comments 1 reaction 0 assignees View on GitHub
checkpoint type:bug
Dominant language
Python
Stars
535
Forks
101
Avg merge
1d 13h
Merged PRs (30d)
23

Description

Hello,

I've recently created a custom type handler.
Using it, and running on a single process I see the following warning

```python
WARNING:absl:[process=0][thread=async_save_18] Skipping merge of OCDBT checkpoints: No per-process OCDBT checkpoint subdirs found in /tmp/ckp3/115.orbax-checkpoint-tmp-136/callbacks.orbax-checkpoint-tmp-139,
```

The custom type handler I wrote serialises some custom type containing some numpy arrays, and if I was to run this across multiple processes I'd like only the master process to serialise the data (which is basically replicated).

How can I silence this warning? Did I forgot to define something?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.