invenia / invenia/Checkpoints.jl

Index checkpoint files incrementally

Open
#27 1 comment 0 reactions 0 assignees View on GitHub
enhancement question
Dominant language
Julia
Stars
12
Forks
4
PR merge metrics
No merged PRs in 30d

Description

[`index_checkpoint_files`](https://github.com/invenia/Checkpoints.jl/blob/92837561306a115df38aa388ad3e394372b41960/src/indexing.jl#L141) walks a given path to find checkpoint files, and organises the segments of each checkpoint path into tags.

One might want to call this many times on the same top-level checkpoint directory, to analyse checkpoint data while the program is running and new checkpoints are added. For example, if a checkpoint is made at regular time intervals, with the timestamp used as a tag.

If there are a lot of checkpoint files (e.g. 100s), walking the whole path becomes a big waste. One could index a subdirectory of the top-level checkpoint directory, but then not all of the tags would be found, because tags are part of the path.

Is there a way to update the checkpoint index incrementally, based on diffs in the file tree? For example, if I want to reindex per timestep, it only searches the checkpoints for that timestep and adds them to an existing index, but still knows all of the tags.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.