invenia / invenia/Checkpoints.jl
Index checkpoint files incrementally
- Dominant language
- Julia
- Stars
- 12
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
[`index_checkpoint_files`](https://github.com/invenia/Checkpoints.jl/blob/92837561306a115df38aa388ad3e394372b41960/src/indexing.jl#L141) walks a given path to find checkpoint files, and organises the segments of each checkpoint path into tags.
One might want to call this many times on the same top-level checkpoint directory, to analyse checkpoint data while the program is running and new checkpoints are added. For example, if a checkpoint is made at regular time intervals, with the timestamp used as a tag.
If there are a lot of checkpoint files (e.g. 100s), walking the whole path becomes a big waste. One could index a subdirectory of the top-level checkpoint directory, but then not all of the tags would be found, because tags are part of the path.
Is there a way to update the checkpoint index incrementally, based on diffs in the file tree? For example, if I want to reindex per timestep, it only searches the checkpoints for that timestep and adds them to an existing index, but still knows all of the tags.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.