google / google/scaaml

ResumeKTI does not work when examples_per_shard = 1

Open
#105 0 comments 0 reactions 1 assignee Claimed by @kralka View on GitHub
Dominant language
Python
Stars
207
Forks
62
Avg merge
12h 23m
Merged PRs (30d)
4

Description

When examples_per_shard == 1 and a restart happens there is no way to tell if the previous shard was already captured and saved or not. Also the scaaml.io.Dataset or DatasetFiller are not equipped to handle saving single example multiple times. To make this reliable there needs to be a way to say "shard saved".

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.