ResumeKTI does not work when examples_per_shard = 1
Open
- Dominant language
- Python
- Stars
- 207
- Forks
- 62
- Avg merge
- 12h 23m
- Merged PRs (30d)
- 4
Description
When examples_per_shard == 1 and a restart happens there is no way to tell if the previous shard was already captured and saved or not. Also the scaaml.io.Dataset or DatasetFiller are not equipped to handle saving single example multiple times. To make this reliable there needs to be a way to say "shard saved".
Contributor guide
Assessment
This issue has not been assessed yet.