tensorflow / tensorflow/datasets
num_shards in SplitGenerator is ignored
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
Short description
Due to introduction of https://github.com/tensorflow/datasets/pull/650 the num_shards parameter in SplitGenerator is ignored and instead the TFRecord file writer will calculate an own number of shards in _get_number_shards.
Expected behavior
num_shards is honored if specified
Additional context
The number of shards determines the maximum number of parallel instances of models that can be trained in TF. So having it set to some "arbitrary" value without any control makes one depend on this magic value guessing algorithm. Furthermore the dataset_info.json reads inconsistent. E.g.:
"numShards": "1",
"shardLengths": [
"128",
"128",
"128",
"128",
"128",
"128",
"128",
"128",
"128",
"128",
"128",
"128",
"128",
"128",
"128",
"128"
],
For bonus points: Almost every comment/docu on the size of those shards in tensorflow_datasets/core/tfrecords_writer.py is wrong (as in: does not match the actual value): https://github.com/tensorflow/datasets/blob/ef0966d8cd57095c5f3e50bac153ebc53f5dfe34/tensorflow_datasets/core/tfrecords_writer.py#L39 https://github.com/tensorflow/datasets/blob/ef0966d8cd57095c5f3e50bac153ebc53f5dfe34/tensorflow_datasets/core/tfrecords_writer.py#L143 https://github.com/tensorflow/datasets/blob/ef0966d8cd57095c5f3e50bac153ebc53f5dfe34/tensorflow_datasets/core/tfrecords_writer.py#L38
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in tensorflow_datasets/core/tfrecords_writer.py, especially _get_number_shards and the comments around shard sizing. Trace how SplitGenerator.num_shards reaches the TFRecord writer, then verify that an explicitly provided value is honored and that dataset_info.json reports matching numShards and shardLengths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100