tensorflow / tensorflow/datasets

num_shards in SplitGenerator is ignored

Open
#1,405 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

Short description
Due to introduction of https://github.com/tensorflow/datasets/pull/650 the num_shards parameter in SplitGenerator is ignored and instead the TFRecord file writer will calculate an own number of shards in _get_number_shards.

Expected behavior
num_shards is honored if specified

Additional context
The number of shards determines the maximum number of parallel instances of models that can be trained in TF. So having it set to some "arbitrary" value without any control makes one depend on this magic value guessing algorithm. Furthermore the dataset_info.json reads inconsistent. E.g.:

      "numShards": "1",
      "shardLengths": [
        "128",
        "128",
        "128",
        "128",
        "128",
        "128",
        "128",
        "128",
        "128",
        "128",
        "128",
        "128",
        "128",
        "128",
        "128",
        "128"
      ],

For bonus points: Almost every comment/docu on the size of those shards in tensorflow_datasets/core/tfrecords_writer.py is wrong (as in: does not match the actual value): https://github.com/tensorflow/datasets/blob/ef0966d8cd57095c5f3e50bac153ebc53f5dfe34/tensorflow_datasets/core/tfrecords_writer.py#L39 https://github.com/tensorflow/datasets/blob/ef0966d8cd57095c5f3e50bac153ebc53f5dfe34/tensorflow_datasets/core/tfrecords_writer.py#L143 https://github.com/tensorflow/datasets/blob/ef0966d8cd57095c5f3e50bac153ebc53f5dfe34/tensorflow_datasets/core/tfrecords_writer.py#L38

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in tensorflow_datasets/core/tfrecords_writer.py, especially _get_number_shards and the comments around shard sizing. Trace how SplitGenerator.num_shards reaches the TFRecord writer, then verify that an explicitly provided value is honored and that dataset_info.json reports matching numShards and shardLengths.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.