Download optimal for device_per_stream batching method.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 206
- PR merge metrics
- No merged PRs in 30d
Description
Background:
Our data is quite large and varies in size. With a size limit of 100 MB, there will only be 8 or 9 samples per shard. I have noticed that many duplicate shards are downloaded on different nodes even with shuffle disabled. I would like your suggestions on how to avoid duplicate shards.
Additional Information that may be related:
batch_size: 4
shuffle: False
sampling_granularity: 1
num_canonical_nodes: Defaults to the number of physical nodes, which is 4 in our current case
batching_method: device_per_stream
Thoughts
When shuffle is disabled, I assume the shards can be evenly divided among different nodes. Perhaps we could implement something like sample_limit instead of size_limit and achieve that with proper configuration?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file or test is named in the issue. Start by reproducing the 100 MB, four-node configuration with device_per_stream, shuffle disabled, and the listed batching settings, then trace shard assignment and download behavior. Done requires an agreed approach and evidence that duplicate shard downloads are avoided.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100