meta-pytorch / meta-pytorch/data
S3FileLoaderIterDataPipe buffer_size
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
📚 The doc issue
The default for S3 buffer size is 128 MB - or 128 * (1024**2)
https://github.com/pytorch/data/blob/a5b4720dece60565788ac4c9a85e01719188b28e/torchdata/csrc/pybind/S3Handler/S3Handler.cpp#L15
The example for S3FileLoaderIterDataPipe uses a buffer_size of 256.
https://github.com/pytorch/data/blob/a5b4720dece60565788ac4c9a85e01719188b28e/torchdata/datapipes/iter/load/s3io.py#L154
Using a 256 bytes buffer degrades performance and allows the assumption buffer_size is provided in mbytes, as the example would double the 128 mbyte default.
Suggest a potential alternative/fix
document buffer_size to be in bytes and have the example use 256 * (1024**2) as value.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with torchdata/datapipes/iter/load/s3io.py and compare its S3FileLoaderIterDataPipe example with the default in torchdata/csrc/pybind/S3Handler/S3Handler.cpp. Update the documentation to state that buffer_size is in bytes and change the example value to 256 * (1024**2); done means the example no longer implies megabytes or uses a 256-byte buffer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- data, documentation
- Issue type
- Documentation
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100