meta-pytorch / meta-pytorch/data
[Experiment] Make S3Handler.s3_read return a stream rather than bytes
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
Originally I was expecting the returned stream from S3handler is non-seekable stream. But, it turns out that the whole archive/files will be dumped into memory based on the implementation (I might be wrong about it then I need someone to validate it)
- C++ side: https://github.com/pytorch/data/blob/a435c7f5f7543e3614130eb2ecee520396b8efd2/torchdata/csrc/pybind/pybind.cpp#L28
- Python side: https://github.com/pytorch/data/blob/a435c7f5f7543e3614130eb2ecee520396b8efd2/torchdata/datapipes/iter/load/s3io.py#L135
And, that is the reason that the performance seems on parity with or withoutBytesIOin this issue.
In order to make it streaming, we need to have a way to pybind C++ stream IO to python, which is non-trivial. See a code example: https://github.com/CadQuery/OCP/blob/master/pystreambuf.h
Potentially this change would accelerate data preprocessing. But, it needs to be extensively benchmarked.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading torchdata/csrc/pybind/pybind.cpp around line 28 and torchdata/datapipes/iter/load/s3io.py around line 135 to verify whether S3Handler.s3_read materializes the archive in memory. Review the linked stream-binding example, then define benchmarks comparing current and streaming behavior. Done means a validated streaming approach with benchmark results showing its preprocessing impact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100