drivendataorg / drivendataorg/cloudpathlib
Implement partial or streaming reads/writes (CloudFile abstraction)
Open
cloudfile
enhancement
- Dominant language
- Python
- Stars
- 628
- Forks
- 88
- Avg merge
- 17h 28m
- Merged PRs (30d)
- 2
Description
Currently, if you want to read a file we download the whole file and then open it for reading. Most backends will have some way to do streaming reads. We may be able to improve the experience if we can do the same.
There may be some tricky bits with caching here. Can we stream to the user and then cache the streamed portion at the same time? Is there an async way that makes sense to do this?
Contributor guide
Assessment
This issue has not been assessed yet.