drivendataorg / drivendataorg/cloudpathlib

Implement partial or streaming reads/writes (CloudFile abstraction)

Open
#9 5 comments 3 reactions 0 assignees View on GitHub
cloudfile enhancement
Dominant language
Python
Stars
628
Forks
88
Avg merge
17h 28m
Merged PRs (30d)
2

Description

Currently, if you want to read a file we download the whole file and then open it for reading. Most backends will have some way to do streaming reads. We may be able to improve the experience if we can do the same.

There may be some tricky bits with caching here. Can we stream to the user and then cache the streamed portion at the same time? Is there an async way that makes sense to do this?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.