Generator for chunks of content stream
- Dominant language
- Python
- Stars
- 459
- Forks
- 223
- Avg merge
- 8h 57m
- Merged PRs (30d)
- 13
Description
Today, there are two options for downloading a file:
- `File.content()`, which requests the entire file at once, and loads it all into memory.
- `File.download_to()`, which requests the file in chunks, and dumps it into a stream (`file` handle or another instance of `io.IOBase`).
The first option always loads the full contents into memory. The second option always loads the full contents into memory except in the case where the stream is a file handle. Additionally, both options always download the full file before handing control back to the caller.
There should be a third option for download, which only downloads and loads individual chunks at a time, and is a generator and therefore hands control back to the caller for each chunk.
I've implemented this both locally and in a comment on #93, and it looks something like this:
``` python
from contextlib import contextmanager, closing
class File(Item):
...
# A contextmanager so that we can force the HTTP stream to be closed when we're done with it.
@contextmanager
def content_chunks(self):
url = self.get_url('content')
with closing(self._session.get(url, expect_json_response=False, stream=True)) as box_response:
yield box_response.network_response.response_as_stream().stream(decode_content=True)
# We can redefine download_to() in terms of content_chunks().
def download_to(self, writeable_stream):
with self.content_chunks() as chunks:
for chunk in chunks:
writeable_stream.write(chunk)
# Usage example 1
with my_file.content_chunks() as chunks:
for chunk in chunks:
# Handle chunk
# Usage example 2
with my_file.content_chunks() as chunks:
# Handle chunks
```
This could be added as-is, but I haven't created a PR yet because I wanted to think more about #87. I was wondering if there might be a better way to expose streams in the abstract `Network` interface. Right now we have a generic `response_as_stream()` method on `NetworkResponse`, but it isn't really generic because:
- It requires `stream=True` to be passed to `request()` on `Network`, which isn't documented because it is specific to the requests library.
- After calling `response_as_stream()`, you must call its `stream()` method, which isn't documented because it is specific to the requests library.
We could just add this as-is, because it doesn't technically add any additional dependencies on requests that weren't already added via `download_to()`.
Contributor guide
Assessment
This issue has not been assessed yet.