box / box/box-python-sdk

Generator for chunks of content stream

Đang mở
#96 2 bình luận 0 reaction 1 người được giao Được @jmoldow nhận Xem trên GitHub
enhancement
Ngôn ngữ chính
Python
Star
459
Fork
223
Merge trung bình
8 giờ 57 phút
Pull request đã merge (30 ngày)
13

Mô tả

Today, there are two options for downloading a file:
- `File.content()`, which requests the entire file at once, and loads it all into memory.
- `File.download_to()`, which requests the file in chunks, and dumps it into a stream (`file` handle or another instance of `io.IOBase`).

The first option always loads the full contents into memory. The second option always loads the full contents into memory except in the case where the stream is a file handle. Additionally, both options always download the full file before handing control back to the caller.

There should be a third option for download, which only downloads and loads individual chunks at a time, and is a generator and therefore hands control back to the caller for each chunk.

I've implemented this both locally and in a comment on #93, and it looks something like this:

``` python
from contextlib import contextmanager, closing

class File(Item):
...

# A contextmanager so that we can force the HTTP stream to be closed when we're done with it.
@contextmanager
def content_chunks(self):
url = self.get_url('content')
with closing(self._session.get(url, expect_json_response=False, stream=True)) as box_response:
yield box_response.network_response.response_as_stream().stream(decode_content=True)

# We can redefine download_to() in terms of content_chunks().
def download_to(self, writeable_stream):
with self.content_chunks() as chunks:
for chunk in chunks:
writeable_stream.write(chunk)

# Usage example 1
with my_file.content_chunks() as chunks:
for chunk in chunks:
# Handle chunk

# Usage example 2
with my_file.content_chunks() as chunks:
# Handle chunks
```

This could be added as-is, but I haven't created a PR yet because I wanted to think more about #87. I was wondering if there might be a better way to expose streams in the abstract `Network` interface. Right now we have a generic `response_as_stream()` method on `NetworkResponse`, but it isn't really generic because:
- It requires `stream=True` to be passed to `request()` on `Network`, which isn't documented because it is specific to the requests library.
- After calling `response_as_stream()`, you must call its `stream()` method, which isn't documented because it is specific to the requests library.

We could just add this as-is, because it doesn't technically add any additional dependencies on requests that weren't already added via `download_to()`.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.