box / box/box-python-sdk

Generator for chunks of content stream

Abierto
#96 2 comentarios 0 reacciones 1 asignado Reclamado por @jmoldow Ver en GitHub
enhancement
Lenguaje dominante
Python
Estrellas
459
Forks
223
Merge medio
8 h 57 min
PR fusionados (30 d)
13

Descripción

Today, there are two options for downloading a file:
- `File.content()`, which requests the entire file at once, and loads it all into memory.
- `File.download_to()`, which requests the file in chunks, and dumps it into a stream (`file` handle or another instance of `io.IOBase`).

The first option always loads the full contents into memory. The second option always loads the full contents into memory except in the case where the stream is a file handle. Additionally, both options always download the full file before handing control back to the caller.

There should be a third option for download, which only downloads and loads individual chunks at a time, and is a generator and therefore hands control back to the caller for each chunk.

I've implemented this both locally and in a comment on #93, and it looks something like this:

``` python
from contextlib import contextmanager, closing

class File(Item):
...

# A contextmanager so that we can force the HTTP stream to be closed when we're done with it.
@contextmanager
def content_chunks(self):
url = self.get_url('content')
with closing(self._session.get(url, expect_json_response=False, stream=True)) as box_response:
yield box_response.network_response.response_as_stream().stream(decode_content=True)

# We can redefine download_to() in terms of content_chunks().
def download_to(self, writeable_stream):
with self.content_chunks() as chunks:
for chunk in chunks:
writeable_stream.write(chunk)

# Usage example 1
with my_file.content_chunks() as chunks:
for chunk in chunks:
# Handle chunk

# Usage example 2
with my_file.content_chunks() as chunks:
# Handle chunks
```

This could be added as-is, but I haven't created a PR yet because I wanted to think more about #87. I was wondering if there might be a better way to expose streams in the abstract `Network` interface. Right now we have a generic `response_as_stream()` method on `NetworkResponse`, but it isn't really generic because:
- It requires `stream=True` to be passed to `request()` on `Network`, which isn't documented because it is specific to the requests library.
- After calling `response_as_stream()`, you must call its `stream()` method, which isn't documented because it is specific to the requests library.

We could just add this as-is, because it doesn't technically add any additional dependencies on requests that weren't already added via `download_to()`.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.