dropbox / dropbox/dropbox-sdk-python
Allow chunked uploads (and maybe even streaming uploads)
Ninguém assumiu esta issue ainda.
- Linguagem predominante
- Python
- Estrelas
- 983
- Forks
- 331
- Merge médio
- 3min
- PRs com merge (30d)
- 10
Descrição
Why is this feature valuable to you? Does it solve a problem you're having?
The requests library allows both streaming and chunked uploads (see https://requests.readthedocs.io/en/latest/user/advanced/#streaming-uploads and https://requests.readthedocs.io/en/latest/user/advanced/#chunk-encoded-requests). This has two benefits:
- It is sufficient to only load small parts of a file into memory before upload.
- It is possible to limit bandwidth usage by using a generator that provides chunks at a limited rate.
The Dropbox API of course already requires upload sessions (files/upload_session_start, files/upload_session_append and files/upload_session_finish) to upload files > 150 MB. However, this approach by itself does not replace chunked or streaming uploads because:
- The request body should be ideally >= 4 MB to reduce the total number of API calls (both for efficiency and to not exhaust data transport API call limits).
- Bandwidth control will be very coarse when performed on chunks of 4 MB compared for example 2 kB.
- Memory usage will still be larger compared to 1 kB or 2 kB chunks, especially for parallel uploads.
Describe the solution you'd like
Requests supports streaming uploads by passing a file-like object as the request body and chunked uploads by passing a generator as the request body. However, the Python SDK explicitly prevents both by requiring the request body to be of type bytes:
It would be good to either completely drop this limitation, with appropriate warnings in the doc string, or at least allow chunked uploads (where requests handles retry / rewind logic) even when disallowing streaming uploads.
Describe alternatives you've considered
Not at present.
Guia de contribuição
Primeiros passos
- Leia a issue inteira e depois o guia de contribuição do projeto.
- Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
- Faça um fork do repositório e trabalhe em uma branch.
- Abra um pull request que referencie o número da issue.
Direção de pesquisa
Comece em dropbox_client.py por volta das linhas 533-539, onde os corpos das requisições são restritos a bytes, e compare esse comportamento com a documentação vinculada do Requests sobre streaming e uploads em partes. Determine como o SDK deve aceitar objetos semelhantes a arquivos ou geradores, documentando também o comportamento de novas tentativas e retrocesso; considera-se concluído quando as formas de upload escolhidas forem suportadas sem a restrição atual de aceitar apenas bytes.
Escrita pelo modelo de indexação a partir do texto da issue.
Avaliação
- Stack de tecnologia
- python
- Domínio
- api
- Tipo de issue
- Funcionalidade
- Dificuldade
- 4/5
- Tempo estimado
- 3-5 dias
- Status de atividade
- Pouca atividade
- Clareza
- Razoavelmente clara
- Facilidade para iniciantes
- 50/100