googleapis / googleapis/google-cloud-python

Support in-memory data for Transfer Manager download_chunks_concurrently and upload_chunks_concurrently

Offen
#15,975 2 Kommentare 0 Reaktionen 1 zugewiesene Person Zugewiesen an @chandra-siri Auf GitHub ansehen
api: storage type: feature request
Vorherrschende Sprache
Python
Sterne
5.4k
Forks
1.8k
Ø Merge
3 T. 4 Std.
Gemergte PRs (30 T.)
122

Beschreibung

A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

* Today Transfer Manager's `download_chunks_concurrently` and `upload_chunks_concurrently` require you to provide a filename. Unfortunately I have bytes that are already in-memory and may not fit on disk (or it may be slow to persist it to disk) and can't use Transfer Manager as a result.

A clear and concise description of what you want to happen.

* Provide functions that support the same interface as `download_chunks_concurrently` and `upload_chunks_concurrently` but take a file-like object instead of a filename
* Or, for write, provide a similar function that returns a stream to the caller, and then writes to the stream are uploaded via multipart upload as needed (not sure if there's any realistic similar thing here for reads)
* It's ok if this functionality can only be threaded and not use multi-processing since in practice we still see a large speed increase from threading

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.