apache / apache/libcloud

Upload/Download bug with Azure Blobs

Aperta
#1,475 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub
api: storage stale
Lingua principale
Python
Stelle
2.1k
Fork
931
Merge medio
1g 2h
PR unite (30g)
4

Descrizione

## Summary
We are facing some issues interacting with Azure Blobs. Whether it is for uploading or downloading, it happens quite often that we get an error like below. This problem frequency increases dramatically with the size of the file we are interacting. For a large file ~100Go, it is almost impossible to run anything without encountering this. Is it something known?

## Detailed Information
Libcloud version: apache-libcloud==3.1.0
python version: 3.7.4
OS: Ubuntu
Within the dask docker image

Stacktrace:

`[2020-07-17 12:25:46] ERROR - prefect.TaskRunner | Unexpected error: ChunkedEncodingError(ProtocolError('Connection broken: OSError("(104, \'ECONNRESET\')")', OSError("(104, 'ECONNRESET')")))
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 297, in recv_into
return self.connection.recv_into(*args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1822, in recv_into
self._raise_ssl_error(self._ssl, result)
File "/opt/conda/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1639, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (104, 'ECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/urllib3/response.py", line 360, in _error_catcher
yield
File "/opt/conda/lib/python3.7/site-packages/urllib3/response.py", line 442, in read
data = self._fp.read(amt)
File "/opt/conda/lib/python3.7/http/client.py", line 457, in read
n = self.readinto(b)
File "/opt/conda/lib/python3.7/http/client.py", line 501, in readinto
n = self.fp.readinto(b)
File "/opt/conda/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
File "/opt/conda/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 302, in recv_into
raise SocketError(str(e))
OSError: (104, 'ECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/requests/models.py", line 750, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/opt/conda/lib/python3.7/site-packages/urllib3/response.py", line 494, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/opt/conda/lib/python3.7/site-packages/urllib3/response.py", line 459, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/opt/conda/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/opt/conda/lib/python3.7/site-packages/urllib3/response.py", line 378, in _error_catcher
raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: OSError("(104, \'ECONNRESET\')")', OSError("(104, 'ECONNRESET')"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 943, in get_task_run_state
self.task.run, timeout=self.task.timeout, **raw_inputs
File "/opt/conda/lib/python3.7/site-packages/prefect/utilities/executors.py", line 182, in timeout_handler
return fn(*args, **kwargs)
File "xxxxx, line 127, in run
video_file.name
File "xxxxx", line 116, in xxxx
for k, line in enumerate(xxxx):
File "/opt/conda/lib/python3.7/site-packages/libcloud/utils/files.py", line 69, in read_in_chunks
chunk = b(get_data(*args))
File "/opt/conda/lib/python3.7/site-packages/requests/models.py", line 753, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: OSError("(104, \'ECONNRESET\')")', OSError("(104, 'ECONNRESET')"))`

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Start with libcloud/utils/files.py at read_in_chunks and trace how Azure blob uploads and downloads use requests while reproducing the ECONNRESET failure with a large file. Compare the observed behavior with the stack trace in requests/models.py; done means identifying the failure boundary and documenting the cause or limitation, since the issue does not specify a proposed fix.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
azure, python
Ambito
cloud
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.