SFTP BULK - Airbyte zip file download issue
- Vorherrschende Sprache
- Python
- Sterne
- 22.1k
- Forks
- 5.3k
- PR-Merge-Kennzahlen
- PR-Kennzahlen ausstehend
Beschreibung
### Connector Name
SFTP Bulk
### Connector Version
v1.9.2
### What step the error happened?
During the sync
### Relevant information
I am using Airbyte to ingest monthly data files from a source SFTP server and transfer them to an Azure Blob Storage container. The source system publishes a ZIP file once per month, and the objective of this pipeline is to automatically download the ZIP file from SFTP and store it in Azure for downstream processing.
The Airbyte connection is configured as follows:
Source: SFTP Server
Destination: Azure Blob Storage
File Type: ZIP Archive
Frequency: Monthly
Use Case: Archive and ingest monthly data files into Azure for further processing and loading into Snowflake.
Issue Description:
While testing the Airbyte ingestion pipeline, I encountered an issue where Airbyte is unable to successfully download and process a ZIP file from the source location.
Observations:
The ZIP file size is approximately 375 MB, which is well below Airbyte's documented file size limit of 1.5 GB per file.
The connection is able to authenticate and access the source location successfully.
Other files are processed as expected.
The failure occurs specifically when Airbyte attempts to download or process this ZIP file.
Since the file size is within Airbyte's supported limits, the issue does not appear to be related to file size restrictions.
Impact:
The ingestion pipeline is unable to load the data contained in the ZIP archive.
Downstream processing is blocked until the file can be successfully retrieved and extracted.
Investigation Performed:
Verified that the file exists and is accessible at the source location.
Confirmed that the file size is within Airbyte's supported limits.
Validated source credentials and connection configuration.
Attempted multiple sync runs with the same result.
### Relevant log output
```shell
2026-06-10 10:59:08 source ERROR An error has occurred while getting file:
2026-06-10 10:59:08 source ERROR Error parsing record. This could be due to a mismatch between the config's file type and the actual file type, or because the file or record is not parseable. stream=UFCW_202605_FINAL_6_3_26 file=/myDownloads//UFCW_202605_FINAL.zip line_no=0 n_skipped=0
Stack Trace: Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/paramiko/channel.py", line 697, in recv
out = self.in_buffer.read(nbytes, self.timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/paramiko/buffered_pipe.py", line 154, in read
raise PipeTimeout()
paramiko.buffered_pipe.PipeTimeout
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/stream/default_file_based_stream.py", line 167, in read_records_from_slice
for file_record_data, file_reference in self._file_transfer.upload(
File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/file_types/file_transfer.py", line 30, in upload
raise ex
File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/file_types/file_transfer.py", line 25, in upload
yield stream_reader.upload(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/file_based_stream_reader.py", line 195, in upload
file.download_to_local_directory(local_file_path)
File "/airbyte/integration_code/source_sftp_bulk/stream_reader.py", line 86, in download_to_local_directory
self.sftp_client.sftp_connection.get(self.uri, local_file_path, callback=progress_handler)
File "/usr/local/lib/python3.11/site-packages/paramiko/sftp_client.py", line 840, in get
size = self.getfo(
^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/paramiko/sftp_client.py", line 799, in getfo
return self._transfer_with_callback(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/paramiko/sftp_client.py", line 678, in _transfer_with_callback
data = reader.read(32768)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/paramiko/file.py", line 203, in read
new_data = self._read(read_size)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/paramiko/sftp_file.py", line 182, in _read
data = self._read_prefetch(size)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/paramiko/sftp_file.py", line 162, in _read_prefetch
self.sftp._read_response()
File "/usr/local/lib/python3.11/site-packages/paramiko/sftp_client.py", line 887, in _read_response
t, data = self._read_packet()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/paramiko/sftp.py", line 212, in _read_packet
x = self._read_all(4)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/paramiko/sftp.py", line 196, in _read_all
x = self.sock.recv(n)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/paramiko/channel.py", line 699, in recv
raise socket.timeout()
TimeoutError
```
### Contribute
- [ ] Yes, I want to contribute
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/12948
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.