SFTP Bulk: CSV header detection can silently skip initial records when remote reads do not honor rewind
- Ngôn ngữ chính
- Python
- Star
- 22.1k
- Fork
- 5.4k
- Merge trung bình
- 5 giờ
- Pull request đã merge (30 ngày)
- 671
Mô tả
## Summary
The SFTP Bulk connector can complete a CSV sync successfully while emitting fewer records than the source file contains.
The discrepancy occurs with **CSV Header Definition: From CSV**. Switching to **User Provided** headers and setting **Skip Rows Before Header: 1** restores the expected record count.
Source-code analysis and an offline reproduction indicate that the header-reading implementation depends on the SFTP server correctly honoring a rewind. If subsequent reads continue from the previous server-side position instead, initial data records can be silently skipped.
The workaround is confirmed against the affected source. The remote server's offset handling has not yet been independently verified.
## Versions investigated
- `source-sftp-bulk`: **1.9.2**
- Airbyte CDK: **7.5.1**, from the connector's dependency lockfile
- Paramiko: **3.4.0**
## Observed behavior
- The CSV's data-record count was independently verified using Python's CSV parser.
- Every record had the expected number of fields.
- The file contained no blank records or repeated headers.
- The source emitted fewer records than expected.
- The destination committed every emitted record.
- The sync completed without reported parsing failures.
- Providing explicit headers and skipping the physical header restored the full count.
## Suspected mechanism
With **From CSV** headers, the CDK CSV reader:
1. Reads the header from the remote file.
2. Calls `seek(0)`.
3. Skips the header line.
4. Parses the remaining records.
Paramiko reads ahead in **8,192-byte blocks** when reading lines. Its `seek(0)` clears the read buffer and causes the next SFTP READ request to use offset zero.
If the server or gateway ignores that offset and continues from its previous position:
- The buffered initial records are lost.
- The subsequent header-skipping step discards the remainder of the line at the block boundary.
- Parsing continues with valid later records, allowing the sync to report success.
## Offline reproduction
Using the actual Paramiko 3.4.0 file implementation, a local CSV, and a simulated SFTP backend:
| Backend behavior | Header mode | Result |
|---|---|---|
| Honors requested offsets | From CSV | All records emitted |
| Ignores requested offsets | From CSV | Initial records silently omitted |
| Ignores requested offsets | User Provided, skip one line | All records emitted |
The omitted records in the reproduction were exactly the initial records consumed by the read-ahead and subsequent line skip.
This does **not** imply that all SFTP servers are affected.
## Expected behavior
The connector should either:
- Emit all CSV data records; or
- Fail clearly when the remote file cannot be rewound reliably.
A successful sync should not silently omit initial records.
## Workaround
- Set **CSV Header Definition** to **User Provided**.
- Supply the column names in file order.
- Set **Skip Rows Before Header** to **1**.
- Leave **Skip Rows After Header** at **0**.
This avoids the header-detection rewind.
## Suggested improvement
Consider parsing the header and data in a single forward-only pass, preserving the existing buffer instead of rewinding the remote file.
If rewinding remains necessary, consider validating that the rewind succeeded and surfacing an explicit error otherwise.
## Relevant code
- [CDK CSV parser](https://github.com/airbytehq/airbyte-python-cdk/blob/v7.5.1/airbyte_cdk/sources/file_based/file_types/csv_parser.py)
- [Paramiko SFTP read and seek implementation](https://github.com/paramiko/paramiko/blob/3.4.0/paramiko/sftp_file.py)
- [Paramiko buffered line reader](https://github.com/paramiko/paramiko/blob/3.4.0/paramiko/file.py)
Hướng dẫn đóng góp
Hướng nghiên cứu
Start by reading the CDK CSV parser at airbyte_cdk/sources/file_based/file_types/csv_parser.py, then inspect Paramiko's sftp_file.py and file.py implementations for the read and rewind behavior described. Reproduce the offset-ignoring backend case from the issue and determine whether the completed change emits every record or fails clearly when rewind cannot be trusted.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- python
- Lĩnh vực
- backend, data-engineering
- Loại issue
- Lỗi
- Độ khó
- 4/5
- Thời gian dự kiến
- 3-5 ngày
- Mức độ hoạt động
- Sôi nổi
- Độ rõ ràng
- Khá rõ ràng
- Mức phù hợp với người mới
- 58/100