airbytehq / airbytehq/airbyte

SFTP Bulk: CSV header detection can silently skip initial records when remote reads do not honor rewind

Đang mở
#85,837 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
autoteam community connectors/source/file connectors/source/sftp-bulk team/use
Ngôn ngữ chính
Python
Star
22.1k
Fork
5.4k
Merge trung bình
5 giờ
Pull request đã merge (30 ngày)
671

Mô tả

## Summary

The SFTP Bulk connector can complete a CSV sync successfully while emitting fewer records than the source file contains.

The discrepancy occurs with **CSV Header Definition: From CSV**. Switching to **User Provided** headers and setting **Skip Rows Before Header: 1** restores the expected record count.

Source-code analysis and an offline reproduction indicate that the header-reading implementation depends on the SFTP server correctly honoring a rewind. If subsequent reads continue from the previous server-side position instead, initial data records can be silently skipped.

The workaround is confirmed against the affected source. The remote server's offset handling has not yet been independently verified.

## Versions investigated

- `source-sftp-bulk`: **1.9.2**
- Airbyte CDK: **7.5.1**, from the connector's dependency lockfile
- Paramiko: **3.4.0**

## Observed behavior

- The CSV's data-record count was independently verified using Python's CSV parser.
- Every record had the expected number of fields.
- The file contained no blank records or repeated headers.
- The source emitted fewer records than expected.
- The destination committed every emitted record.
- The sync completed without reported parsing failures.
- Providing explicit headers and skipping the physical header restored the full count.

## Suspected mechanism

With **From CSV** headers, the CDK CSV reader:

1. Reads the header from the remote file.
2. Calls `seek(0)`.
3. Skips the header line.
4. Parses the remaining records.

Paramiko reads ahead in **8,192-byte blocks** when reading lines. Its `seek(0)` clears the read buffer and causes the next SFTP READ request to use offset zero.

If the server or gateway ignores that offset and continues from its previous position:

- The buffered initial records are lost.
- The subsequent header-skipping step discards the remainder of the line at the block boundary.
- Parsing continues with valid later records, allowing the sync to report success.

## Offline reproduction

Using the actual Paramiko 3.4.0 file implementation, a local CSV, and a simulated SFTP backend:

| Backend behavior | Header mode | Result |
|---|---|---|
| Honors requested offsets | From CSV | All records emitted |
| Ignores requested offsets | From CSV | Initial records silently omitted |
| Ignores requested offsets | User Provided, skip one line | All records emitted |

The omitted records in the reproduction were exactly the initial records consumed by the read-ahead and subsequent line skip.

This does **not** imply that all SFTP servers are affected.

## Expected behavior

The connector should either:

- Emit all CSV data records; or
- Fail clearly when the remote file cannot be rewound reliably.

A successful sync should not silently omit initial records.

## Workaround

- Set **CSV Header Definition** to **User Provided**.
- Supply the column names in file order.
- Set **Skip Rows Before Header** to **1**.
- Leave **Skip Rows After Header** at **0**.

This avoids the header-detection rewind.

## Suggested improvement

Consider parsing the header and data in a single forward-only pass, preserving the existing buffer instead of rewinding the remote file.

If rewinding remains necessary, consider validating that the rewind succeeded and surfacing an explicit error otherwise.

## Relevant code

- [CDK CSV parser](https://github.com/airbytehq/airbyte-python-cdk/blob/v7.5.1/airbyte_cdk/sources/file_based/file_types/csv_parser.py)
- [Paramiko SFTP read and seek implementation](https://github.com/paramiko/paramiko/blob/3.4.0/paramiko/sftp_file.py)
- [Paramiko buffered line reader](https://github.com/paramiko/paramiko/blob/3.4.0/paramiko/file.py)

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Start by reading the CDK CSV parser at airbyte_cdk/sources/file_based/file_types/csv_parser.py, then inspect Paramiko's sftp_file.py and file.py implementations for the read and rewind behavior described. Reproduce the offset-ignoring backend case from the issue and determine whether the completed change emits every record or fails clearly when rewind cannot be trusted.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python
Lĩnh vực
backend, data-engineering
Loại issue
Lỗi
Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
58/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.