apache / apache/iceberg-python

`Table.scan(options=...)` silently ignores S3 properties for FileIO during data materialization (`to_pandas` / `to_arrow`)

未關閉
#3,166 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
1.1k
分支
581
平均合併
1 天 17 小時
30 天內合併 PR
78

描述

### Apache Iceberg version

0.11.0 (latest release)

### Please describe the bug 🐞

**Description:**
When passing an `options` dictionary to `Table.scan(options=...)`, the properties (such as `s3.connect-timeout` or `s3.request-timeout`) are accepted by the `DataScan` object but are never propagated to the underlying `FileIO` (e.g., `PyArrowFileIO`) when actual data materialization occurs via methods like `to_pandas()` or `to_arrow()`.
Because `ArrowScan` is initialized with the `FileIO` that was created during catalog instantiation (`table.io`), any S3-specific configurations provided at the scan level are completely bypassed. This causes operations reading numerous manifest files to fall back to the AWS C++ SDK default timeouts (often 10s-30s), leading to unexpected `curlCode: 28 (Timeout was reached)` errors even when generous timeouts are explicitly requested in the scan options.

**Steps to Reproduce:**
```
# 1. Load catalog with default (or no) S3 timeout properties

from pyiceberg.catalog import load_catalog
catalog = load_catalog("my_catalog", **{
"uri": "...",
"s3.endpoint": "..."
})
table = catalog.load_table("my_namespace.my_table")

# 2. Attempt to scan with explicit S3 timeout options

scan_options = {
"s3.connect-timeout": "600.0",
"s3.request-timeout": "600.0"
}

# The options are accepted by DataScan...

scan = table.scan(options=scan_options)
# 3. ...but completely ignored during S3 I/O operations (ArrowScan)
# This may throw a timeout error if RGW/S3 latency spikes, ignoring the 600s setting above.
df = scan.to_pandas()
```
### Expected Behavior:
Properties passed via options in Table.scan() should cascade down and either update or override the table.io.properties for the duration of the scan. Specifically, s3.* configurations should be respected by the underlying FileIO (e.g., PyArrowFileIO) when downloading manifest lists or data files.
### Actual Behavior:
The options passed to Table.scan() are stored in the DataScan instance but are never passed to the ArrowScan class or the FileIO instance during to_arrow() / to_pandas().
The ArrowScan relies entirely on the unmodified self.io object originally initialized by the catalog:
```
# In pyiceberg/table/__init__.py -> DataScan.to_arrow()
return ArrowScan(
self.table_metadata,
self.io, # <--- options are missing here!
self.projection(),
self.row_filter,
self.case_sensitive,
self.limit
).to_table(self.plan_files())
```
Environment:
- PyIceberg Version: 0.11.1 (and earlier)
- PyArrow Version: 18.0.0
- Storage: Ceph S3 / Rados Gateway (RGW)
### Suggested Fix:
Ideally, DataScan should merge its options with self.io.properties and instantiate a new FileIO, or ArrowScan should be modified to accept the scan-level options and apply them dynamically to the FileSystem instance before reading files.

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

從 pyiceberg/table/__init__.py 中的 DataScan.to_arrow() 開始,接著追蹤在 to_arrow() 和 to_pandas() 具現化期間 ArrowScan 如何接收 self.io。確認 scan 選項和 FileIO 屬性的處理方式,並考慮現有的 scan/具現化測試入口。完成的定義是,scan 層級的 S3 逾時選項在讀取 manifest 和資料期間會由 FileIO 遵循,並具備回歸測試涵蓋。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
aws, python
領域
data-engineering
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
冷清
描述清晰度
基本清楚
新手友好度
55/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。