apache / apache/iceberg-python

`Table.scan(options=...)` silently ignores S3 properties for FileIO during data materialization (`to_pandas` / `to_arrow`)

Open
#3,166 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.1k
Forks
581
Avg merge
1d 17h
Merged PRs (30d)
77

Description

### Apache Iceberg version

0.11.0 (latest release)

### Please describe the bug 🐞

**Description:**
When passing an `options` dictionary to `Table.scan(options=...)`, the properties (such as `s3.connect-timeout` or `s3.request-timeout`) are accepted by the `DataScan` object but are never propagated to the underlying `FileIO` (e.g., `PyArrowFileIO`) when actual data materialization occurs via methods like `to_pandas()` or `to_arrow()`.
Because `ArrowScan` is initialized with the `FileIO` that was created during catalog instantiation (`table.io`), any S3-specific configurations provided at the scan level are completely bypassed. This causes operations reading numerous manifest files to fall back to the AWS C++ SDK default timeouts (often 10s-30s), leading to unexpected `curlCode: 28 (Timeout was reached)` errors even when generous timeouts are explicitly requested in the scan options.

**Steps to Reproduce:**
```
# 1. Load catalog with default (or no) S3 timeout properties

from pyiceberg.catalog import load_catalog
catalog = load_catalog("my_catalog", **{
"uri": "...",
"s3.endpoint": "..."
})
table = catalog.load_table("my_namespace.my_table")

# 2. Attempt to scan with explicit S3 timeout options

scan_options = {
"s3.connect-timeout": "600.0",
"s3.request-timeout": "600.0"
}

# The options are accepted by DataScan...

scan = table.scan(options=scan_options)
# 3. ...but completely ignored during S3 I/O operations (ArrowScan)
# This may throw a timeout error if RGW/S3 latency spikes, ignoring the 600s setting above.
df = scan.to_pandas()
```
### Expected Behavior:
Properties passed via options in Table.scan() should cascade down and either update or override the table.io.properties for the duration of the scan. Specifically, s3.* configurations should be respected by the underlying FileIO (e.g., PyArrowFileIO) when downloading manifest lists or data files.
### Actual Behavior:
The options passed to Table.scan() are stored in the DataScan instance but are never passed to the ArrowScan class or the FileIO instance during to_arrow() / to_pandas().
The ArrowScan relies entirely on the unmodified self.io object originally initialized by the catalog:
```
# In pyiceberg/table/__init__.py -> DataScan.to_arrow()
return ArrowScan(
self.table_metadata,
self.io, # <--- options are missing here!
self.projection(),
self.row_filter,
self.case_sensitive,
self.limit
).to_table(self.plan_files())
```
Environment:
- PyIceberg Version: 0.11.1 (and earlier)
- PyArrow Version: 18.0.0
- Storage: Ceph S3 / Rados Gateway (RGW)
### Suggested Fix:
Ideally, DataScan should merge its options with self.io.properties and instantiate a new FileIO, or ArrowScan should be modified to accept the scan-level options and apply them dynamically to the FileSystem instance before reading files.

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in pyiceberg/table/__init__.py at DataScan.to_arrow(), then trace how ArrowScan receives self.io during to_arrow() and to_pandas() materialization. Verify how scan options and FileIO properties are handled, and consider the existing scan/materialization test entry points. Done means scan-level S3 timeout options are honored by FileIO during manifest and data reads, with regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.