apache / apache/iceberg-python

Add option to pass storage config to polars lazy read

Open
#3,128 3 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.1k
Forks
581
Avg merge
1d 17h
Merged PRs (30d)
78

Description

### Feature Request / Improvement

I was not able to use the `table.to_polars()` method because there is no option to include a storage_options param which is mentioned in the [polars docs](https://docs.pola.rs/api/python/dev/reference/api/polars.scan_iceberg.html).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the implementation of table.to_polars() and trace how it invokes Polars' lazy read path. Check the Polars scan_iceberg documentation linked in the issue for the storage_options interface. Done means callers can provide storage_options through table.to_polars() and the option reaches the lazy read operation, with relevant tests updated or added.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.