apache / apache/iceberg-python

Discrepency in the fetched table properties

未关闭
#3,247 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
1.1k
派生
581
平均合并
1 天 17 小时
30 天内合并 PR
78

描述

### Apache Iceberg version

None

### Please describe the bug 🐞

I am using Apache Polaris as catalog and when i load the table using pyiceberg, the response has config that is different from what i get when i use Polaris CLI and not only that when i use the file system from this table, i am getting `Permission Denied` errors, suggesting the credentials in the table are wrong.

Details:

```
polaris --profile root tables get --catalog "senec_catalog" --namespace "Senec.Stage.Bronze.Ampace_V3LFP" "BmsAmpaceV3ModuleMeasurement"
```
returns a table with the config property as follows:
```
, "config": {"s3.path-style-access": "true", "s3.endpoint": "http://localhost:9008"}
```

however, the rest api cal inside `catalog.load_table(table_identifier)` has a response that contains the config that looks like this:
```
{'s3.path-style-access': 'true', 's3.access-key-id': 'NFAIYHBUDHR1VBDHR8G0', 's3.secret-access
-key': 'L4Q_0W4We1ClcHsg8t0A5RcazPu4Xx1TR9P5-fbn', 's3.session-token': '***', 'client.refresh-credentials-endpoint': 'v1/senec_catalog/namespaces/Senec%1FStage%1FBronze%1FAmpace_V3LFP/tables/BmsAmpaceV3ModuleMeasurement/credentials', '
expiration-time': '1776355429000', 's3.endpoint': 'http://localhost:9008', 's3.session-token-e
xpires-at-ms': '1776355429000'}
```
which would have been okay but using a filesystem created using the io from this table is always resulting in permission errors, so i have to manually set the table properties like so:

```
self.tbl.io.properties["s3.endpoint"] = self.catalog.properties["s3.endpoint"]
self.tbl.io.properties["s3.endpoint"] = self.catalog.properties.get("s3.endpoint")
self.tbl.io.properties["s3.access-key-id"] = self.catalog.properties.get("s3.access-key-id")
self.tbl.io.properties["s3.secret-access-key"] = self.catalog.properties.get("s3.secret-access-key")
self.tbl.io.properties.pop("s3.session-token", None)
```

Could anyone tell me why is this the case? This is related to #1958 as i am trying to use the filesystem to delete the orphan files.

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

贡献指南

这个仓库没有索引到贡献指南

调研方向

Start at catalog.load_table(table_identifier) and trace how the returned table config is passed into the filesystem created from table.io. Reproduce the discrepancy with the shown Polaris CLI and REST calls, then verify that the filesystem uses the returned endpoint and credentials without manual property mutation. Add a regression test if the repository has coverage for catalog-loaded table properties.

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
databases
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
冷清
描述清晰度
基本清楚
新手友好度
48/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。