apache / apache/iceberg-python

Retry Behavior for SigV4Adapter in REST Catalog

未关闭
#3,008 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
1.1k
派生
581
平均合并
1 天 17 小时
30 天内合并 PR
78

描述

### Feature Request / Improvement

Hi team,

I’ve recently migrated to AWS S3 Tables and switched from using the GlueCatalog to the REST catalog. After updating the catalog configuration, everything works correctly in local, single‑process scenarios. However, I’m encountering intermittent failures when scaling out to multiple Dask workers making parallel requests.
Specifically, I’m seeing occasional `ThrottlingException` errors coming from AWS SigV4‑signed requests. Once throttling occurs, subsequent requests sometimes fail with:
`requests.exceptions.HTTPError: 403 Client Error`

My understanding is that throttled SigV4 signing attempts can lead to follow‑on request failures, resulting in unauthorized S3 operations. According to AWS’s recommendation for handling throttling on signed requests, retry configuration should be applied via botocore: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html

While reviewing the PyIceberg implementation, I noticed:

- The GlueCatalog sets reasonable default retry settings on the underlying boto session: https://github.com/apache/iceberg-python/blob/main/pyiceberg/catalog/glue.py#L331-L348
- The REST catalog, and specifically the SigV4Adapter, does not appear to configure any retry behavior by default: https://github.com/apache/iceberg-python/blob/main/pyiceberg/catalog/rest/__init__.py#L684-L694

This creates an inconsistency where switching from Glue to REST results in weaker retry behavior, which becomes visible under parallel load.

### Question / Proposal
Should the REST catalog align its default retry behavior with what GlueCatalog already applies?
At present, users can manually configure retry settings by supplying a custom botocore session via catalog properties, but I am yet to test if this works. It seems reasonable and more consistent for the REST catalog to provide safe defaults, especially since SigV4Adapter is now a common path for AWS S3 Tables.

Matching (or at least approaching) the GlueCatalog’s retry policy would provide the following benefits:

- Avoid intermittent throttling‑triggered failures in distributed workloads
- Improve parity between Glue and REST behavior
- Reduce the configuration burden on users switching to REST for AWS‑backed tables

Happy to discuss or test any proposed changes. Thanks for your work on the project!

贡献指南

这个仓库没有索引到贡献指南

调研方向

比较 pyiceberg/catalog/glue.py#L331-L348 中 GlueCatalog 的重试设置与 pyiceberg/catalog/rest/__init__.py#L684-L694 中 REST SigV4Adapter 的设置,然后阅读 issue 中链接的 botocore 重试指南。检查如何通过 catalog 属性提供自定义 botocore session。当 REST 请求具备已达成共识的安全默认重试策略,并且在 throttling 和并行工作负载下的行为与 Glue 一致时,即视为完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
aws, python
领域
api, cloud
Issue 类型
功能
难度
4/5
预计耗时
3-5 天
活跃度
冷清
描述清晰度
基本清楚
新手友好度
55/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。