apache / apache/iceberg-python

FsspecFileIO remote signing (S3V4RestSigner) never fires on modern aiobotocore — S3 requests sent unsigned

未关闭
#3,625 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
1.1k
派生
581
平均合并
1 天 17 小时
30 天内合并 PR
78

描述

### Apache Iceberg version

0.11.1 (latest release), also reproduced on 0.11.0

### Please describe the bug 🐞

`FsspecFileIO` remote request signing (`s3.signer=S3V4RestSigner`) silently does **not** sign S3 requests when running against a recent `aiobotocore`/`botocore`. The `before-sign.s3` handler that `_s3()` registers never fires for the actual S3 operation, so the request is sent to S3 **unsigned** and S3 rejects it with:

```
InvalidRequest: The authorization mechanism you have provided is not supported. Please use Signature Version 4.
```

This breaks the standard Iceberg REST "remote signing" flow (catalog `/v1/config` sets `s3.signer=S3V4RestSigner`, client holds no AWS credentials, each S3 request is signed by the REST signer endpoint) for anyone on a current dependency set.

### Root cause

In `pyiceberg/io/fsspec.py`, `_s3()` builds the `s3fs.S3FileSystem` **first**, then registers the signer on the already-constructed client:

```python
fs = S3FileSystem(**s3_fs_kwargs)
for event_name, event_function in register_events.items():
fs.s3.meta.events.unregister(event_name, unique_id=1925)
fs.s3.meta.events.register_last(event_name, event_function, unique_id=1925)
```

`fs.s3` triggers a synchronous `connect()` that creates one aiobotocore client. But with modern `aiobotocore` (3.x) the client that actually issues the request is created lazily inside the running event loop (`S3FileSystem.set_session`), and it does **not** carry the `before-sign.s3` handler registered on the earlier `fs.s3` instance. Enabling botocore DEBUG logging confirms the `before-sign.s3.PutObject` event fires with only the stock handlers (`remove_arn_from_signing_path`, `_set_extra_headers_for_unsigned_request`, `resolve_s3express_identity`) — the `S3V4RestSigner` handler is absent — and the REST signer endpoint is never called. Because `config_kwargs["signature_version"] = UNSIGNED` is set, the request goes out with `auth_type: none` and no `Authorization` header.

### How to reproduce

Environment: `pyiceberg[pyiceberg-core]==0.11.1`, `s3fs==2026.2.0`, `aiobotocore==3.1.3`, `botocore==1.42.45`, Python 3.13, against any Iceberg REST catalog whose `/v1/config` returns `s3.signer=S3V4RestSigner` (i.e. remote signing, no vended credentials).

```python
from pyiceberg.catalog import load_catalog
cat = load_catalog("x", type="rest", uri="", token="")
t = cat.create_table("ns.t", schema=...)
t.append(some_arrow_table) # PutObject to S3 goes out UNSIGNED -> InvalidRequest "Please use Signature Version 4"
```

Pinning `botocore < 1.36` (e.g. `aiobotocore < 2.16`) makes the handler fire again and the signer endpoint gets called — confirming it is a client-lifecycle regression, not a catalog/config problem.

### Proposed fix

Register the signer on the `aiobotocore` **session** that the filesystem is built from, so every client the session creates inherits the `before-sign.s3` handler, instead of registering on the post-construction `fs.s3` client. `s3fs.S3FileSystem` accepts a `session=` argument:

```python
import aiobotocore.session

session = aiobotocore.session.AioSession()
if signer := properties.get(S3_SIGNER):
if signer_cls := SIGNERS.get(signer):
session.register("before-sign.s3", signer_cls(properties))
config_kwargs["signature_version"] = botocore.UNSIGNED
fs = S3FileSystem(session=session, **s3_fs_kwargs)
```

Verified locally: with session-level registration the `S3V4RestSigner` handler fires on the real `PutObject`/`GetObject`, the REST signer endpoint is called, and a valid SigV4 `Authorization` header reaches S3 — on the same modern `aiobotocore`/`botocore` that fails today.

### Willingness to contribute

I can submit a PR implementing the session-level registration if the maintainers agree with the approach.

贡献指南

这个仓库没有索引到贡献指南

调研方向

从 pyiceberg/io/fsspec.py 中的 _s3() 开始,然后跟踪 S3FileSystem 如何通过 aiobotocore 创建客户端。使用列出的现代依赖版本和使用 S3V4RestSigner 的 REST catalog 进行复现。完成的标准是:signer handler 在实际的 PutObject/GetObject 请求上运行,REST signer endpoint 被调用,并且 S3 收到有效的 SigV4 Authorization header。

由索引模型根据 Issue 内容生成。

评估

技术栈
aws, python
领域
cloud, security
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
冷清
描述清晰度
描述清楚
新手友好度
72/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。