apache / apache/iceberg-python

FsspecFileIO remote signing (S3V4RestSigner) never fires on modern aiobotocore — S3 requests sent unsigned

オープン
#3,625 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
1.1k
フォーク
581
平均マージ
1日 17時間
マージ済み PR(30日)
78

説明

### Apache Iceberg version

0.11.1 (latest release), also reproduced on 0.11.0

### Please describe the bug 🐞

`FsspecFileIO` remote request signing (`s3.signer=S3V4RestSigner`) silently does **not** sign S3 requests when running against a recent `aiobotocore`/`botocore`. The `before-sign.s3` handler that `_s3()` registers never fires for the actual S3 operation, so the request is sent to S3 **unsigned** and S3 rejects it with:

```
InvalidRequest: The authorization mechanism you have provided is not supported. Please use Signature Version 4.
```

This breaks the standard Iceberg REST "remote signing" flow (catalog `/v1/config` sets `s3.signer=S3V4RestSigner`, client holds no AWS credentials, each S3 request is signed by the REST signer endpoint) for anyone on a current dependency set.

### Root cause

In `pyiceberg/io/fsspec.py`, `_s3()` builds the `s3fs.S3FileSystem` **first**, then registers the signer on the already-constructed client:

```python
fs = S3FileSystem(**s3_fs_kwargs)
for event_name, event_function in register_events.items():
fs.s3.meta.events.unregister(event_name, unique_id=1925)
fs.s3.meta.events.register_last(event_name, event_function, unique_id=1925)
```

`fs.s3` triggers a synchronous `connect()` that creates one aiobotocore client. But with modern `aiobotocore` (3.x) the client that actually issues the request is created lazily inside the running event loop (`S3FileSystem.set_session`), and it does **not** carry the `before-sign.s3` handler registered on the earlier `fs.s3` instance. Enabling botocore DEBUG logging confirms the `before-sign.s3.PutObject` event fires with only the stock handlers (`remove_arn_from_signing_path`, `_set_extra_headers_for_unsigned_request`, `resolve_s3express_identity`) — the `S3V4RestSigner` handler is absent — and the REST signer endpoint is never called. Because `config_kwargs["signature_version"] = UNSIGNED` is set, the request goes out with `auth_type: none` and no `Authorization` header.

### How to reproduce

Environment: `pyiceberg[pyiceberg-core]==0.11.1`, `s3fs==2026.2.0`, `aiobotocore==3.1.3`, `botocore==1.42.45`, Python 3.13, against any Iceberg REST catalog whose `/v1/config` returns `s3.signer=S3V4RestSigner` (i.e. remote signing, no vended credentials).

```python
from pyiceberg.catalog import load_catalog
cat = load_catalog("x", type="rest", uri="", token="")
t = cat.create_table("ns.t", schema=...)
t.append(some_arrow_table) # PutObject to S3 goes out UNSIGNED -> InvalidRequest "Please use Signature Version 4"
```

Pinning `botocore < 1.36` (e.g. `aiobotocore < 2.16`) makes the handler fire again and the signer endpoint gets called — confirming it is a client-lifecycle regression, not a catalog/config problem.

### Proposed fix

Register the signer on the `aiobotocore` **session** that the filesystem is built from, so every client the session creates inherits the `before-sign.s3` handler, instead of registering on the post-construction `fs.s3` client. `s3fs.S3FileSystem` accepts a `session=` argument:

```python
import aiobotocore.session

session = aiobotocore.session.AioSession()
if signer := properties.get(S3_SIGNER):
if signer_cls := SIGNERS.get(signer):
session.register("before-sign.s3", signer_cls(properties))
config_kwargs["signature_version"] = botocore.UNSIGNED
fs = S3FileSystem(session=session, **s3_fs_kwargs)
```

Verified locally: with session-level registration the `S3V4RestSigner` handler fires on the real `PutObject`/`GetObject`, the REST signer endpoint is called, and a valid SigV4 `Authorization` header reaches S3 — on the same modern `aiobotocore`/`botocore` that fails today.

### Willingness to contribute

I can submit a PR implementing the session-level registration if the maintainers agree with the approach.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

pyiceberg/io/fsspec.py の _s3() から始め、S3FileSystem が aiobotocore を通じてクライアントを作成する流れを追跡します。記載されている最新の依存関係バージョンと、S3V4RestSigner を使用する REST カタログで再現してください。実際の PutObject/GetObject リクエストで signer handler が実行され、REST signer エンドポイントが呼び出され、S3 が有効な SigV4 Authorization ヘッダーを受信すれば完了です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
aws, python
領域
cloud, security
issue の種類
バグ
難易度
3/5
見積もり時間
1〜2日
活発さ
静か
明瞭さ
明確に書かれている
初心者へのやさしさ
72/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。