apache / apache/iceberg-python
FsspecFileIO remote signing (S3V4RestSigner) never fires on modern aiobotocore — S3 requests sent unsigned
- Ngôn ngữ chính
- Python
- Star
- 1.1k
- Fork
- 581
- Merge trung bình
- 1 ngày 17 giờ
- Pull request đã merge (30 ngày)
- 78
Mô tả
### Apache Iceberg version
0.11.1 (latest release), also reproduced on 0.11.0
### Please describe the bug 🐞
`FsspecFileIO` remote request signing (`s3.signer=S3V4RestSigner`) silently does **not** sign S3 requests when running against a recent `aiobotocore`/`botocore`. The `before-sign.s3` handler that `_s3()` registers never fires for the actual S3 operation, so the request is sent to S3 **unsigned** and S3 rejects it with:
```
InvalidRequest: The authorization mechanism you have provided is not supported. Please use Signature Version 4.
```
This breaks the standard Iceberg REST "remote signing" flow (catalog `/v1/config` sets `s3.signer=S3V4RestSigner`, client holds no AWS credentials, each S3 request is signed by the REST signer endpoint) for anyone on a current dependency set.
### Root cause
In `pyiceberg/io/fsspec.py`, `_s3()` builds the `s3fs.S3FileSystem` **first**, then registers the signer on the already-constructed client:
```python
fs = S3FileSystem(**s3_fs_kwargs)
for event_name, event_function in register_events.items():
fs.s3.meta.events.unregister(event_name, unique_id=1925)
fs.s3.meta.events.register_last(event_name, event_function, unique_id=1925)
```
`fs.s3` triggers a synchronous `connect()` that creates one aiobotocore client. But with modern `aiobotocore` (3.x) the client that actually issues the request is created lazily inside the running event loop (`S3FileSystem.set_session`), and it does **not** carry the `before-sign.s3` handler registered on the earlier `fs.s3` instance. Enabling botocore DEBUG logging confirms the `before-sign.s3.PutObject` event fires with only the stock handlers (`remove_arn_from_signing_path`, `_set_extra_headers_for_unsigned_request`, `resolve_s3express_identity`) — the `S3V4RestSigner` handler is absent — and the REST signer endpoint is never called. Because `config_kwargs["signature_version"] = UNSIGNED` is set, the request goes out with `auth_type: none` and no `Authorization` header.
### How to reproduce
Environment: `pyiceberg[pyiceberg-core]==0.11.1`, `s3fs==2026.2.0`, `aiobotocore==3.1.3`, `botocore==1.42.45`, Python 3.13, against any Iceberg REST catalog whose `/v1/config` returns `s3.signer=S3V4RestSigner` (i.e. remote signing, no vended credentials).
```python
from pyiceberg.catalog import load_catalog
cat = load_catalog("x", type="rest", uri="", token="")
t = cat.create_table("ns.t", schema=...)
t.append(some_arrow_table) # PutObject to S3 goes out UNSIGNED -> InvalidRequest "Please use Signature Version 4"
```
Pinning `botocore < 1.36` (e.g. `aiobotocore < 2.16`) makes the handler fire again and the signer endpoint gets called — confirming it is a client-lifecycle regression, not a catalog/config problem.
### Proposed fix
Register the signer on the `aiobotocore` **session** that the filesystem is built from, so every client the session creates inherits the `before-sign.s3` handler, instead of registering on the post-construction `fs.s3` client. `s3fs.S3FileSystem` accepts a `session=` argument:
```python
import aiobotocore.session
session = aiobotocore.session.AioSession()
if signer := properties.get(S3_SIGNER):
if signer_cls := SIGNERS.get(signer):
session.register("before-sign.s3", signer_cls(properties))
config_kwargs["signature_version"] = botocore.UNSIGNED
fs = S3FileSystem(session=session, **s3_fs_kwargs)
```
Verified locally: with session-level registration the `S3V4RestSigner` handler fires on the real `PutObject`/`GetObject`, the REST signer endpoint is called, and a valid SigV4 `Authorization` header reaches S3 — on the same modern `aiobotocore`/`botocore` that fails today.
### Willingness to contribute
I can submit a PR implementing the session-level registration if the maintainers agree with the approach.
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Hướng nghiên cứu
Bắt đầu trong pyiceberg/io/fsspec.py tại _s3(), sau đó theo dõi cách S3FileSystem tạo các client thông qua aiobotocore. Tái hiện với các phiên bản dependency hiện đại được liệt kê và một REST catalog sử dụng S3V4RestSigner. Được xem là hoàn tất khi signer handler chạy trên các request PutObject/GetObject thực tế, endpoint của REST signer được gọi và S3 nhận được một header Authorization SigV4 hợp lệ.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- aws, python
- Lĩnh vực
- cloud, security
- Loại issue
- Lỗi
- Độ khó
- 3/5
- Thời gian dự kiến
- 1-2 ngày
- Mức độ hoạt động
- Ít trao đổi
- Độ rõ ràng
- Đặc tả rõ ràng
- Mức phù hợp với người mới
- 72/100