apache / apache/iceberg-python
FsspecFileIO remote signing (S3V4RestSigner) never fires on modern aiobotocore — S3 requests sent unsigned
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 581
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 78
Description
### Apache Iceberg version
0.11.1 (latest release), also reproduced on 0.11.0
### Please describe the bug 🐞
`FsspecFileIO` remote request signing (`s3.signer=S3V4RestSigner`) silently does **not** sign S3 requests when running against a recent `aiobotocore`/`botocore`. The `before-sign.s3` handler that `_s3()` registers never fires for the actual S3 operation, so the request is sent to S3 **unsigned** and S3 rejects it with:
```
InvalidRequest: The authorization mechanism you have provided is not supported. Please use Signature Version 4.
```
This breaks the standard Iceberg REST "remote signing" flow (catalog `/v1/config` sets `s3.signer=S3V4RestSigner`, client holds no AWS credentials, each S3 request is signed by the REST signer endpoint) for anyone on a current dependency set.
### Root cause
In `pyiceberg/io/fsspec.py`, `_s3()` builds the `s3fs.S3FileSystem` **first**, then registers the signer on the already-constructed client:
```python
fs = S3FileSystem(**s3_fs_kwargs)
for event_name, event_function in register_events.items():
fs.s3.meta.events.unregister(event_name, unique_id=1925)
fs.s3.meta.events.register_last(event_name, event_function, unique_id=1925)
```
`fs.s3` triggers a synchronous `connect()` that creates one aiobotocore client. But with modern `aiobotocore` (3.x) the client that actually issues the request is created lazily inside the running event loop (`S3FileSystem.set_session`), and it does **not** carry the `before-sign.s3` handler registered on the earlier `fs.s3` instance. Enabling botocore DEBUG logging confirms the `before-sign.s3.PutObject` event fires with only the stock handlers (`remove_arn_from_signing_path`, `_set_extra_headers_for_unsigned_request`, `resolve_s3express_identity`) — the `S3V4RestSigner` handler is absent — and the REST signer endpoint is never called. Because `config_kwargs["signature_version"] = UNSIGNED` is set, the request goes out with `auth_type: none` and no `Authorization` header.
### How to reproduce
Environment: `pyiceberg[pyiceberg-core]==0.11.1`, `s3fs==2026.2.0`, `aiobotocore==3.1.3`, `botocore==1.42.45`, Python 3.13, against any Iceberg REST catalog whose `/v1/config` returns `s3.signer=S3V4RestSigner` (i.e. remote signing, no vended credentials).
```python
from pyiceberg.catalog import load_catalog
cat = load_catalog("x", type="rest", uri="", token="")
t = cat.create_table("ns.t", schema=...)
t.append(some_arrow_table) # PutObject to S3 goes out UNSIGNED -> InvalidRequest "Please use Signature Version 4"
```
Pinning `botocore < 1.36` (e.g. `aiobotocore < 2.16`) makes the handler fire again and the signer endpoint gets called — confirming it is a client-lifecycle regression, not a catalog/config problem.
### Proposed fix
Register the signer on the `aiobotocore` **session** that the filesystem is built from, so every client the session creates inherits the `before-sign.s3` handler, instead of registering on the post-construction `fs.s3` client. `s3fs.S3FileSystem` accepts a `session=` argument:
```python
import aiobotocore.session
session = aiobotocore.session.AioSession()
if signer := properties.get(S3_SIGNER):
if signer_cls := SIGNERS.get(signer):
session.register("before-sign.s3", signer_cls(properties))
config_kwargs["signature_version"] = botocore.UNSIGNED
fs = S3FileSystem(session=session, **s3_fs_kwargs)
```
Verified locally: with session-level registration the `S3V4RestSigner` handler fires on the real `PutObject`/`GetObject`, the REST signer endpoint is called, and a valid SigV4 `Authorization` header reaches S3 — on the same modern `aiobotocore`/`botocore` that fails today.
### Willingness to contribute
I can submit a PR implementing the session-level registration if the maintainers agree with the approach.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in pyiceberg/io/fsspec.py at _s3(), then trace how S3FileSystem creates clients through aiobotocore. Reproduce with the listed modern dependency versions and a REST catalog using S3V4RestSigner. Done means the signer handler runs on actual PutObject/GetObject requests, the REST signer endpoint is called, and S3 receives a valid SigV4 Authorization header.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, security
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100