apache / apache/iceberg-python

FsspecFileIO remote signing (S3V4RestSigner) never fires on modern aiobotocore — S3 requests sent unsigned

Offen
#3,625 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
1.1k
Forks
581
Ø Merge
1 T. 17 Std.
Gemergte PRs (30 T.)
78

Beschreibung

### Apache Iceberg version

0.11.1 (latest release), also reproduced on 0.11.0

### Please describe the bug 🐞

`FsspecFileIO` remote request signing (`s3.signer=S3V4RestSigner`) silently does **not** sign S3 requests when running against a recent `aiobotocore`/`botocore`. The `before-sign.s3` handler that `_s3()` registers never fires for the actual S3 operation, so the request is sent to S3 **unsigned** and S3 rejects it with:

```
InvalidRequest: The authorization mechanism you have provided is not supported. Please use Signature Version 4.
```

This breaks the standard Iceberg REST "remote signing" flow (catalog `/v1/config` sets `s3.signer=S3V4RestSigner`, client holds no AWS credentials, each S3 request is signed by the REST signer endpoint) for anyone on a current dependency set.

### Root cause

In `pyiceberg/io/fsspec.py`, `_s3()` builds the `s3fs.S3FileSystem` **first**, then registers the signer on the already-constructed client:

```python
fs = S3FileSystem(**s3_fs_kwargs)
for event_name, event_function in register_events.items():
fs.s3.meta.events.unregister(event_name, unique_id=1925)
fs.s3.meta.events.register_last(event_name, event_function, unique_id=1925)
```

`fs.s3` triggers a synchronous `connect()` that creates one aiobotocore client. But with modern `aiobotocore` (3.x) the client that actually issues the request is created lazily inside the running event loop (`S3FileSystem.set_session`), and it does **not** carry the `before-sign.s3` handler registered on the earlier `fs.s3` instance. Enabling botocore DEBUG logging confirms the `before-sign.s3.PutObject` event fires with only the stock handlers (`remove_arn_from_signing_path`, `_set_extra_headers_for_unsigned_request`, `resolve_s3express_identity`) — the `S3V4RestSigner` handler is absent — and the REST signer endpoint is never called. Because `config_kwargs["signature_version"] = UNSIGNED` is set, the request goes out with `auth_type: none` and no `Authorization` header.

### How to reproduce

Environment: `pyiceberg[pyiceberg-core]==0.11.1`, `s3fs==2026.2.0`, `aiobotocore==3.1.3`, `botocore==1.42.45`, Python 3.13, against any Iceberg REST catalog whose `/v1/config` returns `s3.signer=S3V4RestSigner` (i.e. remote signing, no vended credentials).

```python
from pyiceberg.catalog import load_catalog
cat = load_catalog("x", type="rest", uri="", token="")
t = cat.create_table("ns.t", schema=...)
t.append(some_arrow_table) # PutObject to S3 goes out UNSIGNED -> InvalidRequest "Please use Signature Version 4"
```

Pinning `botocore < 1.36` (e.g. `aiobotocore < 2.16`) makes the handler fire again and the signer endpoint gets called — confirming it is a client-lifecycle regression, not a catalog/config problem.

### Proposed fix

Register the signer on the `aiobotocore` **session** that the filesystem is built from, so every client the session creates inherits the `before-sign.s3` handler, instead of registering on the post-construction `fs.s3` client. `s3fs.S3FileSystem` accepts a `session=` argument:

```python
import aiobotocore.session

session = aiobotocore.session.AioSession()
if signer := properties.get(S3_SIGNER):
if signer_cls := SIGNERS.get(signer):
session.register("before-sign.s3", signer_cls(properties))
config_kwargs["signature_version"] = botocore.UNSIGNED
fs = S3FileSystem(session=session, **s3_fs_kwargs)
```

Verified locally: with session-level registration the `S3V4RestSigner` handler fires on the real `PutObject`/`GetObject`, the REST signer endpoint is called, and a valid SigV4 `Authorization` header reaches S3 — on the same modern `aiobotocore`/`botocore` that fails today.

### Willingness to contribute

I can submit a PR implementing the session-level registration if the maintainers agree with the approach.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Beginne in pyiceberg/io/fsspec.py bei _s3() und verfolge anschließend, wie S3FileSystem über aiobotocore Clients erstellt. Reproduziere das Verhalten mit den aufgeführten modernen Dependency-Versionen und einem REST-Katalog, der S3V4RestSigner verwendet. Als abgeschlossen gilt die Aufgabe, wenn der Signer-Handler bei tatsächlichen PutObject/GetObject-Anfragen ausgeführt wird, der REST-Signer-Endpunkt aufgerufen wird und S3 einen gültigen SigV4 Authorization-Header empfängt.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
aws, python
Bereich
cloud, security
Issue-Typ
Bug
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Ruhig
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
72/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.