apache / apache/iceberg-python

Compatible File storage provider that do not support signature with trailer

Offen
#2,529 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
1.1k
Forks
581
Ø Merge
1 T. 17 Std.
Gemergte PRs (30 T.)
78

Beschreibung

### Feature Request / Improvement

When use fsspec io-impl, put object to File Storage Provider(FSP) not support

- [Signature calculations for trailing headers](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming-trailers.html)

- [checking-object-integrity-upload](https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity-upload.html)

Put file like snapxxx.avro will not handle by FSP after upload(put object is success), the file some data signature related things are reserved.

When get snapxxx.avro to read and decoder, raise Error, Because the file contains signature information
```python
File "pyiceberg/avro/decoder_fast.pyx", line 150, in pyiceberg.avro.decoder_fast.CythonBinaryDecoder.read_utf8
cpdef str read_utf8(self):
File "pyiceberg/avro/decoder_fast.pyx", line 156, in pyiceberg.avro.decoder_fast.CythonBinaryDecoder.read_utf8
return self.read_bytes().decode("utf-8")
File "pyiceberg/avro/decoder_fast.pyx", line 130, in pyiceberg.avro.decoder_fast.CythonBinaryDecoder.read_bytes
return r[0:length]
SystemError: Negative size passed to PyBytes_FromStringAndSize
```

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Start with the fsspec I/O implementation and reproduce the upload and download of a snapxxx.avro file against a file storage provider that lacks trailing-header signature support. Inspect the resulting object and the failure in pyiceberg/avro/decoder_fast.pyx. Done means objects uploaded through this path can be read and decoded by the affected provider without retained signature data.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
backend
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Ruhig
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
42/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.