apache / apache/iceberg-python
Compatible File storage provider that do not support signature with trailer
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 581
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 78
Description
### Feature Request / Improvement
When use fsspec io-impl, put object to File Storage Provider(FSP) not support
- [Signature calculations for trailing headers](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming-trailers.html)
- [checking-object-integrity-upload](https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity-upload.html)
Put file like snapxxx.avro will not handle by FSP after upload(put object is success), the file some data signature related things are reserved.
When get snapxxx.avro to read and decoder, raise Error, Because the file contains signature information
```python
File "pyiceberg/avro/decoder_fast.pyx", line 150, in pyiceberg.avro.decoder_fast.CythonBinaryDecoder.read_utf8
cpdef str read_utf8(self):
File "pyiceberg/avro/decoder_fast.pyx", line 156, in pyiceberg.avro.decoder_fast.CythonBinaryDecoder.read_utf8
return self.read_bytes().decode("utf-8")
File "pyiceberg/avro/decoder_fast.pyx", line 130, in pyiceberg.avro.decoder_fast.CythonBinaryDecoder.read_bytes
return r[0:length]
SystemError: Negative size passed to PyBytes_FromStringAndSize
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the fsspec I/O implementation and reproduce the upload and download of a snapxxx.avro file against a file storage provider that lacks trailing-header signature support. Inspect the resulting object and the failure in pyiceberg/avro/decoder_fast.pyx. Done means objects uploaded through this path can be read and decoded by the affected provider without retained signature data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100