apache / apache/iceberg-python

Cython Avro decoder reads past the buffer end on malformed input

Open
#3,952 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
1.1k
Forks
581
Avg merge
1d 17h
Merged PRs (30d)
78

Description

Two places in `pyiceberg/avro/decoder_fast.pyx` advance the read pointer using a value taken from the stream, without bounding it against `self._end`.

**1. `read_bytes` does not validate the decoded length**

```python
cpdef inline bytes read_bytes(self):
cdef uint64_t length;
if self._current >= self._end: # only confirms 1 byte is available
raise EOFError(f"EOF: read 1 bytes")

decode_zigzag_ints(&self._current, 1, &length)

if length <= 0:
return b""
cdef const unsigned char *r = self._current
self._current += length # not checked against self._end
return r[0:length]
```

The guard confirms one byte is available before decoding the length, but the decoded `length` is then used to slice and to advance `_current` with no check that `_current + length <= _end`. A length field larger than the remaining buffer reads beyond it.

**2. `decode_zigzag_ints` has no end pointer to bound against**

```c
void decode_zigzag_ints(const unsigned char **buffer, const uint64_t count, uint64_t *result);
```

The signature takes a buffer and a count but no end, so the varint walk cannot stop at the buffer boundary — a run of bytes with the continuation bit set keeps advancing. It is called from five sites in the decoder (lines 93, 101, 111, 124, 176), including from `read_bytes` above.

Both are reachable from a malformed or hostile Avro manifest.

---
Issue investigation generated via claude, reviewed by Sung, Kevin, Fokko.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in pyiceberg/avro/decoder_fast.pyx, reading read_bytes and decode_zigzag_ints plus the five call sites listed in the issue. Exercise malformed or truncated Avro input through the decoder. Done means malformed input cannot advance past the buffer end and valid decoding remains covered by the existing decoder behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
security
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.