[Bug] [python] Primary-key merge reader fails when advancing to an empty batch
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the issues and found nothing similar.
### Paimon version
Apache Paimon master, commit `c081ccd0d7003ceae2517c6b592a947e824acc25`.
### Compute Engine
PyPaimon, Python 3.12.6, PyArrow 19.0.1.
### Minimal reproduce step
A primary-key merge read fails when an input reader returns an empty batch after a non-empty batch. The following example uses the reader interfaces to reproduce the batch transition without external storage. Run it with `PYTHONPATH` pointing to `paimon-python`:
```python
from pypaimon.read.reader.iface.record_iterator import RecordIterator
from pypaimon.read.reader.iface.record_reader import RecordReader
from pypaimon.read.reader.sort_merge_reader import SortMergeReaderWithMinHeap
from pypaimon.schema.data_types import AtomicType, DataField
from pypaimon.schema.table_schema import TableSchema
from pypaimon.table.row.key_value import KeyValue
from pypaimon.table.row.row_kind import RowKind
class BatchIterator(RecordIterator):
def __init__(self, rows):
self.rows = iter(rows)
def next(self):
return next(self.rows, None)
class BatchReader(RecordReader):
def __init__(self, batches):
self.batches = iter(batches)
def read_batch(self):
batch = next(self.batches, None)
return None if batch is None else BatchIterator(batch)
def close(self):
pass
def kv(key):
return KeyValue(1, 1).replace((key, 1, RowKind.INSERT.value, key))
schema = TableSchema(
fields=[DataField(0, "id", AtomicType("INT"))], primary_keys=["id"])
reader = SortMergeReaderWithMinHeap(
[BatchReader([[kv(1)], [], [], [kv(2)], []])], schema)
try:
result = []
while True:
batch = reader.read_batch()
if batch is None:
break
while True:
record = batch.next()
if record is None:
break
result.append(record.key.get_field(0))
assert result == [1, 2], result
print(result)
finally:
reader.close()
```
### What doesn't meet your expectations?
Expected: read both records and print `[1, 2]`. An empty batch should not end the reader; only `read_batch()` returning `None` indicates EOF.
Actual: the second call to the merge iterator raises:
```text
File "pypaimon/read/reader/sort_merge_reader.py", line 127, in _next_impl
entry = HeapEntry(element.kv.key, element, self.key_comparator,
AttributeError: 'NoneType' object has no attribute 'key'
```
### Anything else?
In `Element.update()`, after exhausting the current batch, we read one more batch and assign its first record to `self.kv`. If that batch is empty, we assign `None` but return `True`. The caller then accesses `element.kv.key`.
I also reproduced this through table writes, commits, and a filtered read using local files: a full scan returns 6,144 rows, while the filtered scan fails with the same exception. With the fix, the filtered scan returns the expected four rows and values.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Run the provided reproduction with PYTHONPATH pointing to paimon-python, then read pypaimon/read/reader/sort_merge_reader.py, especially Element.update() and SortMergeReaderWithMinHeap._next_impl(). Check the reader interfaces used by BatchReader to understand batch and EOF behavior. Done means empty batches are skipped without ending the reader, the example prints [1, 2], and the filtered local-file read returns its expected four rows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 86/100