apache / apache/paimon

[Bug] [python] Primary-key merge reader fails when advancing to an empty batch

Closed Beginner friendly
#9,819 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the issues and found nothing similar.

### Paimon version

Apache Paimon master, commit `c081ccd0d7003ceae2517c6b592a947e824acc25`.

### Compute Engine

PyPaimon, Python 3.12.6, PyArrow 19.0.1.

### Minimal reproduce step

A primary-key merge read fails when an input reader returns an empty batch after a non-empty batch. The following example uses the reader interfaces to reproduce the batch transition without external storage. Run it with `PYTHONPATH` pointing to `paimon-python`:

```python
from pypaimon.read.reader.iface.record_iterator import RecordIterator
from pypaimon.read.reader.iface.record_reader import RecordReader
from pypaimon.read.reader.sort_merge_reader import SortMergeReaderWithMinHeap
from pypaimon.schema.data_types import AtomicType, DataField
from pypaimon.schema.table_schema import TableSchema
from pypaimon.table.row.key_value import KeyValue
from pypaimon.table.row.row_kind import RowKind

class BatchIterator(RecordIterator):
def __init__(self, rows):
self.rows = iter(rows)

def next(self):
return next(self.rows, None)

class BatchReader(RecordReader):
def __init__(self, batches):
self.batches = iter(batches)

def read_batch(self):
batch = next(self.batches, None)
return None if batch is None else BatchIterator(batch)

def close(self):
pass

def kv(key):
return KeyValue(1, 1).replace((key, 1, RowKind.INSERT.value, key))

schema = TableSchema(
fields=[DataField(0, "id", AtomicType("INT"))], primary_keys=["id"])
reader = SortMergeReaderWithMinHeap(
[BatchReader([[kv(1)], [], [], [kv(2)], []])], schema)
try:
result = []
while True:
batch = reader.read_batch()
if batch is None:
break
while True:
record = batch.next()
if record is None:
break
result.append(record.key.get_field(0))
assert result == [1, 2], result
print(result)
finally:
reader.close()
```

### What doesn't meet your expectations?

Expected: read both records and print `[1, 2]`. An empty batch should not end the reader; only `read_batch()` returning `None` indicates EOF.

Actual: the second call to the merge iterator raises:

```text
File "pypaimon/read/reader/sort_merge_reader.py", line 127, in _next_impl
entry = HeapEntry(element.kv.key, element, self.key_comparator,
AttributeError: 'NoneType' object has no attribute 'key'
```

### Anything else?

In `Element.update()`, after exhausting the current batch, we read one more batch and assign its first record to `self.kv`. If that batch is empty, we assign `None` but return `True`. The caller then accesses `element.kv.key`.

I also reproduced this through table writes, commits, and a filtered read using local files: a full scan returns 6,144 rows, while the filtered scan fails with the same exception. With the fix, the filtered scan returns the expected four rows and values.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Run the provided reproduction with PYTHONPATH pointing to paimon-python, then read pypaimon/read/reader/sort_merge_reader.py, especially Element.update() and SortMergeReaderWithMinHeap._next_impl(). Check the reader interfaces used by BatchReader to understand batch and EOF behavior. Done means empty batches are skipped without ending the reader, the example prints [1, 2], and the filtered local-file read returns its expected four rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
86/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.