apache / apache/paimon

[Bug] Comparison of timestamp types, array out of bounds

Open
#5,869 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

0.9

### Compute Engine

spark

### Minimal reproduce step

Execution failure SQL : select * from ods_mdtrt_d where updt_time >= TIMESTAMP '2025-07-10 17:00:00' limit 10;

```
Caused by: java.lang.ArrayIndexOutOfBoundsException: 49153
at org.apache.paimon.memory.MemorySegmentUtils.getLongMultiSegments(MemorySegmentUtils.java:774)
at org.apache.paimon.memory.MemorySegmentUtils.getLong(MemorySegmentUtils.java:764)
at org.apache.paimon.memory.MemorySegmentUtils.readTimestampData(MemorySegmentUtils.java:1060)
at org.apache.paimon.data.BinaryRow.getTimestamp(BinaryRow.java:323)
at org.apache.paimon.utils.InternalRowUtils.get(InternalRowUtils.java:171)
at org.apache.paimon.predicate.LeafPredicate.test(LeafPredicate.java:100)
at org.apache.paimon.predicate.And.test(And.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.operation.KeyValueFileStoreScan.filterByValueFilter(KeyValueFileStoreScan.java:197)
at org.apache.paimon.operation.KeyValueFileStoreScan.filterWholeBucketAllFiles(KeyValueFileStoreScan.java:186)
at org.apache.paimon.operation.KeyValueFileStoreScan.filterWholeBucketByStats(KeyValueFileStoreScan.java:164)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
at org.apache.paimon.operation.AbstractFileStoreScan.doPlan(AbstractFileStoreScan.java:336)
at org.apache.paimon.operation.AbstractFileStoreScan.plan(AbstractFileStoreScan.java:225)
at org.apache.paimon.table.source.snapshot.SnapshotReaderImpl.read(SnapshotReaderImpl.java:259)
at org.apache.paimon.table.source.snapshot.FullStartingScanner.scan(FullStartingScanner.java:54)
at org.apache.paimon.table.source.DataTableBatchScan.plan(DataTableBatchScan.java:76)
at org.apache.paimon.spark.PaimonBaseScan.getOriginSplits(PaimonBaseScan.scala:101)
at org.apache.paimon.spark.PaimonBaseScan.lazyInputPartitions(PaimonBaseScan.scala:110)
at org.apache.paimon.spark.PaimonBaseScan.toBatch(PaimonBaseScan.scala:121)
```

The following SQL execution was successful:

select * from ods_mdtrt_d WHERE from_unixtime(unix_timestamp(updt_time, 'yyyy-MM-dd HH:mm: ss')) >= '2025-07-10 17:00:00' limit 10;

### What doesn't meet your expectations?

-

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the Spark SQL timestamp comparison and start with MemorySegmentUtils.java, BinaryRow.java, LeafPredicate.java, and KeyValueFileStoreScan.java from the stack trace. Compare the failing direct predicate with the successful converted-timestamp query, then verify that the original query completes without an ArrayIndexOutOfBoundsException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.