apache / apache/paimon

[Bug] IndexOutOfBoundsException when lookup join sql

未關閉
#8,653 3 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
bug
主要語言
Java
星號
3.4k
分支
1.4k
平均合併
1 天 9 小時
30 天內合併 PR
423

描述

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

paimon 1.1.1 +flink 1.18
Basically, all the trigger conditions are met:
1. DEBUG logging was enabled for org.apache.paimon.flink.lookup.FileStoreLookupFunction in the production environment.
2. A lookup join is being used, and the table created is a SecondaryIndexLookupTable.
3. The join key is contract_no, which is not the complete primary key. The logs show _KEY_contract_no and _KEY_reg_valid, indicating that the full key also involves reg_valid.
4. There is a mismatch in the field order among the lookup projection, Paimon's internal read schema, and the original table schema.
5. The DEBUG branch attempts to print the matched rows, which leads to the execution of logRow().

LookupJoin[8] switched from RUNNING to FAILED

java.lang.RuntimeException: java.lang.IndexOutOfBoundsException
at org.apache.paimon.flink.lookup.FileStoreLookupFunction.lookup
...
Caused by: java.lang.IndexOutOfBoundsException
at org.apache.paimon.memory.MemorySegment.get
at org.apache.paimon.memory.MemorySegmentUtils.copyToBytes
at org.apache.paimon.data.BinaryString.toString
at org.apache.paimon.flink.lookup.FileStoreLookupFunction.logRow
at org.apache.paimon.flink.lookup.FileStoreLookupFunction.lambda$lookupInternal$4

### Compute Engine

flink 1.18

### Minimal reproduce step

1. DEBUG logging was enabled for org.apache.paimon.flink.lookup.FileStoreLookupFunction in the production environment.
2. A lookup join is being used, and the table created is a SecondaryIndexLookupTable.
3. The join key is contract_no, which is not the complete primary key. The logs show _KEY_contract_no and _KEY_reg_valid, indicating that the full key also involves reg_valid.
4. There is a mismatch in the field order among the lookup projection, Paimon's internal read schema, and the original table schema.
5. The DEBUG branch attempts to print the matched rows, which leads to the execution of logRow().

### What doesn't meet your expectations?

none

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

從 FileStoreLookupFunction.lookup 開始,追蹤 lookupInternal 進入呼叫 logRow() 的 DEBUG 記錄路徑。使用 Flink 1.18 在 SecondaryIndexLookupTable 上執行 lookup join,提供部分 contract_no 索引鍵並啟用 DEBUG 記錄,以重現此問題。當 lookup 在記錄相符資料列時不再因 IndexOutOfBoundsException 而失敗時,即表示完成。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
java
領域
data-engineering, databases
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
活躍
描述清晰度
基本清楚
新手友好度
55/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。