apache / apache/paimon

[Bug] IndexOutOfBoundsException when lookup join sql

Offen
#8,653 3 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug
Vorherrschende Sprache
Java
Sterne
3.4k
Forks
1.4k
Ø Merge
1 T. 11 Std.
Gemergte PRs (30 T.)
396

Beschreibung

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

paimon 1.1.1 +flink 1.18
Basically, all the trigger conditions are met:
1. DEBUG logging was enabled for org.apache.paimon.flink.lookup.FileStoreLookupFunction in the production environment.
2. A lookup join is being used, and the table created is a SecondaryIndexLookupTable.
3. The join key is contract_no, which is not the complete primary key. The logs show _KEY_contract_no and _KEY_reg_valid, indicating that the full key also involves reg_valid.
4. There is a mismatch in the field order among the lookup projection, Paimon's internal read schema, and the original table schema.
5. The DEBUG branch attempts to print the matched rows, which leads to the execution of logRow().

LookupJoin[8] switched from RUNNING to FAILED

java.lang.RuntimeException: java.lang.IndexOutOfBoundsException
at org.apache.paimon.flink.lookup.FileStoreLookupFunction.lookup
...
Caused by: java.lang.IndexOutOfBoundsException
at org.apache.paimon.memory.MemorySegment.get
at org.apache.paimon.memory.MemorySegmentUtils.copyToBytes
at org.apache.paimon.data.BinaryString.toString
at org.apache.paimon.flink.lookup.FileStoreLookupFunction.logRow
at org.apache.paimon.flink.lookup.FileStoreLookupFunction.lambda$lookupInternal$4

### Compute Engine

flink 1.18

### Minimal reproduce step

1. DEBUG logging was enabled for org.apache.paimon.flink.lookup.FileStoreLookupFunction in the production environment.
2. A lookup join is being used, and the table created is a SecondaryIndexLookupTable.
3. The join key is contract_no, which is not the complete primary key. The logs show _KEY_contract_no and _KEY_reg_valid, indicating that the full key also involves reg_valid.
4. There is a mismatch in the field order among the lookup projection, Paimon's internal read schema, and the original table schema.
5. The DEBUG branch attempts to print the matched rows, which leads to the execution of logRow().

### What doesn't meet your expectations?

none

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Beginne bei FileStoreLookupFunction.lookup und folge lookupInternal in den DEBUG-Logging-Pfad, der logRow() aufruft. Reproduziere das Problem mit einem Flink 1.18 Lookup-Join auf einer SecondaryIndexLookupTable, einem partiellen contract_no-Schlüssel und aktiviertem DEBUG-Logging. Erledigt ist die Aufgabe, wenn der Lookup beim Logging übereinstimmender Zeilen nicht mehr mit IndexOutOfBoundsException fehlschlägt.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
java
Bereich
data-engineering, databases
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Aktiv
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
55/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.