apache / apache/paimon

[Bug] IndexOutOfBoundsException when lookup join sql

Ouverte
#8,653 3 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
bug
Langage dominant
Java
Étoiles
3.4k
Forks
1.4k
Merge moyen
1 j 11 h
PR mergées (30 j)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

paimon 1.1.1 +flink 1.18
Basically, all the trigger conditions are met:
1. DEBUG logging was enabled for org.apache.paimon.flink.lookup.FileStoreLookupFunction in the production environment.
2. A lookup join is being used, and the table created is a SecondaryIndexLookupTable.
3. The join key is contract_no, which is not the complete primary key. The logs show _KEY_contract_no and _KEY_reg_valid, indicating that the full key also involves reg_valid.
4. There is a mismatch in the field order among the lookup projection, Paimon's internal read schema, and the original table schema.
5. The DEBUG branch attempts to print the matched rows, which leads to the execution of logRow().

LookupJoin[8] switched from RUNNING to FAILED

java.lang.RuntimeException: java.lang.IndexOutOfBoundsException
at org.apache.paimon.flink.lookup.FileStoreLookupFunction.lookup
...
Caused by: java.lang.IndexOutOfBoundsException
at org.apache.paimon.memory.MemorySegment.get
at org.apache.paimon.memory.MemorySegmentUtils.copyToBytes
at org.apache.paimon.data.BinaryString.toString
at org.apache.paimon.flink.lookup.FileStoreLookupFunction.logRow
at org.apache.paimon.flink.lookup.FileStoreLookupFunction.lambda$lookupInternal$4

### Compute Engine

flink 1.18

### Minimal reproduce step

1. DEBUG logging was enabled for org.apache.paimon.flink.lookup.FileStoreLookupFunction in the production environment.
2. A lookup join is being used, and the table created is a SecondaryIndexLookupTable.
3. The join key is contract_no, which is not the complete primary key. The logs show _KEY_contract_no and _KEY_reg_valid, indicating that the full key also involves reg_valid.
4. There is a mismatch in the field order among the lookup projection, Paimon's internal read schema, and the original table schema.
5. The DEBUG branch attempts to print the matched rows, which leads to the execution of logRow().

### What doesn't meet your expectations?

none

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

Commencez par FileStoreLookupFunction.lookup et suivez lookupInternal jusqu’au chemin de journalisation DEBUG qui appelle logRow(). Reproduisez le problème avec un lookup join Flink 1.18 sur une SecondaryIndexLookupTable, une clé contract_no partielle et la journalisation DEBUG activée. C’est terminé lorsque le lookup n’échoue plus avec IndexOutOfBoundsException lors de la journalisation des lignes correspondantes.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
java
Domaine
data-engineering, databases
Type d'issue
Bug
Difficulté
4/5
Temps estimé
3-5 jours
Activité
Active
Clarté
Plutôt claire
Accessibilité débutants
55/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.