apache / apache/paimon

[Bug] IndexOutOfBoundsException when lookup join sql

Aberta
#8,653 3 comentários 0 reações 0 responsáveis Ver no GitHub
bug
Linguagem predominante
Java
Estrelas
3.4k
Forks
1.4k
Merge médio
1d 11h
PRs com merge (30d)
396

Descrição

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

paimon 1.1.1 +flink 1.18
Basically, all the trigger conditions are met:
1. DEBUG logging was enabled for org.apache.paimon.flink.lookup.FileStoreLookupFunction in the production environment.
2. A lookup join is being used, and the table created is a SecondaryIndexLookupTable.
3. The join key is contract_no, which is not the complete primary key. The logs show _KEY_contract_no and _KEY_reg_valid, indicating that the full key also involves reg_valid.
4. There is a mismatch in the field order among the lookup projection, Paimon's internal read schema, and the original table schema.
5. The DEBUG branch attempts to print the matched rows, which leads to the execution of logRow().

LookupJoin[8] switched from RUNNING to FAILED

java.lang.RuntimeException: java.lang.IndexOutOfBoundsException
at org.apache.paimon.flink.lookup.FileStoreLookupFunction.lookup
...
Caused by: java.lang.IndexOutOfBoundsException
at org.apache.paimon.memory.MemorySegment.get
at org.apache.paimon.memory.MemorySegmentUtils.copyToBytes
at org.apache.paimon.data.BinaryString.toString
at org.apache.paimon.flink.lookup.FileStoreLookupFunction.logRow
at org.apache.paimon.flink.lookup.FileStoreLookupFunction.lambda$lookupInternal$4

### Compute Engine

flink 1.18

### Minimal reproduce step

1. DEBUG logging was enabled for org.apache.paimon.flink.lookup.FileStoreLookupFunction in the production environment.
2. A lookup join is being used, and the table created is a SecondaryIndexLookupTable.
3. The join key is contract_no, which is not the complete primary key. The logs show _KEY_contract_no and _KEY_reg_valid, indicating that the full key also involves reg_valid.
4. There is a mismatch in the field order among the lookup projection, Paimon's internal read schema, and the original table schema.
5. The DEBUG branch attempts to print the matched rows, which leads to the execution of logRow().

### What doesn't meet your expectations?

none

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Direção de pesquisa

Comece em FileStoreLookupFunction.lookup e siga lookupInternal até o caminho de logging DEBUG que chama logRow(). Reproduza o problema com um lookup join do Flink 1.18 em uma SecondaryIndexLookupTable, uma chave contract_no parcial e o logging DEBUG habilitado. Está concluído quando o lookup não falhar mais com IndexOutOfBoundsException ao fazer logging das linhas correspondentes.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
java
Domínio
data-engineering, databases
Tipo de issue
Bug
Dificuldade
4/5
Tempo estimado
3-5 dias
Status de atividade
Ativa
Clareza
Razoavelmente clara
Facilidade para iniciantes
55/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.