apache / apache/paimon

[Bug] RemoteTableQuery keySerializer should not include partition keys

Open Beginner friendly
#8,145 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

When doing a lookup join to the remote query service (for primary key tables), we pass the partition keys, a bucket number, and a row to join to the existing table. Right now, serialization of the row fails because the serializer expected by the remote query service client assumes that the row includes partition fields, but we do not pass them (this is the same in LocalTableQuery, not a bug).

Let's change the key serializer here to avoid looking for partition keys.

### Compute Engine

Flink

### Minimal reproduce step

Use the remote query service to perform a lookup join from a flink row to a paimon table. The table must be partitioned, and the partitioned key and trimmed key must have different types (e.g. int vs. string). I'll include a test case in the PR that trips it up.

### What doesn't meet your expectations?

```
java.lang.ClassCastException: class org.apache.paimon.data.BinaryString cannot be cast to class java.lang.Integer
at org.apache.paimon.data.GenericRow.getInt(GenericRow.java:152)
at org.apache.paimon.data.serializer.InternalRowSerializer.toBinaryRow(InternalRowSerializer.java:123)
at org.apache.paimon.flink.query.RemoteTableQuery.lookup(RemoteTableQuery.java:73)
```

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in org.apache.paimon.flink.query.RemoteTableQuery.lookup and trace its use of InternalRowSerializer.toBinaryRow. Reproduce a remote lookup join against a partitioned primary-key table where partition and trimmed key types differ, then verify serialization no longer reads partition fields and the ClassCastException is gone.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.