[Bug] lookup with retry is very slow if there are equality in a where clause after the join
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
1.20
### Compute Engine
Flink
### Minimal reproduce step
I am having an issue with a `WHERE` condition on a lookup join. Taking the customer and orders example from the documentation, if I want to filter the results of the join by the country=US, it doesn't work. Because it seems to optimizes the SQL by moving the `c.country = 'US'` clause from the where condition to the join condition and do a lookup join based on both customer id and country. But since many customers are not in US, this lookup fails, and it keeps on retrying.
```
SELECT /*+ LOOKUP('table'='c', 'retry-predicate'='lookup_miss', 'retry-strategy'='fixed_delay', 'fixed-delay'='1s', 'max-attempts'='600') */
o.order_id, o.total, c.country, c.zip
FROM orders AS o
JOIN customers
FOR SYSTEM_TIME AS OF o.proc_time AS c
ON o.customer_id = c.id
WHERE c.country = 'US'
;
```
I have found a simple workaround , by adding an OR condition to the where clause i.e `WHERE c.country = 'US' or c.country = 'dummy'` . The OR part of the condition never matches, but adding this prevents the SQL optimization from being used, and the lookup only uses customer id.
### What doesn't meet your expectations?
I would expect only columns in the ON condition to be used for the Lookup join, not columns in the WHERE condition.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the customer/orders lookup-join example from the documentation and run the supplied SQL reproduction in Paimon 1.20 with Flink. Compare the lookup behavior with the direct WHERE equality and the documented OR workaround. Done means the WHERE-only equality does not become a lookup key or trigger repeated retries, while the ON-condition lookup still works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100