apache / apache/paimon

[Bug] The filter calc operator exit when filter push down

Open
#1,953 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Paimon version

0.5

### Compute Engine

flink1.14

### Minimal reproduce step

CREATE CATALOG my_catalog
WITH
('type' = 'paimon', 'metastore' = 'hive' ....
) ;

CREATE
TABLE printsink
(
user_id BIGINT,
item_id BIGINT,
behavior STRING,
dt STRING
)
WITH
( 'connector' = 'print'
) ;

INSERT INTO printsink
SELECT
user_id,
item_id,
behavior,
dt
FROM
my_catalog.tmp.paimon_test_table13 where dt = '2023-08-30'

### What doesn't meet your expectations?
The filter calc operator exit when filter push down, is this correct ?

![image](https://github.com/apache/incubator-paimon/assets/18002496/b9d3aa9b-d82a-4319-82a7-5bbfccdb24dc)

optimize result:
Sink(table=[default_catalog.default_database.printsink], fields=[user_id, item_id, behavior, dt])
+- Calc(select=[user_id, item_id, behavior, _UTF-16LE'2023-08-30':VARCHAR(2147483647) CHARACTER SET "UTF-16LE" AS dt])
+- ChangelogNormalize(key=[dt, user_id])
+- Exchange(distribution=[hash[dt, user_id]])
+- Calc(select=[user_id, item_id, behavior, dt], where=[=(dt, _UTF-16LE'2023-08-30':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")])
+- TableSourceScan(table=[[my_catalog, tmp, paimon_test_table13, filter=[equals(dt, '2023-08-30')]]], fields=[user_id, item_id, behavior, dt])

**When scan push down close:**

![image](https://github.com/apache/incubator-paimon/assets/18002496/049829a8-d4d5-48f8-8683-98d5e5c94788)

2023-09-06 14:05:20.166 DEBUG [Flink-DispatcherRest:t-thread-1] o.a.f.t.planner.plan.optimize.program.FlinkChainedProgram - optimize physical_rewrite cost 100 ms.
optimize result:
Sink(table=[default_catalog.default_database.printsink], fields=[user_id, item_id, behavior, dt])
+- Calc(select=[user_id, item_id, behavior, _UTF-16LE'2023-08-30':VARCHAR(2147483647) CHARACTER SET "UTF-16LE" AS dt])
+- ChangelogNormalize(key=[dt, user_id])
+- Exchange(distribution=[hash[dt, user_id]])
+- Calc(select=[user_id, item_id, behavior, dt], where=[=(dt, _UTF-16LE'2023-08-30':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")])
+- TableSourceScan(table=[[my_catalog, tmp, paimon_test_table13]], fields=[user_id, item_id, behavior, dt], hints=[[[OPTIONS options:{scan.push-down=false}]]])

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

No source files or tests are named. Start by reproducing the minimal Flink 1.14 SQL query and compare the optimizer plans with scan pushdown enabled and disabled; determine whether the remaining Calc filter is expected, then use the relevant planner or scan-pushdown tests to define the corrected behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.