[Bug] The filter calc operator exit when filter push down
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Paimon version
0.5
### Compute Engine
flink1.14
### Minimal reproduce step
CREATE CATALOG my_catalog
WITH
('type' = 'paimon', 'metastore' = 'hive' ....
) ;
CREATE
TABLE printsink
(
user_id BIGINT,
item_id BIGINT,
behavior STRING,
dt STRING
)
WITH
( 'connector' = 'print'
) ;
INSERT INTO printsink
SELECT
user_id,
item_id,
behavior,
dt
FROM
my_catalog.tmp.paimon_test_table13 where dt = '2023-08-30'
### What doesn't meet your expectations?
The filter calc operator exit when filter push down, is this correct ?

optimize result:
Sink(table=[default_catalog.default_database.printsink], fields=[user_id, item_id, behavior, dt])
+- Calc(select=[user_id, item_id, behavior, _UTF-16LE'2023-08-30':VARCHAR(2147483647) CHARACTER SET "UTF-16LE" AS dt])
+- ChangelogNormalize(key=[dt, user_id])
+- Exchange(distribution=[hash[dt, user_id]])
+- Calc(select=[user_id, item_id, behavior, dt], where=[=(dt, _UTF-16LE'2023-08-30':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")])
+- TableSourceScan(table=[[my_catalog, tmp, paimon_test_table13, filter=[equals(dt, '2023-08-30')]]], fields=[user_id, item_id, behavior, dt])
**When scan push down close:**

2023-09-06 14:05:20.166 DEBUG [Flink-DispatcherRest:t-thread-1] o.a.f.t.planner.plan.optimize.program.FlinkChainedProgram - optimize physical_rewrite cost 100 ms.
optimize result:
Sink(table=[default_catalog.default_database.printsink], fields=[user_id, item_id, behavior, dt])
+- Calc(select=[user_id, item_id, behavior, _UTF-16LE'2023-08-30':VARCHAR(2147483647) CHARACTER SET "UTF-16LE" AS dt])
+- ChangelogNormalize(key=[dt, user_id])
+- Exchange(distribution=[hash[dt, user_id]])
+- Calc(select=[user_id, item_id, behavior, dt], where=[=(dt, _UTF-16LE'2023-08-30':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")])
+- TableSourceScan(table=[[my_catalog, tmp, paimon_test_table13]], fields=[user_id, item_id, behavior, dt], hints=[[[OPTIONS options:{scan.push-down=false}]]])
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
No source files or tests are named. Start by reproducing the minimal Flink 1.14 SQL query and compare the optimizer plans with scan pushdown enabled and disabled; determine whether the remaining Calc filter is expected, then use the relevant planner or scan-pushdown tests to define the corrected behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100