apache / apache/doris-flink-connector

[Bug] DorisSource createEnumerator error when has filter condition(pushDown)

Open
#376 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
385
Forks
279
Avg merge
2d 2h
Merged PRs (30d)
6

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.

### Version

1.6.0

### What's Wrong?

my sql exsit filter query but column name is "null".
Here is the detailed log:

2024-05-06 09:45:01.836 [flink-akka.actor.default-dispatcher-8970] INFO org.apache.doris.flink.source.DorisSource - Query SQL Sending to Doris FE is: 'select `FINISHEDFLAG_`, `FIELD0006_`, `FIELD0031_`, `FIELD0030_` from `qixi_ods`.`ods_dcb850f793dd87c6ae90e3aa81875608` where (null = 4)'.
2024-05-06 09:45:01.836 [flink-akka.actor.default-dispatcher-8970] INFO org.apache.doris.flink.source.DorisSource - Send request to Doris FE 'http://192.168.9.200:8030/api/qixi_ods/ods_dcb850f793dd87c6ae90e3aa81875608/_query_plan' with user 'root'.
2024-05-06 09:45:01.843 [flink-akka.actor.default-dispatcher-8970] ERROR org.apache.doris.flink.source.DorisSource - Doris FE's response cannot map to schema. res: {"exception":"PlannershouldplanjustonlyoneScanNodebutfound[0]","status":500}

Caused by: org.apache.doris.flink.exception.DorisException: Doris FE's response cannot map to schema. res: {"exception":"PlannershouldplanjustonlyoneScanNodebutfound[0]","status":500}
at org.apache.doris.flink.rest.RestService.getQueryPlan(RestService.java:653)
at org.apache.doris.flink.rest.RestService.findPartitions(RestService.java:620)
at org.apache.doris.flink.source.DorisSource.createEnumerator(DorisSource.java:97)
at org.apache.flink.runtime.source.coordinator.SourceCoordinator.start(SourceCoordinator.java:222)

### What You Expected?

no error. BTW how to disable the pushdown?

### How to Reproduce?

_No response_

### Anything Else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start at org.apache.doris.flink.source.DorisSource.createEnumerator, then trace the calls to RestService.findPartitions and RestService.getQueryPlan shown in the stack trace. Reproduce the filter-pushdown query that generates `where (null = 4)` and determine the expected behavior, including whether pushdown can be disabled; done means the enumerator no longer fails for the reported case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.