apache / apache/doris-spark-connector

[Bug] spark connector, when doris table is non-paritioned. where clause is useless

Open
#223 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
107
Forks
126
Avg merge
3d 10h
Merged PRs (30d)
4

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.

### Version

1.2.9

### What's Wrong?

doris表是非分区表的情况下,用spark connector去读取到df以后。无法使用where 算子。写了也不生效,变成了全部读取,然后全表写入。

![image](https://github.com/user-attachments/assets/e28a8811-a208-4b20-a76e-554186aade00)

### What You Expected?

组件bug。

### How to Reproduce?

1.准备一个非分区的doris表。
2.插入100条数据
3.使用spark doris connector读取到df
4.随便加一个where算子,带上条件
最后会发现条件没用

### Anything Else?

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Reproduce with Doris 1.2.9 using a non-partitioned table containing 100 rows, read it through the Spark connector, and apply a where condition. Trace the connector's read and predicate-handling path; done means the condition is honored so only matching rows are read rather than the entire table.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.