apache / apache/doris-spark-connector

[Bug] doris.read.fields parameter does not work in arrow flight sql

Open
#333 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
107
Forks
126
Avg merge
3d 10h
Merged PRs (30d)
4

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.

### Version

25.1.0

### What's Wrong?

The test.table1 table has c1, c2, c3, c4, c5, c6, c7, c8 fields

Use Spark's DataFrame to get data and set the parameter doris.read.fields
var dataDF = spark.read.format("doris")
.option("doris.table.identifier", "test.table1")
.option("doris.fenodes", dorisHost + ":" + dorisHttpPort)
.option("doris.user", dorisUsername)
.option("doris.password", dorisPassword)
.option("doris.read.fields","c1,c2,c3,c4")
.option("doris.read.mode", "arrow")
.option("doris.read.arrow-flight-sql.port", dorisArrowPort)
.load()

The sql should be: SELECT `c1`,`c2`,`c3`,`c4` FROM `test`.`table1`
But the query sql is: SELECT `c1`,`c2`,`c3`,`c4`,`c5`,`c6`,`c7`,`c8` FROM `test`.`table1`

### What You Expected?

Using Arrow Fillght SQL mode in DataFrame allows you to query the corresponding column data according to the doris.read.fields parameter instead of querying all columns in the table.

### How to Reproduce?

_No response_

### Anything Else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Reproduce the Spark DataFrame read using doris.read.fields and Arrow Flight SQL mode, then trace how the connector builds the SELECT list in that path. Done means the generated query selects only c1, c2, c3, and c4 rather than every column in test.table1.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.