apache / apache/doris-spark-connector
[Bug] doris.read.fields parameter does not work in arrow flight sql
- Dominant language
- Java
- Stars
- 107
- Forks
- 126
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 4
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Version
25.1.0
### What's Wrong?
The test.table1 table has c1, c2, c3, c4, c5, c6, c7, c8 fields
Use Spark's DataFrame to get data and set the parameter doris.read.fields
var dataDF = spark.read.format("doris")
.option("doris.table.identifier", "test.table1")
.option("doris.fenodes", dorisHost + ":" + dorisHttpPort)
.option("doris.user", dorisUsername)
.option("doris.password", dorisPassword)
.option("doris.read.fields","c1,c2,c3,c4")
.option("doris.read.mode", "arrow")
.option("doris.read.arrow-flight-sql.port", dorisArrowPort)
.load()
The sql should be: SELECT `c1`,`c2`,`c3`,`c4` FROM `test`.`table1`
But the query sql is: SELECT `c1`,`c2`,`c3`,`c4`,`c5`,`c6`,`c7`,`c8` FROM `test`.`table1`
### What You Expected?
Using Arrow Fillght SQL mode in DataFrame allows you to query the corresponding column data according to the doris.read.fields parameter instead of querying all columns in the table.
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Reproduce the Spark DataFrame read using doris.read.fields and Arrow Flight SQL mode, then trace how the connector builds the SELECT list in that path. Done means the generated query selects only c1, c2, c3, and c4 rather than every column in test.table1.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100