apache / apache/doris-spark-connector
[Bug] org.apache.doris.shaded.org.apache.arrow.vector.VectorSchemaRoot#getFieldVectors中获取到了__DORIS_DELETE_SIGN__, 导致字段数量比较异常
- Dominant language
- Java
- Stars
- 107
- Forks
- 126
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 4
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Version
spark: 3.1.4
doris: doris-2.0.2.1-b387ac2
connector: spark-doris-connector-spark-3.1, version: 25.1.0
### What's Wrong?
org.apache.doris.spark.client.read.RowBatch#readBatch
this.fieldVectors contains __DORIS_DELETE_SIGN__
### What You Expected?
spark read doris table, compare field size need filter invisible_col
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start at org.apache.doris.spark.client.read.RowBatch#readBatch and inspect how fieldVectors is populated and compared with the expected field count. Reproduce with Spark 3.1.4, Doris 2.0.2.1-b387ac2, and connector 25.1.0 if possible. Done means __DORIS_DELETE_SIGN__ or other invisible columns no longer cause the field-count comparison to be abnormal.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100