apache / apache/doris-spark-connector

[Bug] org.apache.doris.shaded.org.apache.arrow.vector.VectorSchemaRoot#getFieldVectors中获取到了__DORIS_DELETE_SIGN__, 导致字段数量比较异常

Open
#349 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
107
Forks
126
Avg merge
3d 10h
Merged PRs (30d)
4

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.

### Version

spark: 3.1.4
doris: doris-2.0.2.1-b387ac2
connector: spark-doris-connector-spark-3.1, version: 25.1.0

### What's Wrong?

org.apache.doris.spark.client.read.RowBatch#readBatch
this.fieldVectors contains __DORIS_DELETE_SIGN__

### What You Expected?

spark read doris table, compare field size need filter invisible_col

### How to Reproduce?

_No response_

### Anything Else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start at org.apache.doris.spark.client.read.RowBatch#readBatch and inspect how fieldVectors is populated and compared with the expected field count. Reproduce with Spark 3.1.4, Doris 2.0.2.1-b387ac2, and connector 25.1.0 if possible. Done means __DORIS_DELETE_SIGN__ or other invisible columns no longer cause the field-count comparison to be abnormal.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.