apache / apache/doris-spark-connector

[Bug] data lose when doris2hive using dataFrame

Open
#314 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
107
Forks
126
Avg merge
3d 10h
Merged PRs (30d)
4

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.

### Version

spark-doris-connector: 25.0.1
doris: 3.0.0
spark: 3.0.1

### What's Wrong?

```
val dorisTableIdentifier = "doris_db.doris_table"
val hiveTableName = "hive_db.hive_table"
val timeColumn = "ctime"
val selectedColumnsStr = args(5).trim
val startTime = "2025-05-06 00:00:00"
val endTime = "2025-05-07 00:00:00"

val appName = s"doris-to-hive-$hiveTableName"
val spark = SparkSession.builder()
.appName(appName)
.enableHiveSupport()
.getOrCreate()

// 1. read data from doris
val dorisDF = spark.read
.format("doris")
.option("doris.fenodes", feNodes)
.option("doris.table.identifier", dorisTableIdentifier)
.option("user", user)
.option("password", password)
.load()
.filter(col(timeColumn) >= lit(startTime) && col(timeColumn) < lit(endTime)) // limit timespan
.select(selectedColumns.map(col): _*) // select columns

log.info("doris data count: {}", dorisDF.count())

Thread.sleep(1000)
log.info("doris data count: {}", dorisDF.count())

Thread.sleep(5000)
log.info("doris data count: {}", dorisDF.count())

dorisDF.createOrReplaceTempView("doris_data_detail")

// 2. write to hive
val insertSql =
s"""
|INSERT OVERWRITE TABLE $hiveTableName PARTITION (pt='20250410000000')
|SELECT
|$selectedColumnsStr
|FROM doris_data_detail
|""".stripMargin
log.info("insert hive sql: {}", insertSql)
spark.sql(insertSql)

spark.stop()
```
I used this code to implement doris2hive, and I found that the amount of data in the hive table was smaller than that in the doris table, so I added some logs to record the number of dataframes. The log is as follows:
```
25/05/07 19:43:56 INFO Doris2HiveTask$: doris data count: 68684
25/05/07 19:43:59 INFO Doris2HiveTask$: doris data count: 97918
25/05/07 19:44:05 INFO Doris2HiveTask$: doris data count: 99903
```
the amount in doris:

Image

I am certain that the data count of the doris table has not changed during this period.

Why did this happen , is this a bug?

### What You Expected?

The reason for this situation

### How to Reproduce?

_No response_

### Anything Else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start at the spark.read.format("doris") entry point and trace how repeated dorisDF.count() actions execute before createOrReplaceTempView and spark.sql. Compare the connector's read results with the final Hive write using the versions and time filter shown. Done means identifying the cause of changing counts and adding a reproducible test or documented explanation.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.