apache / apache/doris-spark-connector

[Bug] The write speed is too slow due to unreasonable locking time during retries

Open
#316 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
107
Forks
126
Avg merge
3d 10h
Merged PRs (30d)
4

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.

### Version

spark-doris-connector-spark-3.3-25.0.1

### What's Wrong?

A single spark task writes doris at a speed of about 20 records/s, which is too slow.

### What You Expected?

The write speed is close to streamload.

### How to Reproduce?

Use spark 3.3.1 to configure doris catalog and write data to doris.

### Anything Else?

![Image](https://github.com/user-attachments/assets/ea47a904-b947-45eb-9011-a94aaff13810)
I found that in the Retry class, even if the write is successful, it will still lock for a period of time. The default time is 50ms, which corresponds to the time of 20 records/s. I understand that if the execution is successful, there should be no need to lock for a period of time?

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start with the Retry class implicated in the report and reproduce the write using Spark 3.3.1 with a Doris catalog. Observe whether successful writes still incur the default 50 ms lock interval, and compare the resulting throughput with the expected streamload-like speed. Done means successful retries no longer add unnecessary locking time and the reported write-speed issue is resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.