apache / apache/doris-spark-connector
[Bug] The write speed is too slow due to unreasonable locking time during retries
- Dominant language
- Java
- Stars
- 107
- Forks
- 126
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 4
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Version
spark-doris-connector-spark-3.3-25.0.1
### What's Wrong?
A single spark task writes doris at a speed of about 20 records/s, which is too slow.
### What You Expected?
The write speed is close to streamload.
### How to Reproduce?
Use spark 3.3.1 to configure doris catalog and write data to doris.
### Anything Else?

I found that in the Retry class, even if the write is successful, it will still lock for a period of time. The default time is 50ms, which corresponds to the time of 20 records/s. I understand that if the execution is successful, there should be no need to lock for a period of time?
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start with the Retry class implicated in the report and reproduce the write using Spark 3.3.1 with a Doris catalog. Observe whether successful writes still incur the default 50 ms lock interval, and compare the resulting throughput with the expected streamload-like speed. Done means successful retries no longer add unnecessary locking time and the reported write-speed issue is resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100