apache / apache/doris-spark-connector
基于spark 数据集成工具datatunnel,支持doris
- Dominant language
- Java
- Stars
- 107
- Forks
- 126
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 4
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Description
一些海外客户,在使用datatunnel。已有一个买咖啡客户,基于dataunnel 读写redshift,最近在测试doris。
https://github.com/melin/datatunnel/tree/master/examples/src/main/kotlin/com/superior/datatunnel/examples/doris
```
DATATUNNEL SOURCE("mysql") OPTIONS (
username = "root",
password = "",
host = '172.18.6.181',
port = 3306,
databaseName = 'demos',
tableName = 'account',
columns = ["*"]
)
SINK("doris") OPTIONS (
feEnpoints="172.18.5.44:18030",
username = "root",
password = "",
databaseName = "testdb",
tableName = "ods_account"
)
```
```
DATATUNNEL SOURCE("kafka") OPTIONS (
format="text",
subscribe = "doris_demos",
servers = "172.18.6.181:9092",
'properties.kafka.security.protocol' = 'SASL_PLAINTEXT',
'properties.kafka.sasl.mechanism' = 'PLAIN',
'properties.kafka.sasl.jaas.config' = 'org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin2024";',
includeHeaders = false,
startingOffsets = 'earliest',
checkpointLocation = "/Users/melin/stream_checkpoint/datatunnel/doris_demos",
columns = ['*']
)
SINK("doris") OPTIONS (
feEnpoints="172.18.5.44:18030",
username = "root",
password = "",
databaseName = "testdb",
tableName = "table_hash",
passthrough = true,
fileFormat = "json",
columns = ["k1", "k2", "k3", "k4"]
)
```
### Solution
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start with the linked datatunnel Doris example at examples/src/main/kotlin/com/superior/datatunnel/examples/doris and compare its MySQL-to-Doris and Kafka-to-Doris configurations with this connector’s integration points. Confirm the intended Spark/Doris support scope and define completion through working equivalent examples or tests; the issue names no repository implementation file or test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kotlin, spark
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100