apache / apache/doris-spark-connector

基于spark 数据集成工具datatunnel,支持doris

Open
#262 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
107
Forks
126
Avg merge
3d 10h
Merged PRs (30d)
4

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.

### Description

一些海外客户,在使用datatunnel。已有一个买咖啡客户,基于dataunnel 读写redshift,最近在测试doris。

https://github.com/melin/datatunnel/tree/master/examples/src/main/kotlin/com/superior/datatunnel/examples/doris
```
DATATUNNEL SOURCE("mysql") OPTIONS (
username = "root",
password = "",
host = '172.18.6.181',
port = 3306,
databaseName = 'demos',
tableName = 'account',
columns = ["*"]
)
SINK("doris") OPTIONS (
feEnpoints="172.18.5.44:18030",
username = "root",
password = "",
databaseName = "testdb",
tableName = "ods_account"
)
```

```
DATATUNNEL SOURCE("kafka") OPTIONS (
format="text",
subscribe = "doris_demos",
servers = "172.18.6.181:9092",
'properties.kafka.security.protocol' = 'SASL_PLAINTEXT',
'properties.kafka.sasl.mechanism' = 'PLAIN',
'properties.kafka.sasl.jaas.config' = 'org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin2024";',
includeHeaders = false,
startingOffsets = 'earliest',
checkpointLocation = "/Users/melin/stream_checkpoint/datatunnel/doris_demos",
columns = ['*']
)
SINK("doris") OPTIONS (
feEnpoints="172.18.5.44:18030",
username = "root",
password = "",
databaseName = "testdb",
tableName = "table_hash",
passthrough = true,
fileFormat = "json",
columns = ["k1", "k2", "k3", "k4"]
)
```

### Solution

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start with the linked datatunnel Doris example at examples/src/main/kotlin/com/superior/datatunnel/examples/doris and compare its MySQL-to-Doris and Kafka-to-Doris configurations with this connector’s integration points. Confirm the intended Spark/Doris support scope and define completion through working equivalent examples or tests; the issue names no repository implementation file or test.

Written by the indexing model from the issue text.

Assessment

Tech stack
kotlin, spark
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.