apache / apache/doris-spark-connector

[Bug] Duplicate column: ??\n\n\t0# doris::Status doris::Status::create<true>(doris::TStatus const&)

Open
#238 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
107
Forks
126
Avg merge
3d 10h
Merged PRs (30d)
4

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.

### Version

doris-spark-connector:master
spark 3.1.2

### What's Wrong?

"TxnId": 1425436,
"Label": "spark_streamload_20241106_205702_9dd5634a98214aa08d5bb0254bee8930",
"Comment": "",
"TwoPhaseCommit": "false",
"Status": "Fail",
"Message": "[ANALYSIS_ERROR]TStatus: errCode = 2, detailMessage = Duplicate column: ??\n\n\t0# doris::Status doris::Status::create(doris::TStatus const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187\n\t1# doris::StreamLoadAction::_process_put(doris::HttpRequest*, std::shared_ptr) at /opt/tiger/compile_path/src/code.byted.org/emr/doris/be/src/common/status.h:446\n\t2# doris::StreamLoadAction::_on_header(doris::HttpRequest*, std::shared_ptr) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:701\n\t3# doris::StreamLoadAction::on_header(doris::HttpRequest*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:701\n\t4# doris::EvHttpServer::on_header(evhttp_request*) at /opt/tiger/compile_path/src/code.byted.org/emr/doris/be/src/http/ev_http_server.cpp:255\n\t5# ?\n\t6# bufferevent_run_readcb_\n\t7# ?\n\t8# ?\n\t9# ?\n\t10# ?\n\t11# std::_Function_handler::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/atomicity.h:98\n\t12# doris::ThreadPool::dispatch_thread() at /opt/tiger/compile_path/src/code.byted.org/emr/doris/be/src/util/threadpool.cpp:0\n\t13# doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562\n\t14# start_thread\n\t15# __clone\n"

### What You Expected?

Expect to support writing Chinese field names

### How to Reproduce?

![image](https://github.com/user-attachments/assets/6cddfa4f-6354-46c0-83ad-d7eefbc14c72)

https://github.com/apache/doris-spark-connector/blob/master/spark-doris-connector/src/test/scala/org/apache/doris/spark/sql/TestSparkConnector.scala

def dataframeWriteTest(): Unit = {
val session = SparkSession.builder().master("local[*]").getOrCreate()
val df = session.createDataFrame(Seq(
(1,"zhangsan-1", 18),
(2,"lisi-2", 19),
(3,"wangwu-1", 20)
)).toDF("id","名称","年龄")

df.write
.format("doris")
.option("doris.query.port", "9030")
.option("doris.fenodes", dorisFeNodes)
.option("doris.table.identifier", dorisTable)
.option("user", dorisUser)
.option("password", dorisPwd)
.option("doris.table.pk.keys", "id")
.option("doris.connection.jdbc.url", "jdbc:mysql://localhost:9030")
//.option("doris.write.fields", "id,名称,年龄")
.option("sink.batch.size",2)
.option("sink.max-retries",2)
//.mode("overwrite")
.save()

//set global enable_unicode_name_support=true

session.stop()
}

### Anything Else?

No response

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start with spark-doris-connector/src/test/scala/org/apache/doris/spark/sql/TestSparkConnector.scala and its dataframeWriteTest case, which reproduces the failure using the Chinese fields 名称 and 年龄. Run that case and trace the connector's write path to determine why the duplicate-column error occurs; done means the example writes successfully with those field names.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.