apache / apache/seatunnel

[Bug] [Connector-V2] Issues with LocalFile CSV Source and Doris Sink in SeaTunnel

Open
#9,948 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.

### What happened

I am using a LocalFile CSV source and a Doris sink in SeaTunnel, and I encountered several problems during data synchronization:

csv_use_header_line seems ineffective – even though I specified csv_use_header_line = true, it appears to have no effect, and I still need to manually define the schema.

### SeaTunnel Version

2.3.8

### SeaTunnel Config

```conf
env {
parallelism = 3
job.mode = "BATCH"
}

source {
LocalFile {
path = "/data/home/conf/test.csv"
file_format_type = "csv"
skip_header_row_number = 1
csv_use_header_line = true
schema = {
fields {
platform = "string"
platform_order_no = "string"
}
}
}
}

sink {
Doris {
fenodes = "${doris.fenodes}"
username = "${doris.username}"
password = "${doris.password}"
database = "${doris.database}"
table = "test"
sink.enable-2pc = "true"
sink.label-prefix = "test_csv"
sink.enable-delete = "true"
doris.config = {
format = "csv"
column_separator = ","
}
}
}
```

### Running Command

```shell
$SEATUNNEL_HOME/bin/seatunnel.sh --config ${config} --variable $secret
```

### Error Exception

```log
When I only define csv_use_header_line = true and there is no schema, this error is reported.

Caused by: org.apache.seatunnel.api.configuration.util.OptionValidationException: ErrorCode:[API-02], ErrorDescription:[Option item validate failed] - There are unconfigured options, the options('schema') are required because ['file_format_type' == TEXT || 'file_format_type' == JSON || 'file_format_type' == EXCEL || 'file_format_type' == CSV || 'file_format_type' == XML] is true.
at org.apache.seatunnel.api.configuration.util.ConfigValidator.validate(ConfigValidator.java:200)
at org.apache.seatunnel.api.configuration.util.ConfigValidator.validate(ConfigValidator.java:107)
at org.apache.seatunnel.api.configuration.util.ConfigValidator.validate(ConfigValidator.java:47)
at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:111)
at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:74)
... 7 more
```

### Zeta or Flink or Spark Version

_No response_

### Java or Scala Version

_No response_

### Screenshots

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the LocalFile source configuration and the validation path shown in ConfigValidator.java, FactoryUtil.createAndPrepareSource, and the supplied SeaTunnel command. Reproduce the 2.3.8 CSV configuration with csv_use_header_line enabled and no schema, then trace why schema remains required. Done means the header-based configuration is accepted and the CSV data synchronizes to the Doris sink without requiring a manually defined schema.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.