[Bug] [Connector-V2] Issues with LocalFile CSV Source and Doris Sink in SeaTunnel
- Dominant language
- Java
- Stars
- 9.7k
- Forks
- 2.4k
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 204
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
I am using a LocalFile CSV source and a Doris sink in SeaTunnel, and I encountered several problems during data synchronization:
csv_use_header_line seems ineffective – even though I specified csv_use_header_line = true, it appears to have no effect, and I still need to manually define the schema.
### SeaTunnel Version
2.3.8
### SeaTunnel Config
```conf
env {
parallelism = 3
job.mode = "BATCH"
}
source {
LocalFile {
path = "/data/home/conf/test.csv"
file_format_type = "csv"
skip_header_row_number = 1
csv_use_header_line = true
schema = {
fields {
platform = "string"
platform_order_no = "string"
}
}
}
}
sink {
Doris {
fenodes = "${doris.fenodes}"
username = "${doris.username}"
password = "${doris.password}"
database = "${doris.database}"
table = "test"
sink.enable-2pc = "true"
sink.label-prefix = "test_csv"
sink.enable-delete = "true"
doris.config = {
format = "csv"
column_separator = ","
}
}
}
```
### Running Command
```shell
$SEATUNNEL_HOME/bin/seatunnel.sh --config ${config} --variable $secret
```
### Error Exception
```log
When I only define csv_use_header_line = true and there is no schema, this error is reported.
Caused by: org.apache.seatunnel.api.configuration.util.OptionValidationException: ErrorCode:[API-02], ErrorDescription:[Option item validate failed] - There are unconfigured options, the options('schema') are required because ['file_format_type' == TEXT || 'file_format_type' == JSON || 'file_format_type' == EXCEL || 'file_format_type' == CSV || 'file_format_type' == XML] is true.
at org.apache.seatunnel.api.configuration.util.ConfigValidator.validate(ConfigValidator.java:200)
at org.apache.seatunnel.api.configuration.util.ConfigValidator.validate(ConfigValidator.java:107)
at org.apache.seatunnel.api.configuration.util.ConfigValidator.validate(ConfigValidator.java:47)
at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:111)
at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:74)
... 7 more
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the LocalFile source configuration and the validation path shown in ConfigValidator.java, FactoryUtil.createAndPrepareSource, and the supplied SeaTunnel command. Reproduce the 2.3.8 CSV configuration with csv_use_header_line enabled and no schema, then trace why schema remains required. Done means the header-based configuration is accepted and the CSV data synchronizes to the Doris sink without requiring a manually defined schema.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100