通过jdbc读取doris数据过程中出现查询记录条数不正确的问题
- Dominant language
- Java
- Stars
- 9.7k
- Forks
- 2.4k
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 210
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
在通过jdbc方式读取doris数据过程中,读取数据条数不正确
doris表四个字段联合主键,用第一个字段作为分区键(当前字段值具有唯一约束),当前表 记录条数12621964条,通过doris数据源方式读取结果正确,通过jdbc的FixedChunkSplitter分片进行读取数据正确
在用DynamicChunkSplitter分片计算时,读取数据条数不正确
### SeaTunnel Version
2.3.11
### SeaTunnel Config
```conf
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
######
###### This config file is a demonstration of batch processing in SeaTunnel config
######
env {
# You can set spark configuration here
# see available properties defined by spark: https://spark.apache.org/docs/latest/configuration.html#available-properties
#job.mode = BATCH
job.name = "SeaTunnel"
spark.executor.instances = 2
spark.executor.cores = 1
spark.executor.memory = "1024m"
spark.master = local
parallelism = 2
}
source{
Jdbc {
url ="jdbc:mysql://xxxx.xxxx.xxxx.xxxx:9030/database"
driver = "com.mysql.cj.jdbc.Driver"
connection_check_timeout_sec = 100
user = "root"
password = "xxxxxxxx"
split.size="30000"
table_list = [ {
query = "select PARAMETER_ID,WELL_ID from `DGE`.`ml_ach_depth_parameter`"
partition_column = "PARAMETER_ID"
} ]
plugin_output ="ml_ach_depth_parameter_112233_635297931"
}
}
transform {
}
sink {
# choose stdout output plugin to output data to console
Console {
parallelism = 2
}
# you can also you other output plugins, such as sql
# hdfs {
# path = "hdfs://hadoop-cluster-01/nginx/accesslog_processed"
# save_mode = "append"
# }
# If you would like to get more information about how to configure seatunnel and see full list of output plugins,
# please go to https://seatunnel.apache.org/docs/connector-v2/sink
}
```
### Running Command
```shell
wu
```
### Error Exception
```log
wu
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the JDBC source's DynamicChunkSplitter and reproduce the reported configuration against the Doris-compatible MySQL endpoint. Compare its record count with the Doris source and FixedChunkSplitter results; done means DynamicChunkSplitter returns all 12,621,964 records without discrepancy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- backend, data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100