apache / apache/seatunnel

通过jdbc读取doris数据过程中出现查询记录条数不正确的问题

Open
#9,724 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 17h
Merged PRs (30d)
210

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.

### What happened

在通过jdbc方式读取doris数据过程中,读取数据条数不正确
doris表四个字段联合主键,用第一个字段作为分区键(当前字段值具有唯一约束),当前表 记录条数12621964条,通过doris数据源方式读取结果正确,通过jdbc的FixedChunkSplitter分片进行读取数据正确
在用DynamicChunkSplitter分片计算时,读取数据条数不正确

### SeaTunnel Version

2.3.11

### SeaTunnel Config

```conf
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

######
###### This config file is a demonstration of batch processing in SeaTunnel config
######

env {
# You can set spark configuration here
# see available properties defined by spark: https://spark.apache.org/docs/latest/configuration.html#available-properties
#job.mode = BATCH
job.name = "SeaTunnel"
spark.executor.instances = 2
spark.executor.cores = 1
spark.executor.memory = "1024m"
spark.master = local
parallelism = 2
}

source{
Jdbc {
url ="jdbc:mysql://xxxx.xxxx.xxxx.xxxx:9030/database"
driver = "com.mysql.cj.jdbc.Driver"
connection_check_timeout_sec = 100
user = "root"
password = "xxxxxxxx"
split.size="30000"
table_list = [ {
query = "select PARAMETER_ID,WELL_ID from `DGE`.`ml_ach_depth_parameter`"
partition_column = "PARAMETER_ID"
} ]
plugin_output ="ml_ach_depth_parameter_112233_635297931"
}
}

transform {

}

sink {
# choose stdout output plugin to output data to console
Console {
parallelism = 2
}

# you can also you other output plugins, such as sql
# hdfs {
# path = "hdfs://hadoop-cluster-01/nginx/accesslog_processed"
# save_mode = "append"
# }

# If you would like to get more information about how to configure seatunnel and see full list of output plugins,
# please go to https://seatunnel.apache.org/docs/connector-v2/sink
}
```

### Running Command

```shell
wu
```

### Error Exception

```log
wu
```

### Zeta or Flink or Spark Version

_No response_

### Java or Scala Version

_No response_

### Screenshots

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the JDBC source's DynamicChunkSplitter and reproduce the reported configuration against the Doris-compatible MySQL endpoint. Compare its record count with the Doris source and FixedChunkSplitter results; done means DynamicChunkSplitter returns all 12,621,964 records without discrepancy.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
backend, data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.