alibaba / alibaba/DataX

向db2数据库入库慢

Open
#548 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

向db2(10.5 fp8)插入数据,无论数据源是csv还是另一个db2库,rdbmswriter速度都很慢,单个channel约1000rec/s. 数据源为db2时, reader端可以跑满千兆带宽,BatchSize=4096,JVM虚机4G, job配置如下:
```
"writer": {
"name": "rdbmswriter",
"parameter": {
"connection": [
{
"jdbcUrl": "jdbc:db2://********",
"table": ["********"]
}
],
"username": "********",
"password": "********",
"table": "********",
"column": [
"STATIS_MONTH",
"SUBS_ID",
"GRP_SUBS_ID",
"PHONE_NO",
"ISPRIMA",
"START_TIME"
],
"#preSql": [
"delete from ********;"
]
}
}
```
多channel下,基本上等于1000*n rec/s,开8个channel时,偶发ERROR=-4229,然后批量插入失败, 变为逐条插入. 单个channel时日志中未发现此情况.
插入速度实在是太慢了,求教解决方法.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the rdbmswriter implementation and its batch-insert error handling, using the supplied DB2 JDBC configuration, BatchSize=4096, and single-channel versus eight-channel results as the reproduction. Check DB2 ERROR=-4229 and the fallback to row-by-row inserts; done means explaining the throughput limit and correctly handling the reported batch failure, with a regression check if the project provides one.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.