pingcap / pingcap/tidb

Dumpling times out and fails to generate metadata on large dumps

Open
#51,661 0 comments 0 reactions 0 assignees View on GitHub
affects-8.5 component/dumpling may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

When dumping a large dataset from MySQL (Mariadb 10.4.x) while the full dump is successful (all the tables / data are fetched) at the end dumpling fails to create the 'metadata' file/information.

### 1. Minimal reproduce step (Required)

`dumpling -h 10.10.10.2 -P 3306 -u dm -t 16 -F 256MB -B database --order-by-primary-key=FALSE --password "xxxx" -o /data/dump/`

### 2. What did you expect to see? (Required)

metadata file in the dump location.

### 3. What did you see instead (Required)

```
[2024/03/10 19:38:09.948 +02:00] [INFO] [status.go:37] [progress] [tables="1246/1247 (99.9%)"] ["finished rows"=136065787435] ["estimate total rows"=130357542865] ["finished size"=11.78TB] ["average speed(MiB/s)"=15.141337335299763] ["recent speed bps"=15876827.553732995] ["chunks progress"="99.92 %"]
[2024/03/10 19:40:09.949 +02:00] [INFO] [status.go:37] [progress] [tables="1246/1247 (99.9%)"] ["finished rows"=136066390862] ["estimate total rows"=130357542865] ["finished size"=11.78TB] ["average speed(MiB/s)"=2.5163668416159424] ["recent speed bps"=2638598.0351836598] ["chunks progress"="99.92 %"]
[2024/03/10 19:42:09.948 +02:00] [INFO] [status.go:37] [progress] [tables="1246/1247 (99.9%)"] ["finished rows"=136066390862] ["estimate total rows"=130357542865] ["finished size"=11.78TB] ["average speed(MiB/s)"=0] ["recent speed bps"=2638598.0351836598] ["chunks progress"="99.92 %"]
[2024/03/10 19:44:09.949 +02:00] [INFO] [status.go:37] [progress] [tables="1246/1247 (99.9%)"] ["finished rows"=136066390862] ["estimate total rows"=130357542865] ["finished size"=11.78TB] ["average speed(MiB/s)"=0] ["recent speed bps"=2638598.0351836598] ["chunks progress"="99.92 %"]
[2024/03/10 19:46:09.950 +02:00] [INFO] [status.go:37] [progress] [tables="1246/1247 (99.9%)"] ["finished rows"=136066390862] ["estimate total rows"=130357542865] ["finished size"=11.78TB] ["average speed(MiB/s)"=0] ["recent speed bps"=2638598.0351836598] ["chunks progress"="99.92 %"]
[2024/03/10 19:48:09.949 +02:00] [INFO] [status.go:37] [progress] [tables="1246/1247 (99.9%)"] ["finished rows"=136066390862] ["estimate total rows"=130357542865] ["finished size"=11.78TB] ["average speed(MiB/s)"=0] ["recent speed bps"=2638598.0351836598] ["chunks progress"="99.92 %"]
[2024/03/10 19:50:09.950 +02:00] [INFO] [status.go:37] [progress] [tables="1246/1247 (99.9%)"] ["finished rows"=136066390862] ["estimate total rows"=130357542865] ["finished size"=11.78TB] ["average speed(MiB/s)"=0] ["recent speed bps"=2638598.0351836598] ["chunks progress"="99.92 %"]
[2024/03/10 19:52:09.950 +02:00] [INFO] [status.go:37] [progress] [tables="1246/1247 (99.9%)"] ["finished rows"=136066390862] ["estimate total rows"=130357542865] ["finished size"=11.78TB] ["average speed(MiB/s)"=0] ["recent speed bps"=2638598.0351836598] ["chunks progress"="99.92 %"]
mysql] 2024/03/10 19:54:03 packets.go:73: read tcp 10.10.10.1:57786->10.10.10.2:3306: i/o timeout
[mysql] 2024/03/10 19:54:03 packets.go:427: busy buffer
[2024/03/10 19:54:03.863 +02:00] [WARN] [writer_util.go:194] ["fail to dumping table(chunk), will revert some metrics and start a retry if possible"] [database=mapon] [table=g_routes] ["finished rows"=450577] ["finished size"=222833842] [error="invalid connection"]
[2024/03/10 19:54:03.863 +02:00] [INFO] [collector.go:224] ["units canceled"] [cancel-unit=0]

[2024/03/10 19:54:03.863 +02:00] [INFO] [collector.go:225] ["backup failed summary"] [total-ranges=1] [ranges-succeed=0] [ranges-failed=1] [unit-name="dump table data"] [error="invalid connection"] [errorVerbose="invalid connection
github.com/pingcap/errors.AddStack
github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174
github.com/pingcap/errors.Trace
github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15
github.com/pingcap/tidb/dumpling/export.(*rowIter).Error
github.com/pingcap/tidb/dumpling/export/ir_impl.go:42
github.com/pingcap/tidb/dumpling/export.WriteInsert
github.com/pingcap/tidb/dumpling/export/writer_util.go:285
github.com/pingcap/tidb/dumpling/export.FileFormat.WriteInsert
github.com/pingcap/tidb/dumpling/export/writer_util.go:667
github.com/pingcap/tidb/dumpling/export.(*Writer).tryToWriteTableData
github.com/pingcap/tidb/dumpling/export/writer.go:243
github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData.func1
github.com/pingcap/tidb/dumpling/export/writer.go:228
github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1
github.com/pingcap/tidb/br/pkg/utils/retry.go:59
github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]
github.com/pingcap/tidb/br/pkg/utils/retry.go:77
github.com/pingcap/tidb/br/pkg/utils.WithRetry
github.com/pingcap/tidb/br/pkg/utils/retry.go:58
github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData
github.com/pingcap/tidb/dumpling/export/writer.go:192
github.com/pingcap/tidb/dumpling/export.(*Writer).handleTask
github.com/pingcap/tidb/dumpling/export/writer.go:115
github.com/pingcap/tidb/dumpling/export.(*Writer).run
github.com/pingcap/tidb/dumpling/export/writer.go:93
github.com/pingcap/tidb/dumpling/export.(*Dumper).startWriters.func4
github.com/pingcap/tidb/dumpling/export/dump.go:376
golang.org/x/sync/errgroup.(*Group).Go.func1
golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75
runtime.goexit
runtime/asm_amd64.s:1650"]

[2024/03/10 19:54:03.863 +02:00] [ERROR] [main.go:78] ["dump failed error stack info"] [error="invalid connection"] [errorVerbose="invalid connection
github.com/pingcap/errors.AddStack
github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174
github.com/pingcap/errors.Trace
github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15
github.com/pingcap/tidb/dumpling/export.(*rowIter).Error
github.com/pingcap/tidb/dumpling/export/ir_impl.go:42
github.com/pingcap/tidb/dumpling/export.WriteInsert
github.com/pingcap/tidb/dumpling/export/writer_util.go:285
github.com/pingcap/tidb/dumpling/export.FileFormat.WriteInsert
github.com/pingcap/tidb/dumpling/export/writer_util.go:667
github.com/pingcap/tidb/dumpling/export.(*Writer).tryToWriteTableData
github.com/pingcap/tidb/dumpling/export/writer.go:243
github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData.func1
github.com/pingcap/tidb/dumpling/export/writer.go:228
github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1
github.com/pingcap/tidb/br/pkg/utils/retry.go:59
github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]
github.com/pingcap/tidb/br/pkg/utils/retry.go:77
github.com/pingcap/tidb/br/pkg/utils.WithRetry
github.com/pingcap/tidb/br/pkg/utils/retry.go:58
github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData
github.com/pingcap/tidb/dumpling/export/writer.go:192
github.com/pingcap/tidb/dumpling/export.(*Writer).handleTask
github.com/pingcap/tidb/dumpling/export/writer.go:115
github.com/pingcap/tidb/dumpling/export.(*Writer).run
github.com/pingcap/tidb/dumpling/export/writer.go:93
github.com/pingcap/tidb/dumpling/export.(*Dumper).startWriters.func4
github.com/pingcap/tidb/dumpling/export/dump.go:376
golang.org/x/sync/errgroup.(*Group).Go.func1
golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75
runtime.goexit
runtime/asm_amd64.s:1650"]

dump failed: invalid connection

```

### 4. What is your TiDB version? (Required)

```
dumpling -V
Release version: v7.5.0
Git commit hash: 069631e2ecfedc000ffb92c67207bea81380f020
Git branch: heads/refs/tags/v7.5.0
Build timestamp: 2023-11-24 08:41:54Z
Go version: go version go1.21.3 linux/amd64

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.