pingcap / pingcap/tidb-binlog

drainer upgrade fail from v7.0.0 to v7.1.1

Open
#1,267 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
291
Forks
131
Avg merge
5m
Merged PRs (30d)
2

Description

Bug Report
Please answer these questions before submitting your issue. Thanks!

What did you do?
install v7.0.0 tidb cluster with drainer
upgrade to v7.1.1

What did you expect to see?

upgrade successfully

What did you see instead?
upgrade fail for:
[2023/09/26 21:21:20.318 +08:00] [ERROR] [main.go:69] ["start drainer server failed"] [error="handle ddl job ID:102, Type:add column, State:synced, SchemaState:public, SchemaID:1, TableID:281474976710650, RowCount:0, ArgLen:0, start time: 2023-09-26 21:11:22.969 +0800 CST, Err:, ErrCount:0, SnapshotVersion:0 failed, the schema info: {\n\t\t"hasImplicitCol": false,\n\t\t"schemaMetaVersion": 0,\n\t\t"schemaNameToID": {\n\t\t\t"loadtest": 88,\n\t\t\t"mysql": 1,\n\t\t\t"test": 2\n\t\t},\n\t\t"tableIDToName": {\n\t\t\t"10": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "db"\n\t\t\t},\n\t\t\t"12": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tables_priv"\n\t\t\t},\n\t\t\t"14": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "columns_priv"\n\t\t\t},\n\t\t\t"16": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "GLOBAL_VARIABLES"\n\t\t\t},\n\t\t\t"18": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb"\n\t\t\t},\n\t\t\t"20": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "help_topic"\n\t\t\t},\n\t\t\t"22": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_meta"\n\t\t\t},\n\t\t\t"24": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_histograms"\n\t\t\t},\n\t\t\t"26": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_buckets"\n\t\t\t},\n\t\t\t"28": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "gc_delete_range"\n\t\t\t},\n\t\t\t"30": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "gc_delete_range_done"\n\t\t\t},\n\t\t\t"32": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_feedback"\n\t\t\t},\n\t\t\t"34": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "role_edges"\n\t\t\t},\n\t\t\t"36": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "default_roles"\n\t\t\t},\n\t\t\t"38": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "bind_info"\n\t\t\t},\n\t\t\t"4": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "user"\n\t\t\t},\n\t\t\t"40": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_top_n"\n\t\t\t},\n\t\t\t"42": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "expr_pushdown_blacklist"\n\t\t\t},\n\t\t\t"44": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "opt_rule_blacklist"\n\t\t\t},\n\t\t\t"46": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_extended"\n\t\t\t},\n\t\t\t"48": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "schema_index_usage"\n\t\t\t},\n\t\t\t"50": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_fm_sketch"\n\t\t\t},\n\t\t\t"52": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "global_grants"\n\t\t\t},\n\t\t\t"54": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "capture_plan_baselines_blacklist"\n\t\t\t},\n\t\t\t"56": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "column_stats_usage"\n\t\t\t},\n\t\t\t"58": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "table_cache_meta"\n\t\t\t},\n\t\t\t"6": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "password_history"\n\t\t\t},\n\t\t\t"60": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "analyze_options"\n\t\t\t},\n\t\t\t"62": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_history"\n\t\t\t},\n\t\t\t"64": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_meta_history"\n\t\t\t},\n\t\t\t"66": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "analyze_jobs"\n\t\t\t},\n\t\t\t"68": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "advisory_locks"\n\t\t\t},\n\t\t\t"70": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_mdl_view"\n\t\t\t},\n\t\t\t"72": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "plan_replayer_status"\n\t\t\t},\n\t\t\t"74": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "plan_replayer_task"\n\t\t\t},\n\t\t\t"76": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_table_locked"\n\t\t\t},\n\t\t\t"78": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_ttl_table_status"\n\t\t\t},\n\t\t\t"8": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "global_priv"\n\t\t\t},\n\t\t\t"80": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_ttl_task"\n\t\t\t},\n\t\t\t"82": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_ttl_job_history"\n\t\t\t},\n\t\t\t"84": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_global_task"\n\t\t\t},\n\t\t\t"86": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "load_data_jobs"\n\t\t\t},\n\t\t\t"90": {\n\t\t\t\t"db-name": "loadtest",\n\t\t\t\t"tbl-name": "sbtest1"\n\t\t\t},\n\t\t\t"91": {\n\t\t\t\t"db-name": "loadtest",\n\t\t\t\t"tbl-name": "sbtest3"\n\t\t\t},\n\t\t\t"92": {\n\t\t\t\t"db-name": "loadtest",\n\t\t\t\t"tbl-name": "sbtest2"\n\t\t\t}\n\t\t}\n\t}: table tidb_background_subtask(281474976710650) not found"] [errorVerbose="table tidb_background_subtask(281474976710650) not found\ngithub.com/pingcap/errors.NotFoundf\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:117\ngithub.com/pingcap/tidb-binlog/drainer.(*Schema).ReplaceTable\n\t/mnt/data1/jenkins/workspace/build-common@2/go/src/github.com/pingcap/tidb-binlog/drainer/schema.go:258\ngithub.com/pingcap/tidb-binlog/drainer.(*Schema).handleDDL\n\t/mnt/data1/jenkins/workspace/build-common@2/go/src/github.com/pingcap/tidb-binlog/drainer/schema.go:540\ngithub.com/pingcap/tidb-binlog/drainer.(*Schema).handlePreviousDDLJobIfNeed\n\t/mnt/data1/jenkins/workspace/build-common@2/go/src/github.com/pingcap/tidb-binlog/drainer/schema.go:319\ngithub.com/pingcap/tidb-binlog/drainer.(*Syncer).run\n\t/mnt/data1/jenkins/workspace/build-common@2/go/src/github.com/pingcap/tidb-binlog/drainer/syncer.go:448\ngithub.com/pingcap/tidb-binlog/drainer.(*Syncer).Start\n\t/mnt/data1/jenkins/workspace/build-common@2/go/src/github.com/pingcap/tidb-binlog/drainer/syncer.go:152\ngithub.com/pingcap/tidb-binlog/drainer.(*Server).Start.func5\n\t/mnt/data1/jenkins/workspace/build-common@2/go/src/github.com/pingcap/tidb-binlog/drainer/server.go:310\ngithub.com/pingcap/tidb-binlog/drainer.(*taskGroup).start.func1\n\t/mnt/data1/jenkins/workspace/build-common@2/go/src/github.com/pingcap/tidb-binlog/drainer/util.go:82\nruntime.goexit\n\t/usr/local/go1.20.5/src/runtime/asm_arm64.s:1172\nhandle ddl job ID:102, Type:add column, State:synced, SchemaState:public, SchemaID:1, TableID:281474976710650, RowCount:0, ArgLen:0, start time: 2023-09-26 21:11:22.969 +0800 CST, Err:, ErrCount:0, SnapshotVersion:0 failed, the schema info: {\n\t\t"hasImplicitCol": false,\n\t\t"schemaMetaVersion": 0,\n\t\t"schemaNameToID": {\n\t\t\t"loadtest": 88,\n\t\t\t"mysql": 1,\n\t\t\t"test": 2\n\t\t},\n\t\t"tableIDToName": {\n\t\t\t"10": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "db"\n\t\t\t},\n\t\t\t"12": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tables_priv"\n\t\t\t},\n\t\t\t"14": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "columns_priv"\n\t\t\t},\n\t\t\t"16": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "GLOBAL_VARIABLES"\n\t\t\t},\n\t\t\t"18": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb"\n\t\t\t},\n\t\t\t"20": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "help_topic"\n\t\t\t},\n\t\t\t"22": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_meta"\n\t\t\t},\n\t\t\t"24": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_histograms"\n\t\t\t},\n\t\t\t"26": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_buckets"\n\t\t\t},\n\t\t\t"28": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "gc_delete_range"\n\t\t\t},\n\t\t\t"30": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "gc_delete_range_done"\n\t\t\t},\n\t\t\t"32": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_feedback"\n\t\t\t},\n\t\t\t"34": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "role_edges"\n\t\t\t},\n\t\t\t"36": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "default_roles"\n\t\t\t},\n\t\t\t"38": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "bind_info"\n\t\t\t},\n\t\t\t"4": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "user"\n\t\t\t},\n\t\t\t"40": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_top_n"\n\t\t\t},\n\t\t\t"42": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "expr_pushdown_blacklist"\n\t\t\t},\n\t\t\t"44": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "opt_rule_blacklist"\n\t\t\t},\n\t\t\t"46": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_extended"\n\t\t\t},\n\t\t\t"48": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "schema_index_usage"\n\t\t\t},\n\t\t\t"50": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_fm_sketch"\n\t\t\t},\n\t\t\t"52": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "global_grants"\n\t\t\t},\n\t\t\t"54": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "capture_plan_baselines_blacklist"\n\t\t\t},\n\t\t\t"56": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "column_stats_usage"\n\t\t\t},\n\t\t\t"58": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "table_cache_meta"\n\t\t\t},\n\t\t\t"6": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "password_history"\n\t\t\t},\n\t\t\t"60": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "analyze_options"\n\t\t\t},\n\t\t\t"62": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_history"\n\t\t\t},\n\t\t\t"64": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_meta_history"\n\t\t\t},\n\t\t\t"66": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "analyze_jobs"\n\t\t\t},\n\t\t\t"68": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "advisory_locks"\n\t\t\t},\n\t\t\t"70": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_mdl_view"\n\t\t\t},\n\t\t\t"72": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "plan_replayer_status"\n\t\t\t},\n\t\t\t"74": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "plan_replayer_task"\n\t\t\t},\n\t\t\t"76": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_table_locked"\n\t\t\t},\n\t\t\t"78": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_ttl_table_status"\n\t\t\t},\n\t\t\t"8": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "global_priv"\n\t\t\t},\n\t\t\t"80": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_ttl_task"\n\t\t\t},\n\t\t\t"82": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_ttl_job_history"\n\t\t\t},\n\t\t\t"84": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_global_task"\n\t\t\t},\n\t\t\t"86": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "load_data_jobs"\n\t\t\t},\n\t\t\t"90": {\n\t\t\t\t"db-name": "loadtest",\n\t\t\t\t"tbl-name": "sbtest1"\n\t\t\t},\n\t\t\t"91": {\n\t\t\t\t"db-name": "loadtest",\n\t\t\t\t"tbl-name": "sbtest3"\n\t\t\t},\n\t\t\t"92": {\n\t\t\t\t"db-name": "loadtest",\n\t\t\t\t"tbl-name": "sbtest2"\n\t\t\t}\n\t\t}\n\t}"]
Please provide the relate downstream type and version of drainer.
(run drainer -V in terminal to get drainer's version)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with drainer/schema.go, especially ReplaceTable, handleDDL, and handlePreviousDDLJobIfNeed, then trace the callers in syncer.go and server.go. Reproduce the v7.0.0-to-v7.1.1 upgrade and determine why the missing tidb_background_subtask table aborts startup; done means the upgrade starts successfully without this failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.