drainer upgrade fail from v7.0.0 to v7.1.1
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 291
- Forks
- 131
- Avg merge
- 5m
- Merged PRs (30d)
- 2
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
What did you do?
install v7.0.0 tidb cluster with drainer
upgrade to v7.1.1
What did you expect to see?
upgrade successfully
What did you see instead?
upgrade fail for:
[2023/09/26 21:21:20.318 +08:00] [ERROR] [main.go:69] ["start drainer server failed"] [error="handle ddl job ID:102, Type:add column, State:synced, SchemaState:public, SchemaID:1, TableID:281474976710650, RowCount:0, ArgLen:0, start time: 2023-09-26 21:11:22.969 +0800 CST, Err:, ErrCount:0, SnapshotVersion:0 failed, the schema info: {\n\t\t"hasImplicitCol": false,\n\t\t"schemaMetaVersion": 0,\n\t\t"schemaNameToID": {\n\t\t\t"loadtest": 88,\n\t\t\t"mysql": 1,\n\t\t\t"test": 2\n\t\t},\n\t\t"tableIDToName": {\n\t\t\t"10": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "db"\n\t\t\t},\n\t\t\t"12": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tables_priv"\n\t\t\t},\n\t\t\t"14": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "columns_priv"\n\t\t\t},\n\t\t\t"16": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "GLOBAL_VARIABLES"\n\t\t\t},\n\t\t\t"18": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb"\n\t\t\t},\n\t\t\t"20": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "help_topic"\n\t\t\t},\n\t\t\t"22": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_meta"\n\t\t\t},\n\t\t\t"24": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_histograms"\n\t\t\t},\n\t\t\t"26": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_buckets"\n\t\t\t},\n\t\t\t"28": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "gc_delete_range"\n\t\t\t},\n\t\t\t"30": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "gc_delete_range_done"\n\t\t\t},\n\t\t\t"32": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_feedback"\n\t\t\t},\n\t\t\t"34": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "role_edges"\n\t\t\t},\n\t\t\t"36": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "default_roles"\n\t\t\t},\n\t\t\t"38": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "bind_info"\n\t\t\t},\n\t\t\t"4": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "user"\n\t\t\t},\n\t\t\t"40": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_top_n"\n\t\t\t},\n\t\t\t"42": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "expr_pushdown_blacklist"\n\t\t\t},\n\t\t\t"44": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "opt_rule_blacklist"\n\t\t\t},\n\t\t\t"46": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_extended"\n\t\t\t},\n\t\t\t"48": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "schema_index_usage"\n\t\t\t},\n\t\t\t"50": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_fm_sketch"\n\t\t\t},\n\t\t\t"52": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "global_grants"\n\t\t\t},\n\t\t\t"54": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "capture_plan_baselines_blacklist"\n\t\t\t},\n\t\t\t"56": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "column_stats_usage"\n\t\t\t},\n\t\t\t"58": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "table_cache_meta"\n\t\t\t},\n\t\t\t"6": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "password_history"\n\t\t\t},\n\t\t\t"60": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "analyze_options"\n\t\t\t},\n\t\t\t"62": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_history"\n\t\t\t},\n\t\t\t"64": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_meta_history"\n\t\t\t},\n\t\t\t"66": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "analyze_jobs"\n\t\t\t},\n\t\t\t"68": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "advisory_locks"\n\t\t\t},\n\t\t\t"70": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_mdl_view"\n\t\t\t},\n\t\t\t"72": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "plan_replayer_status"\n\t\t\t},\n\t\t\t"74": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "plan_replayer_task"\n\t\t\t},\n\t\t\t"76": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_table_locked"\n\t\t\t},\n\t\t\t"78": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_ttl_table_status"\n\t\t\t},\n\t\t\t"8": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "global_priv"\n\t\t\t},\n\t\t\t"80": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_ttl_task"\n\t\t\t},\n\t\t\t"82": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_ttl_job_history"\n\t\t\t},\n\t\t\t"84": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_global_task"\n\t\t\t},\n\t\t\t"86": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "load_data_jobs"\n\t\t\t},\n\t\t\t"90": {\n\t\t\t\t"db-name": "loadtest",\n\t\t\t\t"tbl-name": "sbtest1"\n\t\t\t},\n\t\t\t"91": {\n\t\t\t\t"db-name": "loadtest",\n\t\t\t\t"tbl-name": "sbtest3"\n\t\t\t},\n\t\t\t"92": {\n\t\t\t\t"db-name": "loadtest",\n\t\t\t\t"tbl-name": "sbtest2"\n\t\t\t}\n\t\t}\n\t}: table tidb_background_subtask(281474976710650) not found"] [errorVerbose="table tidb_background_subtask(281474976710650) not found\ngithub.com/pingcap/errors.NotFoundf\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:117\ngithub.com/pingcap/tidb-binlog/drainer.(*Schema).ReplaceTable\n\t/mnt/data1/jenkins/workspace/build-common@2/go/src/github.com/pingcap/tidb-binlog/drainer/schema.go:258\ngithub.com/pingcap/tidb-binlog/drainer.(*Schema).handleDDL\n\t/mnt/data1/jenkins/workspace/build-common@2/go/src/github.com/pingcap/tidb-binlog/drainer/schema.go:540\ngithub.com/pingcap/tidb-binlog/drainer.(*Schema).handlePreviousDDLJobIfNeed\n\t/mnt/data1/jenkins/workspace/build-common@2/go/src/github.com/pingcap/tidb-binlog/drainer/schema.go:319\ngithub.com/pingcap/tidb-binlog/drainer.(*Syncer).run\n\t/mnt/data1/jenkins/workspace/build-common@2/go/src/github.com/pingcap/tidb-binlog/drainer/syncer.go:448\ngithub.com/pingcap/tidb-binlog/drainer.(*Syncer).Start\n\t/mnt/data1/jenkins/workspace/build-common@2/go/src/github.com/pingcap/tidb-binlog/drainer/syncer.go:152\ngithub.com/pingcap/tidb-binlog/drainer.(*Server).Start.func5\n\t/mnt/data1/jenkins/workspace/build-common@2/go/src/github.com/pingcap/tidb-binlog/drainer/server.go:310\ngithub.com/pingcap/tidb-binlog/drainer.(*taskGroup).start.func1\n\t/mnt/data1/jenkins/workspace/build-common@2/go/src/github.com/pingcap/tidb-binlog/drainer/util.go:82\nruntime.goexit\n\t/usr/local/go1.20.5/src/runtime/asm_arm64.s:1172\nhandle ddl job ID:102, Type:add column, State:synced, SchemaState:public, SchemaID:1, TableID:281474976710650, RowCount:0, ArgLen:0, start time: 2023-09-26 21:11:22.969 +0800 CST, Err:, ErrCount:0, SnapshotVersion:0 failed, the schema info: {\n\t\t"hasImplicitCol": false,\n\t\t"schemaMetaVersion": 0,\n\t\t"schemaNameToID": {\n\t\t\t"loadtest": 88,\n\t\t\t"mysql": 1,\n\t\t\t"test": 2\n\t\t},\n\t\t"tableIDToName": {\n\t\t\t"10": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "db"\n\t\t\t},\n\t\t\t"12": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tables_priv"\n\t\t\t},\n\t\t\t"14": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "columns_priv"\n\t\t\t},\n\t\t\t"16": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "GLOBAL_VARIABLES"\n\t\t\t},\n\t\t\t"18": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb"\n\t\t\t},\n\t\t\t"20": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "help_topic"\n\t\t\t},\n\t\t\t"22": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_meta"\n\t\t\t},\n\t\t\t"24": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_histograms"\n\t\t\t},\n\t\t\t"26": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_buckets"\n\t\t\t},\n\t\t\t"28": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "gc_delete_range"\n\t\t\t},\n\t\t\t"30": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "gc_delete_range_done"\n\t\t\t},\n\t\t\t"32": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_feedback"\n\t\t\t},\n\t\t\t"34": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "role_edges"\n\t\t\t},\n\t\t\t"36": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "default_roles"\n\t\t\t},\n\t\t\t"38": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "bind_info"\n\t\t\t},\n\t\t\t"4": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "user"\n\t\t\t},\n\t\t\t"40": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_top_n"\n\t\t\t},\n\t\t\t"42": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "expr_pushdown_blacklist"\n\t\t\t},\n\t\t\t"44": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "opt_rule_blacklist"\n\t\t\t},\n\t\t\t"46": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_extended"\n\t\t\t},\n\t\t\t"48": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "schema_index_usage"\n\t\t\t},\n\t\t\t"50": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_fm_sketch"\n\t\t\t},\n\t\t\t"52": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "global_grants"\n\t\t\t},\n\t\t\t"54": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "capture_plan_baselines_blacklist"\n\t\t\t},\n\t\t\t"56": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "column_stats_usage"\n\t\t\t},\n\t\t\t"58": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "table_cache_meta"\n\t\t\t},\n\t\t\t"6": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "password_history"\n\t\t\t},\n\t\t\t"60": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "analyze_options"\n\t\t\t},\n\t\t\t"62": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_history"\n\t\t\t},\n\t\t\t"64": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_meta_history"\n\t\t\t},\n\t\t\t"66": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "analyze_jobs"\n\t\t\t},\n\t\t\t"68": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "advisory_locks"\n\t\t\t},\n\t\t\t"70": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_mdl_view"\n\t\t\t},\n\t\t\t"72": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "plan_replayer_status"\n\t\t\t},\n\t\t\t"74": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "plan_replayer_task"\n\t\t\t},\n\t\t\t"76": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "stats_table_locked"\n\t\t\t},\n\t\t\t"78": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_ttl_table_status"\n\t\t\t},\n\t\t\t"8": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "global_priv"\n\t\t\t},\n\t\t\t"80": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_ttl_task"\n\t\t\t},\n\t\t\t"82": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_ttl_job_history"\n\t\t\t},\n\t\t\t"84": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "tidb_global_task"\n\t\t\t},\n\t\t\t"86": {\n\t\t\t\t"db-name": "mysql",\n\t\t\t\t"tbl-name": "load_data_jobs"\n\t\t\t},\n\t\t\t"90": {\n\t\t\t\t"db-name": "loadtest",\n\t\t\t\t"tbl-name": "sbtest1"\n\t\t\t},\n\t\t\t"91": {\n\t\t\t\t"db-name": "loadtest",\n\t\t\t\t"tbl-name": "sbtest3"\n\t\t\t},\n\t\t\t"92": {\n\t\t\t\t"db-name": "loadtest",\n\t\t\t\t"tbl-name": "sbtest2"\n\t\t\t}\n\t\t}\n\t}"]
Please provide the relate downstream type and version of drainer.
(run drainer -V in terminal to get drainer's version)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with drainer/schema.go, especially ReplaceTable, handleDDL, and handlePreviousDDLJobIfNeed, then trace the callers in syncer.go and server.go. Reproduce the v7.0.0-to-v7.1.1 upgrade and determine why the missing tidb_background_subtask table aborts startup; done means the upgrade starts successfully without this failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100