ClusterLabs / ClusterLabs/resource-agents
Galera: Unable to detect last known write sequence number
- Dominant language
- Shell
- Stars
- 519
- Forks
- 608
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 7
Description
Our setup:
Pacemaker cluster:
pacemaker-cluster-libs-1.1.15-11.el7_3.2.x86_64
pacemaker-libs-1.1.15-11.el7_3.2.x86_64
pacemaker-cli-1.1.15-11.el7_3.2.x86_64
pacemaker-1.1.15-11.el7_3.2.x86_64
4x nodes galera-cluster:
MariaDB-server-10.1.21-1.el7.centos.x86_64
MariaDB-common-10.1.21-1.el7.centos.x86_64
MariaDB-client-10.1.21-1.el7.centos.x86_64
MariaDB-shared-10.1.21-1.el7.centos.x86_64
MariaDB-devel-10.1.21-1.el7.centos.x86_64
galera-25.3.19-1.rhel7.el7.centos.x86_64
And last lime we use follow resource agent:
- https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/galera
In most cases after reboot all nodes in cluster we have:
* galera_start_0 on node04 'unknown error' (1): call=890, status=complete, exitreason='Unable to detect last known write sequence number',
last-rc-change='Mon Mar 6 12:32:37 2017', queued=0ms, exec=6555ms
This error replyes from 1 to 4 nodes.
So we can see logs from pacemaker and mariadb, and it looks like pacemaker can't send wsrep_cluster_address to mysql when it starts.
This error repeats for SST methods - rsync and xtrabackup-v2
170306 12:32:38 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
170306 12:32:38 mysqld_safe WSREP: Running position recovery with —log_error='/var/lib/mysql/wsrep_recovery.V5Cuta' —pid-file='/var/lib/mysql/node03-recover.pid'
2017-03-06 12:32:38 140267724499200 [Note] /usr/sbin/mysqld (mysqld 10.1.21-MariaDB) starting as process 61644 ...
2017-03-06 12:32:38 140267724499200 [ERROR] WSREP: xtrabackup-v2 SST method requires wsrep_cluster_address to be configured on startup.
170306 12:32:41 mysqld_safe WSREP: Position recovery skipped
2017-03-06 12:32:41 140431155935488 [Warning] You need to use —log-bin to make —binlog-format work.
2017-03-06 12:32:41 140431155935488 [Note] /usr/sbin/mysqld (mysqld 10.1.21-MariaDB) starting as process 61735 ...
2017-03-06 12:32:41 140431155935488 [ERROR] WSREP: xtrabackup-v2 SST method requires wsrep_cluster_address to be configured on startup.
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: The InnoDB memory heap is disabled
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Compressed tables use zlib 1.2.7
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Using Linux native AIO
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Using SSE crc32 instructions
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Completed initialization of buffer pool
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Highest supported file format is Barracuda.
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: 128 rollback segment(s) are active.
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Waiting for purge to start
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.34-79.1 started; log sequence number 2458757979
2017-03-06 12:32:41 140431155935488 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
2017-03-06 12:32:41 140431155935488 [Note] Plugin 'FEEDBACK' is disabled.
2017-03-06 12:32:41 140431155935488 [Note] Server socket created on IP: '0.0.0.0'.
2017-03-06 12:32:41 140431155935488 [Note] WSREP: disabled, skipping position recovery
170306 12:32:44 mysqld_safe mysqld from pid file /var/run/mysql/mysqld.pid ended
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with heartbeat/galera and the Pacemaker and MariaDB logs shown in the report. Reproduce a post-reboot startup using the listed rsync and xtrabackup-v2 SST methods, focusing on whether wsrep_cluster_address is available during position recovery. Done means the resource agent reliably detects the last known write sequence number and starts the node without the reported error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mariadb, shell
- Domain
- databases, devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100