ClusterLabs / ClusterLabs/resource-agents

Galera: Unable to detect last known write sequence number

Open
#940 16 comments 1 reaction 0 assignees View on GitHub
Dominant language
Shell
Stars
519
Forks
608
Avg merge
6d 1h
Merged PRs (30d)
7

Description

Our setup:

Pacemaker cluster:
pacemaker-cluster-libs-1.1.15-11.el7_3.2.x86_64
pacemaker-libs-1.1.15-11.el7_3.2.x86_64
pacemaker-cli-1.1.15-11.el7_3.2.x86_64
pacemaker-1.1.15-11.el7_3.2.x86_64

4x nodes galera-cluster:
MariaDB-server-10.1.21-1.el7.centos.x86_64
MariaDB-common-10.1.21-1.el7.centos.x86_64
MariaDB-client-10.1.21-1.el7.centos.x86_64
MariaDB-shared-10.1.21-1.el7.centos.x86_64
MariaDB-devel-10.1.21-1.el7.centos.x86_64
galera-25.3.19-1.rhel7.el7.centos.x86_64

And last lime we use follow resource agent:
- https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/galera

In most cases after reboot all nodes in cluster we have:

* galera_start_0 on node04 'unknown error' (1): call=890, status=complete, exitreason='Unable to detect last known write sequence number',
last-rc-change='Mon Mar 6 12:32:37 2017', queued=0ms, exec=6555ms

This error replyes from 1 to 4 nodes.

So we can see logs from pacemaker and mariadb, and it looks like pacemaker can't send wsrep_cluster_address to mysql when it starts.
This error repeats for SST methods - rsync and xtrabackup-v2

170306 12:32:38 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
170306 12:32:38 mysqld_safe WSREP: Running position recovery with —log_error='/var/lib/mysql/wsrep_recovery.V5Cuta' —pid-file='/var/lib/mysql/node03-recover.pid'
2017-03-06 12:32:38 140267724499200 [Note] /usr/sbin/mysqld (mysqld 10.1.21-MariaDB) starting as process 61644 ...
2017-03-06 12:32:38 140267724499200 [ERROR] WSREP: xtrabackup-v2 SST method requires wsrep_cluster_address to be configured on startup.
170306 12:32:41 mysqld_safe WSREP: Position recovery skipped
2017-03-06 12:32:41 140431155935488 [Warning] You need to use —log-bin to make —binlog-format work.
2017-03-06 12:32:41 140431155935488 [Note] /usr/sbin/mysqld (mysqld 10.1.21-MariaDB) starting as process 61735 ...
2017-03-06 12:32:41 140431155935488 [ERROR] WSREP: xtrabackup-v2 SST method requires wsrep_cluster_address to be configured on startup.
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: The InnoDB memory heap is disabled
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Compressed tables use zlib 1.2.7
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Using Linux native AIO
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Using SSE crc32 instructions
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Completed initialization of buffer pool
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Highest supported file format is Barracuda.
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: 128 rollback segment(s) are active.
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Waiting for purge to start
2017-03-06 12:32:41 140431155935488 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.34-79.1 started; log sequence number 2458757979
2017-03-06 12:32:41 140431155935488 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
2017-03-06 12:32:41 140431155935488 [Note] Plugin 'FEEDBACK' is disabled.
2017-03-06 12:32:41 140431155935488 [Note] Server socket created on IP: '0.0.0.0'.
2017-03-06 12:32:41 140431155935488 [Note] WSREP: disabled, skipping position recovery
170306 12:32:44 mysqld_safe mysqld from pid file /var/run/mysql/mysqld.pid ended

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with heartbeat/galera and the Pacemaker and MariaDB logs shown in the report. Reproduce a post-reboot startup using the listed rsync and xtrabackup-v2 SST methods, focusing on whether wsrep_cluster_address is available during position recovery. Done means the resource agent reliably detects the last known write sequence number and starts the node without the reported error.

Written by the indexing model from the issue text.

Assessment

Tech stack
mariadb, shell
Domain
databases, devops, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.