ClusterLabs / ClusterLabs/resource-agents

galera: when grastate.dat is invalid, mysqld_safe overrides log_error, so the RA cannot detect the sequence number

Open
#1,094 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
519
Forks
608
Avg merge
6d 1h
Merged PRs (30d)
7

Description

So after a normal shutdown:
````
# /usr/bin/mysqld_safe --log-error=/var/log/mysql/tttt.tmp --defaults-file=/etc/mysql/my.cnf --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --datadir=/var/lib/mysql --user=mysql --wsrep-recover
2018-01-29T14:56:23.237146Z mysqld_safe Logging to '/var/log/mysql/tttt.tmp'.
2018-01-29T14:56:23.239823Z mysqld_safe Logging to '/var/log/mysql/tttt.tmp'.
2018-01-29T14:56:23.269245Z mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2018-01-29T14:56:23.280053Z mysqld_safe Skipping wsrep-recover for b4e86214-1f80-11e7-b536-865e1a70c5ca:135427399 pair
2018-01-29T14:56:23.281643Z mysqld_safe Assigning b4e86214-1f80-11e7-b536-865e1a70c5ca:135427399 to wsrep_start_position
````
OK

No grastate.dat:
````
# mv grastate.dat grastate.dat.bak
# /usr/bin/mysqld_safe --log-error=/var/log/mysql/tttt.tmp --defaults-file=/etc/mysql/my.cnf --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --datadir=/var/lib/mysql --user=mysql --wsrep-recover
2018-01-29T14:56:41.249408Z mysqld_safe Logging to '/var/log/mysql/tttt.tmp'.
2018-01-29T14:56:41.252233Z mysqld_safe Logging to '/var/log/mysql/tttt.tmp'.
2018-01-29T14:56:41.279073Z mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2018-01-29T14:56:41.284927Z mysqld_safe Skipping wsrep-recover for empty datadir: /var/lib/mysql
2018-01-29T14:56:41.286358Z mysqld_safe Assigning 00000000-0000-0000-0000-000000000000:-1 to wsrep_start_position
````
OK

And after an invalid grastate.dat:
````
# touch grastate.dat
# /usr/bin/mysqld_safe --log-error=/var/log/mysql/tttt.tmp --defaults-file=/etc/mysql/my.cnf --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --datadir=/var/lib/mysql --user=mysql --wsrep-recover
2018-01-29T14:56:52.456398Z mysqld_safe Logging to '/var/log/mysql/tttt.tmp'.
2018-01-29T14:56:52.458947Z mysqld_safe Logging to '/var/log/mysql/tttt.tmp'.
2018-01-29T14:56:52.485519Z mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2018-01-29T14:56:52.500324Z mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.E83x4g' --pid-file='/var/lib/mysql/jsm-devops-openstack-controller-1-recover.pid'
````
FAIL. The error log goes to /var/lib/mysql/wsrep_recovery.xxxxxxx

Workaround: don't use mysqld_safe.

Percona server used: percona-xtradb-cluster-server-5.7 5.7.20-29.24-1.xenial

And seems even in the first two cases, mysqld_safe doesn't really start mysqld, it just assumes the start position on its own. Maybe it would be good to use mysqld directly for the --wsrep-recovery process, and mysqld_safe for starting the real server.
My idea is to add a recovery_binary reskey, defaulting to /usr/sbin/mysqld, and use this when --wsrep-recovery is needed.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the mysqld_safe --wsrep-recover path described in the issue and how the resource agent obtains the sequence number when grastate.dat is invalid. Compare the requested recovery_binary setting with the existing recovery commands. Done means recovery honors the requested log_error location and the RA can detect the recovered sequence number.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, shell
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.