ClusterLabs / ClusterLabs/resource-agents
galera: when grastate.dat is invalid, mysqld_safe overrides log_error, so the RA cannot detect the sequence number
- Dominant language
- Shell
- Stars
- 519
- Forks
- 608
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 7
Description
So after a normal shutdown:
````
# /usr/bin/mysqld_safe --log-error=/var/log/mysql/tttt.tmp --defaults-file=/etc/mysql/my.cnf --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --datadir=/var/lib/mysql --user=mysql --wsrep-recover
2018-01-29T14:56:23.237146Z mysqld_safe Logging to '/var/log/mysql/tttt.tmp'.
2018-01-29T14:56:23.239823Z mysqld_safe Logging to '/var/log/mysql/tttt.tmp'.
2018-01-29T14:56:23.269245Z mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2018-01-29T14:56:23.280053Z mysqld_safe Skipping wsrep-recover for b4e86214-1f80-11e7-b536-865e1a70c5ca:135427399 pair
2018-01-29T14:56:23.281643Z mysqld_safe Assigning b4e86214-1f80-11e7-b536-865e1a70c5ca:135427399 to wsrep_start_position
````
OK
No grastate.dat:
````
# mv grastate.dat grastate.dat.bak
# /usr/bin/mysqld_safe --log-error=/var/log/mysql/tttt.tmp --defaults-file=/etc/mysql/my.cnf --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --datadir=/var/lib/mysql --user=mysql --wsrep-recover
2018-01-29T14:56:41.249408Z mysqld_safe Logging to '/var/log/mysql/tttt.tmp'.
2018-01-29T14:56:41.252233Z mysqld_safe Logging to '/var/log/mysql/tttt.tmp'.
2018-01-29T14:56:41.279073Z mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2018-01-29T14:56:41.284927Z mysqld_safe Skipping wsrep-recover for empty datadir: /var/lib/mysql
2018-01-29T14:56:41.286358Z mysqld_safe Assigning 00000000-0000-0000-0000-000000000000:-1 to wsrep_start_position
````
OK
And after an invalid grastate.dat:
````
# touch grastate.dat
# /usr/bin/mysqld_safe --log-error=/var/log/mysql/tttt.tmp --defaults-file=/etc/mysql/my.cnf --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --datadir=/var/lib/mysql --user=mysql --wsrep-recover
2018-01-29T14:56:52.456398Z mysqld_safe Logging to '/var/log/mysql/tttt.tmp'.
2018-01-29T14:56:52.458947Z mysqld_safe Logging to '/var/log/mysql/tttt.tmp'.
2018-01-29T14:56:52.485519Z mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2018-01-29T14:56:52.500324Z mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.E83x4g' --pid-file='/var/lib/mysql/jsm-devops-openstack-controller-1-recover.pid'
````
FAIL. The error log goes to /var/lib/mysql/wsrep_recovery.xxxxxxx
Workaround: don't use mysqld_safe.
Percona server used: percona-xtradb-cluster-server-5.7 5.7.20-29.24-1.xenial
And seems even in the first two cases, mysqld_safe doesn't really start mysqld, it just assumes the start position on its own. Maybe it would be good to use mysqld directly for the --wsrep-recovery process, and mysqld_safe for starting the real server.
My idea is to add a recovery_binary reskey, defaulting to /usr/sbin/mysqld, and use this when --wsrep-recovery is needed.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the mysqld_safe --wsrep-recover path described in the issue and how the resource agent obtains the sequence number when grastate.dat is invalid. Compare the requested recovery_binary setting with the existing recovery commands. Done means recovery honors the requested log_error location and the RA can detect the recovered sequence number.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, shell
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100