Regression in 2.2.8: operator restart may no longer trigger recovery for clusters already OFFLINE at startup
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 948
- Forks
- 168
- PR merge metrics
- No merged PRs in 30d
Description
Summary
After upgrading from 2.2.6 to 2.2.8, an existing InnoDBCluster that is already OFFLINE when the operator restarts may remain OFFLINE indefinitely if no further pod status change occurs after operator startup.
Versions
- works as expected: 2.2.6 (
9.5.0-2.2.6) - problematic: 2.2.8 (
9.7.0-2.2.8)
Test setup
The reproduction I used was with an InnoDBCluster configured with 1 instance.
Steps to reproduce
- Deploy MySQL Operator and create an
InnoDBClusterwith 1 instance - Put the cluster into a state where it is already OFFLINE when the operator comes back up
- for example, after a node reboot or a simultaneous operator/pod restart scenario
- Let the operator restart
- Ensure no additional pod status change occurs after startup
- Observe the cluster status
Expected behavior
After operator restart, an existing cluster that is already OFFLINE should still be re-evaluated for recovery.
Actual behavior
In 2.2.8, the cluster may remain OFFLINE indefinitely unless a later pod status change happens.
Additional note
I compared the relevant source between 2.2.6 and 2.2.8, and this looks related to a change in the pod status watch / restart detection behavior after operator restart. I can provide the exact code comparison if that would be helpful.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing the pod status watch and restart detection paths between versions 2.2.6 and 2.2.8, as suggested in the issue. Reproduce with a one-instance InnoDBCluster that is OFFLINE before operator startup and has no later pod status change. Done means the existing OFFLINE cluster is re-evaluated for recovery after restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, mysql, python
- Domain
- databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100