mysql / mysql/mysql-operator

Regression in 2.2.8: operator restart may no longer trigger recovery for clusters already OFFLINE at startup

Open
#52 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
948
Forks
168
PR merge metrics
No merged PRs in 30d

Description

Summary

After upgrading from 2.2.6 to 2.2.8, an existing InnoDBCluster that is already OFFLINE when the operator restarts may remain OFFLINE indefinitely if no further pod status change occurs after operator startup.

Versions

  • works as expected: 2.2.6 (9.5.0-2.2.6)
  • problematic: 2.2.8 (9.7.0-2.2.8)

Test setup

The reproduction I used was with an InnoDBCluster configured with 1 instance.

Steps to reproduce

  1. Deploy MySQL Operator and create an InnoDBCluster with 1 instance
  2. Put the cluster into a state where it is already OFFLINE when the operator comes back up
    • for example, after a node reboot or a simultaneous operator/pod restart scenario
  3. Let the operator restart
  4. Ensure no additional pod status change occurs after startup
  5. Observe the cluster status

Expected behavior

After operator restart, an existing cluster that is already OFFLINE should still be re-evaluated for recovery.

Actual behavior

In 2.2.8, the cluster may remain OFFLINE indefinitely unless a later pod status change happens.

Additional note

I compared the relevant source between 2.2.6 and 2.2.8, and this looks related to a change in the pod status watch / restart detection behavior after operator restart. I can provide the exact code comparison if that would be helpful.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the pod status watch and restart detection paths between versions 2.2.6 and 2.2.8, as suggested in the issue. Reproduce with a one-instance InnoDBCluster that is OFFLINE before operator startup and has no later pod status change. Done means the existing OFFLINE cluster is re-evaluated for recovery after restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, mysql, python
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.