pingcap / pingcap/tidb-operator

Instance cr maybe outdated too long

Open
#6,470 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

v2
Dominant language
Go
Stars
1.3k
Forks
540
Avg merge
3d 2h
Merged PRs (30d)
18

Description

Now we use instance status to check what should do next.

If the operator is crashed and restarts after long time. The instance status maybe outdated and cannot be trusted.

  1. Operator is crashed.
  2. Group CR is updated and rolling update is needed.
  3. long time later.
  4. Pod may be not ready but instance is marked as ready, or pod may be ready but instance is marked as not ready.
  5. Invalid action to restart a running pod.

It's OK if instance status is not synced in a short term. Any pod maybe crashed when rolling update. However, a long term unsync makes some unexpected accidents 100% happen.

Maybe we need sync all instances before we run group controllers.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the group controllers and the instance-status synchronization path described in the issue. Reproduce the long-outage sequence where a group CR changes while the operator is stopped, then compare pod readiness with stored instance status. Done means stale status cannot trigger an invalid action after a prolonged outage, with synchronization behavior verified for the restart and rolling-update cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.