planetscale / planetscale/vitess-operator
Vitess operator trying to run vtbackup-init pods for old shards after resharding worfklow finished and old shard tablets still available before yaml apply to remove it
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 372
- Forks
- 102
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 15
Description
Vitess latest operator 2.16.0 after reshard workflow completed, removing definition for old shards so it should be removed all vttablet pods of those, we get suddenly spawning vtbackup-init pods for old shards... operator stuck and not removing old shards because of those init pods trying to run.
{"level":"error","ts":"2026-01-23T10:58:03Z","msg":"Reconciler error","controller":"vitessshard-controller","object":{"name":"dev-vitess2-loadtest-c0-x-22f2ea3f","namespace":"vitess-test"},"namespace":"vitess-test","name":"dev-vitess2-loadtest-c0-x-22f2ea3f","reconcileID":"1af80deb-a82f-438b-aaec-28ef444974f9","error":"object is being deleted: pods \"dev-vitess2-loadtest-c0-x-vtbackup-init-c1bee57e\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:341\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:249"}
{"level":"info","ts":"2026-01-23T10:58:03Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"vitessshard-controller","object":{"name":"dev-vitess2-loadtest-c0-x-22f2ea3f","namespace":"vitess-test"},"namespace":"vitess-test","name":"dev-vitess2-loadtest-c0-x-22f2ea3f","reconcileID":"72271ce3-0eff-4e94-a6d3-0e4e82dae22a"}
{"level":"error","ts":"2026-01-23T10:58:03Z","msg":"Reconciler error","controller":"vitessshard-controller","object":{"name":"dev-vitess2-loadtest-c0-x-22f2ea3f","namespace":"vitess-test"},"namespace":"vitess-test","name":"dev-vitess2-loadtest-c0-x-22f2ea3f","reconcileID":"72271ce3-0eff-4e94-a6d3-0e4e82dae22a","error":"object is being deleted: pods \"dev-vitess2-loadtest-c0-x-vtbackup-init-c1bee57e\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:341\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:249"}
{"level":"info","ts":"2026-01-23T10:58:03Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"vitessshard-controller","object":{"name":"dev-vitess2-loadtest-c0-x-22f2ea3f","namespace":"vitess-test"},"namespace":"vitess-test","name":"dev-vitess2-loadtest-c0-x-22f2ea3f","reconcileID":"bfad7523-ac33-4603-8842-dc6290976ff2"}
{"level":"error","ts":"2026-01-23T10:58:03Z","msg":"Reconciler error","controller":"vitessshard-controller","object":{"name":"dev-vitess2-loadtest-c0-x-22f2ea3f","namespace":"vitess-test"},"namespace":"vitess-test","name":"dev-vitess2-loadtest-c0-x-22f2ea3f","reconcileID":"bfad7523-ac33-4603-8842-dc6290976ff2","error":"object is being deleted: pods \"dev-vitess2-loadtest-c0-x-vtbackup-init-c1bee57e\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:341\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:249"}```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the vitessshard-controller reconciliation path for object dev-vitess2-loadtest-c0-x-22f2ea3f and inspect the vtbackup-init pod lifecycle during removal of old shards. Reproduce the completed resharding workflow and compare the controller logs, especially the already-exists error. Done means old shard tablets can be removed without repeatedly creating conflicting init pods or leaving reconciliation stuck.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100