paused annotation holding upgrade for etcdadm controller
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 328
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 9
Description
**What would you like to be added**:
- When a Cluster upgrade fails (ETCD node unhealthy), a paused annotation is added to etcdadm controller. When user re-initiates cluster upgrade, upgrade does not continue until paused annotation is removed manually.
- Requesting for following,
1. eksctl should log about "paused" annotation on client side log
2. eksctl `--force-cleanup` should auto remove the annotation so that etcdadm controller can proceed with upgrade.
Error log:
```
0428 17:34:43.006301 1 controller.go:117] "msg"="Observed a panic in reconciler: key is not a string: \"could not find owner UID for eksa-system/gbicloud-cluster03-etcd-template-1682634216301-g2gfw\"" "CloudStackMachine"={"name":"gbicloud-cluster03-etcd-template-1682634216301-g2gfw","namespace":"eksa-system"} "controller"="cloudstackmachine" "controllerGroup"="[infrastructure.cluster.x-k8s.io](http://infrastructure.cluster.x-k8s.io/) " "controllerKind"="CloudStackMachine" "name"="gbicloud-cluster03-etcd-template-1682634216301-g2gfw" "namespace"="eksa-system" "reconcileID"="f3084748-4d5c-4c39-8327-abbbb82c5516"
panic: key is not a string: "could not find owner UID for eksa-system/gbicloud-cluster03-etcd-template-1682634216301-g2gfw" [recovered]
panic: key is not a string: "could not find owner UID for eksa-system/gbicloud-cluster03-etcd-template-1682634216301-g2gfw"
goroutine 1433 [running]:
[sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller](http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller)).Reconcile.func1()
[sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:118](http://sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:118) +0x1f4
panic({0x1908140, 0xc0022e7d10})
runtime/panic.go:838 +0x207
[k8s.io/klog/v2/klogr.flatten](http://k8s.io/klog/v2/klogr.flatten)({0xc004bf6420, 0x6, 0x0?})
[k8s.io/klog/v2@v2.80.1/klogr/klogr.go:82](http://k8s.io/klog/v2@v2.80.1/klogr/klogr.go:82) +0x4ff
[k8s.io/klog/v2/klogr.klogger.Info](http://k8s.io/klog/v2/klogr.klogger.Info)({0x0, 0x1, {0xc005174980, 0x1d}, {0xc000882f80, 0x4, 0x4}, {0x1c55ce5, 0x9}}, 0x0, ...)
[k8s.io/klog/v2@v2.80.1/klogr/klogr.go:127](http://k8s.io/klog/v2@v2.80.1/klogr/klogr.go:127) +0x28a
[github.com/go-logr/logr.Logger.Info](http://github.com/go-logr/logr.Logger.Info)({{0x1ee7d20?, 0xc0027b5180?}, 0xc004d3cd20?}, {0x1c71ad6, 0x1b}, {0xc0022e7cc0, 0x1, 0x1})
[github.com/go-logr/logr@v1.2.3/logr.go:261](http://github.com/go-logr/logr@v1.2.3/logr.go:261) +0xd0
[sigs.k8s.io/cluster-api-provider-cloudstack/controllers.(*CloudStackMachineReconciliationRunner](http://sigs.k8s.io/cluster-api-provider-cloudstack/controllers.(*CloudStackMachineReconciliationRunner)).ConsiderAffinity(0xc000882e80)
[sigs.k8s.io/cluster-api-provider-cloudstack/controllers/cloudstackmachine_controller.go:141](http://sigs.k8s.io/cluster-api-provider-cloudstack/controllers/cloudstackmachine_controller.go:141) +0x16b
[sigs.k8s.io/cluster-api-provider-cloudstack/controllers/utils.(*ReconciliationRunner](http://sigs.k8s.io/cluster-api-provider-cloudstack/controllers/utils.(*ReconciliationRunner)).RunReconciliationStages(0xc000205b20, {0xc00249d9b8, 0x8, 0x19?})
[sigs.k8s.io/cluster-api-provider-cloudstack/controllers/utils/base_reconciler.go:375](http://sigs.k8s.io/cluster-api-provider-cloudstack/controllers/utils/base_reconciler.go:375) +0x55
[sigs.k8s.io/cluster-api-provider-cloudstack/controllers.(*CloudStackMachineReconciliationRunner](http://sigs.k8s.io/cluster-api-provider-cloudstack/controllers.(*CloudStackMachineReconciliationRunner)).Reconcile(0xc000882e80)
[sigs.k8s.io/cluster-api-provider-cloudstack/controllers/cloudstackmachine_controller.go:117](http://sigs.k8s.io/cluster-api-provider-cloudstack/controllers/cloudstackmachine_controller.go:117) +0x2f4
[sigs.k8s.io/cluster-api-provider-cloudstack/controllers/utils.(*ReconciliationRunner](http://sigs.k8s.io/cluster-api-provider-cloudstack/controllers/utils.(*ReconciliationRunner)).RunIf.func1()
[sigs.k8s.io/cluster-api-provider-cloudstack/controllers/utils/base_reconciler.go:146](http://sigs.k8s.io/cluster-api-provider-cloudstack/controllers/utils/base_reconciler.go:146) +0xa6
[sigs.k8s.io/cluster-api-provider-cloudstack/controllers/utils.(*ReconciliationRunner](http://sigs.k8s.io/cluster-api-provider-cloudstack/controllers/utils.(*ReconciliationRunner)).RunReconciliationStages(0xc000205b20, {0xc0028cd260, 0xe, 0xc0038f8c60?})
[sigs.k8s.io/cluster-api-provider-cloudstack/controllers/utils/base_reconciler.go:375](http://sigs.k8s.io/cluster-api-provider-cloudstack/controllers/utils/base_reconciler.go:375) +0x55
[sigs.k8s.io/cluster-api-provider-cloudstack/controllers/utils.(*ReconciliationRunner](http://sigs.k8s.io/cluster-api-provider-cloudstack/controllers/utils.(*ReconciliationRunner)).RunBaseReconciliationStages(0xc000205b20)
[sigs.k8s.io/cluster-api-provider-cloudstack/controllers/utils/base_reconciler.go:412](http://sigs.k8s.io/cluster-api-provider-cloudstack/controllers/utils/base_reconciler.go:412) +0x494
[sigs.k8s.io/cluster-api-provider-cloudstack/controllers.(*CloudStackMachineReconciler](http://sigs.k8s.io/cluster-api-provider-cloudstack/controllers.(*CloudStackMachineReconciler)).Reconcile(0xc0007e01e0, {0x1ee5a48?, 0xc0038f8c00}, {{{0xc0009f7f40?, 0x10?}, {0xc000a6ae00?, 0x40d787?}}})
[sigs.k8s.io/cluster-api-provider-cloudstack/controllers/cloudstackmachine_controller.go:113](http://sigs.k8s.io/cluster-api-provider-cloudstack/controllers/cloudstackmachine_controller.go:113) +0x39b
[sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller](http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller)).Reconcile(0x1ee59a0?, {0x1ee5a48?, 0xc0038f8c00?}, {{{0xc0009f7f40?, 0x1b11600?}, {0xc000a6ae00?, 0x4041f4?}}})
[sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:121](http://sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:121) +0xc8
[sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller](http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller)).reconcileHandler(0xc0003d0140, {0x1ee59a0, 0xc000882380}, {0x1a21b20?, 0xc000c3df40?})
[sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:320](http://sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:320) +0x33c
[sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller](http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller)).processNextWorkItem(0xc0003d0140, {0x1ee59a0, 0xc000882380})
[sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273](http://sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273) +0x1d9
[sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller](http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller)).Start.func2.2()
[sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234](http://sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234) +0x85
created by [sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller](http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller)).Start.func2
[sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:230](http://sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:230) +0x325
```
**Why is this needed**:
- to recover cluster from upgrade failures
Contributor guide
Assessment
This issue has not been assessed yet.