GKE Autopilot constant re-deployment of pods
- Dominant language
- Go
- Stars
- 283
- Forks
- 148
- PR merge metrics
- No merged PRs in 30d
Description
I'm running solr operator on GKE Autopilot clust with example from [here](https://solr.apache.org/operator/articles/explore-v030-gke.html) - the example requires to round-up resource requests to values supported by autopilot. However, operator keeps re-deploying deployments due to change in resource requests, but the 'to' and 'from' valus are exactly the same. This bug is best observable with prometheus exporter which is redeployed couple times per second, the same is happening with cluster but less often - seem like every couple minutes.
Logs, seem like relevant code is [this](https://github.com/apache/solr-operator/blob/94b62dc967ac4c6e1c8f16acf45d43298ca103d3/controllers/util/common.go#L545) - note the to and from clauses
```
controller-runtime.manager.controller.solrprometheusexporter Update required because field changed {"reconciler group": "solr.apache.org", "reconciler kind": "SolrPrometheusExporter", "name": "explore-prom-exporter", "namespace": "default", "deployment": "explore-prom-exporter-solr-metrics", "kind": "deployment", "field": "Spec.Template.Spec.Containers[0].Resources", "from": {"limits":{"cpu":"250m","ephemeral-storage":"1Gi","memory":"512Mi"},"requests":{"cpu":"250m","ephemeral-storage":"1Gi","memory":"512Mi"}}, "to": {"limits":{"cpu":"250m","ephemeral-storage":"1Gi","memory":"512Mi"},"requests":{"cpu":"250m","ephemeral-storage":"1Gi","memory":"512Mi"}}}
```
```
Error
2022-01-13T21:46:02.884596420Z2022-01-13T21:46:02.884Z INFO controller-runtime.manager.controller.solrcloud Update required because field changed {"reconciler group": "solr.apache.org", "reconciler kind": "SolrCloud", "name": "explore", "namespace": "default", "zookeeperCluster": "explore-solrcloud-zookeeper", "kind": "zookeeperCluster", "field": "Spec.Pod.Resources", "from": {"limits":{"cpu":"250m","ephemeral-storage":"512Mi","memory":"500Mi"},"requests":{"cpu":"250m","ephemeral-storage":"512Mi","memory":"500Mi"}}, "to": {"limits":{"cpu":"250m","ephemeral-storage":"512Mi","memory":"500Mi"},"requests":{"cpu":"250m","ephemeral-storage":"512Mi","memory":"500Mi"}}}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in controllers/util/common.go around the linked resource comparison and reproduce the repeated updates using the Solr operator example on GKE Autopilot. Compare the logged resource values and reconciliation behavior for the exporter and SolrCloud; done means identical resources no longer trigger repeated redeployments.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, go, kubernetes
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100