apache / apache/solr-operator

GKE Autopilot constant re-deployment of pods

Open
#393 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
283
Forks
148
PR merge metrics
No merged PRs in 30d

Description

I'm running solr operator on GKE Autopilot clust with example from [here](https://solr.apache.org/operator/articles/explore-v030-gke.html) - the example requires to round-up resource requests to values supported by autopilot. However, operator keeps re-deploying deployments due to change in resource requests, but the 'to' and 'from' valus are exactly the same. This bug is best observable with prometheus exporter which is redeployed couple times per second, the same is happening with cluster but less often - seem like every couple minutes.

Logs, seem like relevant code is [this](https://github.com/apache/solr-operator/blob/94b62dc967ac4c6e1c8f16acf45d43298ca103d3/controllers/util/common.go#L545) - note the to and from clauses
```
controller-runtime.manager.controller.solrprometheusexporter Update required because field changed {"reconciler group": "solr.apache.org", "reconciler kind": "SolrPrometheusExporter", "name": "explore-prom-exporter", "namespace": "default", "deployment": "explore-prom-exporter-solr-metrics", "kind": "deployment", "field": "Spec.Template.Spec.Containers[0].Resources", "from": {"limits":{"cpu":"250m","ephemeral-storage":"1Gi","memory":"512Mi"},"requests":{"cpu":"250m","ephemeral-storage":"1Gi","memory":"512Mi"}}, "to": {"limits":{"cpu":"250m","ephemeral-storage":"1Gi","memory":"512Mi"},"requests":{"cpu":"250m","ephemeral-storage":"1Gi","memory":"512Mi"}}}
```
```
Error
2022-01-13T21:46:02.884596420Z2022-01-13T21:46:02.884Z INFO controller-runtime.manager.controller.solrcloud Update required because field changed {"reconciler group": "solr.apache.org", "reconciler kind": "SolrCloud", "name": "explore", "namespace": "default", "zookeeperCluster": "explore-solrcloud-zookeeper", "kind": "zookeeperCluster", "field": "Spec.Pod.Resources", "from": {"limits":{"cpu":"250m","ephemeral-storage":"512Mi","memory":"500Mi"},"requests":{"cpu":"250m","ephemeral-storage":"512Mi","memory":"500Mi"}}, "to": {"limits":{"cpu":"250m","ephemeral-storage":"512Mi","memory":"500Mi"},"requests":{"cpu":"250m","ephemeral-storage":"512Mi","memory":"500Mi"}}}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in controllers/util/common.go around the linked resource comparison and reproduce the repeated updates using the Solr operator example on GKE Autopilot. Compare the logged resource values and reconciliation behavior for the exporter and SolrCloud; done means identical resources no longer trigger repeated redeployments.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go, kubernetes
Domain
devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.