canonical / canonical/observability-libs

k8s resource patch is losing the fight with juju

Open
#128 6 comments 0 reactions 0 assignees View on GitHub
Status: Triage Type: Bug
Dominant language
Python
Stars
4
Forks
8
Avg merge
1h 26m
Merged PRs (30d)
1

Description

### Bug Description

In some charms, we apply resource constraints on the charm container because we wanted fine-grained control over how much memory/cpu the charm container got without having to apply the same to the workload container(s).

juju 3.6.9 started setting a default memory limit (64M) **on the charm container**.
This breaks the service patch because we now get conflicts when we try to override it.
Our goal is to have per-container resource requests.

AFFECTED CHARMS:
🐛 loki-coordinator-k8s
🐛 mimir-coordinator-k8s
🐛 pyroscope-operators
🐛 tempo-operators

### To Reproduce

deploy anything using the service patch on juju 3.6.9

### Environment

juju 3.6.9

### Relevant log output

```shell
iunit-tempo-0: 10:15:28 ERROR unit.tempo/0.juju-log resource patcher failed with message 'Kubernetes resources patch failed: Apply failed with 1 conflict: conflict with "juju" using apps/v1: .spec.template.spec.contain
ers[name="charm"].resources.requests.memory'
unit-tempo-0: 10:15:28 INFO unit.tempo/0.juju-log HTTP Request: GET https://10.152.183.1/apis/apps/v1/namespaces/test-tempo/statefulsets/tempo "HTTP/1.1 200 OK"
unit-tempo-0: 10:15:28 INFO unit.tempo/0.juju-log HTTP Request: PATCH https://10.152.183.1/apis/apps/v1/namespaces/test-tempo/statefulsets/tempo?fieldManager=ResourcePatcher&dryRun=All "HTTP/1.1 409 Conflict"
unit-tempo-0: 10:15:29 ERROR unit.tempo/0.juju-log resource patcher failed with message 'Kubernetes resources patch failed: Apply failed with 1 conflict: conflict with "juju" using apps/v1: .spec.template.spec.contain
ers[name="charm"].resources.requests.memory'
```

### Additional context

Even if juju were to fix this, 3.6.9 would remain broken (and it'd take time). So we may need a release on our side with force=True so the charms aren't broken for now.

Our only stable track is 1/
We need to guarantee successful in-track upgrades.

https://github.com/juju/juju/issues/20446

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the service-patch failure on Juju 3.6.9 and inspect the resource patcher entry point; the issue does not name specific files or tests. Trace the conflict on the charm container's memory request and verify that the fix preserves per-container requests and successful in-track upgrades for the affected charms.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.