aws / aws/eks-charts

[aws-for-fluent-bit] Upgrading to 0.1.14

Open
#699 2 comments 5 reactions 0 assignees View on GitHub
bug
Dominant language
Mustache
Stars
1.3k
Forks
1.1k
Avg merge
22m
Merged PRs (30d)
5

Description

**Describe the bug**

We had an unpinned version of AWS for FluentBit running.
Yesterday when the [update was released](https://github.com/aws/eks-charts/releases/tag/v0.0.79)
It caused the issue seen below:
```
Error: UPGRADE FAILED: cannot patch "aws-for-fluent-bit" with kind DaemonSet: DaemonSet.apps "aws-for-fluent-bit" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"aws-for-fluent-bit", "app.kubernetes.io/name":"aws-for-fluent-bit"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
```

I believe this the diff that caused the issue:

```diff
- app.kubernetes.io/instance: {{ include "aws-for-fluent-bit.namespace" . }}
+ app.kubernetes.io/instance: {{ include "aws-for-fluent-bit.fullname" . }}
```
- [selector label diff in latest update](https://github.com/aws/eks-charts/commit/ca72d20536641131909b647b3711315a9d5294eb#diff-5f9bb02234e04bd342796bd0480c10b50e0ae760fc7b407fdc207364f50ef04fR51)

Workarounds:
- delete the release - this is not an option I think as we might lose logs?
- pin the version to 0.1.13 - this is what we did because it was blocking our CI/CD pipeline. But we need some way of upgrading to the latest version.

Read K8s discussion: https://github.com/kubernetes/kubernetes/issues/50808

**Steps to reproduce**

Run the following while having 0.1.13 running before
```bash
helm upgrade --install aws-for-fluent-bit eks/aws-for-fluent-bit --version 0.1.14 -n kube-system -f helm/fluent-bit.yml \
--set 'cloudWatch.region'=us-east-1 \
--set 'cloudWatch.logGroupName'=/test_flb \
--set 'cloudWatch.logRetentionDays'=30
```

**Expected outcome**
A concise description of what you expected to happen.

**Environment**

* Chart name: aws-for-fluent-bit
* Chart version: 0.1.14
* Kubernetes version: 1.21
* Using EKS (yes/no), if so version? yes

**Additional Context**:

Contributor guide

Open the contributing guide

Research direction

Start with the aws-for-fluent-bit chart and the helm/fluent-bit.yml values used in the reproduction, then inspect the selector-label diff linked from commit ca72d205. Reproduce an upgrade from chart 0.1.13 to 0.1.14 on Kubernetes 1.21 and determine the chart change needed. Done means the upgrade succeeds without the immutable DaemonSet selector error and does not require deleting the release.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.