[aws-for-fluent-bit] Connection Refused in Liveness Probe
- Dominant language
- Mustache
- Stars
- 1.3k
- Forks
- 1.1k
- Avg merge
- 22m
- Merged PRs (30d)
- 5
Description
**Describe the bug**
When upgrading the `aws-for-fluent-bit` from version `0.1.27` to `0.1.28`, our Fluent Bit pods enter a `CrashLoopBackoff` state, due to failures in the newly introduced [Liveness Probe](https://github.com/aws/eks-charts/pull/975).
Pod events show the following message:
```
Liveness probe failed: Get "http://[2600:1f18:REDACTED::2]:2000/api/v1/health": dial tcp [2600:1f18:REDACTED::2]:2000: connect: connection refused
```
I believe this is related to the fact that the default `HTTP_Listen` is set to `0.0.0.0`, which means it will not respond to any IPv6 probes (also confirmed this by shelling into the container and trying out `curl -6 localhost:2020`, which failed).
Changing the Chart Values to set `HTTP_Listen` to `[::]` fixes the issue, and it appears that the probes listen on both IPv4 and IPv6 addresses (but I don't have an IPv4 EKS Cluster to test this).
**Steps to reproduce**
Spin up an IPv6 EKS Cluster, install the `aws-for-fluent-bit` Chart in version `0.1.28`. The pods will enter `CrashLoopBackoff`.
**Expected outcome**
The new Liveness Probe works correctly with the default Chart configuration.
**Environment**
* Chart name: `aws-for-fluent-bit`
* Chart version: `0.1.28`
* Kubernetes version: 1.26
* Using EKS (yes/no), if so version? Yes, `v1.26.7-eks-2d98532`
**Additional Context**:
The EKS Cluster is configured for IPv6 addressing.
Contributor guide
Research direction
Start by inspecting the aws-for-fluent-bit chart values and the newly introduced liveness probe configuration from version 0.1.28. Reproduce the failure on an IPv6 EKS cluster and compare probe behavior with the default HTTP_Listen setting. Done means the default chart configuration passes liveness checks without CrashLoopBackoff on IPv6, while remaining correct for IPv4.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, helm, kubernetes
- Domain
- cloud, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100