aws / aws/eks-charts

[aws-for-fluent-bit] Connection Refused in Liveness Probe

Open
#983 2 comments 6 reactions 0 assignees View on GitHub
bug
Dominant language
Mustache
Stars
1.3k
Forks
1.1k
Avg merge
22m
Merged PRs (30d)
5

Description

**Describe the bug**
When upgrading the `aws-for-fluent-bit` from version `0.1.27` to `0.1.28`, our Fluent Bit pods enter a `CrashLoopBackoff` state, due to failures in the newly introduced [Liveness Probe](https://github.com/aws/eks-charts/pull/975).

Pod events show the following message:

```
Liveness probe failed: Get "http://[2600:1f18:REDACTED::2]:2000/api/v1/health": dial tcp [2600:1f18:REDACTED::2]:2000: connect: connection refused
```

I believe this is related to the fact that the default `HTTP_Listen` is set to `0.0.0.0`, which means it will not respond to any IPv6 probes (also confirmed this by shelling into the container and trying out `curl -6 localhost:2020`, which failed).

Changing the Chart Values to set `HTTP_Listen` to `[::]` fixes the issue, and it appears that the probes listen on both IPv4 and IPv6 addresses (but I don't have an IPv4 EKS Cluster to test this).

**Steps to reproduce**
Spin up an IPv6 EKS Cluster, install the `aws-for-fluent-bit` Chart in version `0.1.28`. The pods will enter `CrashLoopBackoff`.

**Expected outcome**
The new Liveness Probe works correctly with the default Chart configuration.

**Environment**

* Chart name: `aws-for-fluent-bit`
* Chart version: `0.1.28`
* Kubernetes version: 1.26
* Using EKS (yes/no), if so version? Yes, `v1.26.7-eks-2d98532`

**Additional Context**:
The EKS Cluster is configured for IPv6 addressing.

Contributor guide

Open the contributing guide

Research direction

Start by inspecting the aws-for-fluent-bit chart values and the newly introduced liveness probe configuration from version 0.1.28. Reproduce the failure on an IPv6 EKS cluster and compare probe behavior with the default HTTP_Listen setting. Done means the default chart configuration passes liveness checks without CrashLoopBackoff on IPv6, while remaining correct for IPv4.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, helm, kubernetes
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.