aws-samples / aws-samples/amazon-cloudwatch-container-insights

In the DeamonSet yaml the FluentBit pod has blanket tolerations, so EKS tried to schedule them on Fargate nodes

Open
#88 0 comments 5 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
178
Forks
117
Avg merge
1d 10h
Merged PRs (30d)
3

Description

## Actual

I've installed CloudWatch Container Insights per the [DaemonSet QuickStart](https://github.com/aws-samples/amazon-cloudwatch-container-insights/blob/master/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluent-bit-quickstart.yaml).

This creates 2 DeamonSets, 1 for CloudWatch agent and 1 for FluentBit. I notice that Kubernetes tries to schedule the Fluentbit DaemonSet pod also on (pre-existing) Fargate nodes, which obviously does not work (the pods stay pending forever). Interestingly, no pods are created against Fargate nodes for the CloudWatch daemonset (which is good).

## Expected

FluentBid pods are not scheduled against Fargate nodes (as that is not supported on Fargate anyway, these pods stay pending forever).

If I want to run FluentBit in Fargate, I'll use a sidecar container for it.

## Root cause

At the bottom of [DaemonSet QuickStart](https://github.com/aws-samples/amazon-cloudwatch-container-insights/blob/master/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluent-bit-quickstart.yaml) there's the pod spec for FluentBit that has these tolerations:

```
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
- operator: "Exists"
effect: "NoExecute"
- operator: "Exists"
effect: "NoSchedule"
```

The last 2 tolerations have no `key` and are thus blanket tolerations, triggering the scheduling on (pre-existing) Fargate nodes.

Note that the pod spec for the CloudWatch agent in the same yaml doesn't have any tolerations.

Contributor guide

Open the contributing guide

Research direction

Open k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluent-bit-quickstart.yaml and inspect the FluentBit pod tolerations alongside the CloudWatch agent spec. Confirm the resulting DaemonSet does not remain pending on pre-existing EKS Fargate nodes, while the intended node tolerations still work.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.