aws-samples / aws-samples/amazon-cloudwatch-container-insights
In the DeamonSet yaml the FluentBit pod has blanket tolerations, so EKS tried to schedule them on Fargate nodes
- Dominant language
- Shell
- Stars
- 178
- Forks
- 117
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 3
Description
## Actual
I've installed CloudWatch Container Insights per the [DaemonSet QuickStart](https://github.com/aws-samples/amazon-cloudwatch-container-insights/blob/master/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluent-bit-quickstart.yaml).
This creates 2 DeamonSets, 1 for CloudWatch agent and 1 for FluentBit. I notice that Kubernetes tries to schedule the Fluentbit DaemonSet pod also on (pre-existing) Fargate nodes, which obviously does not work (the pods stay pending forever). Interestingly, no pods are created against Fargate nodes for the CloudWatch daemonset (which is good).
## Expected
FluentBid pods are not scheduled against Fargate nodes (as that is not supported on Fargate anyway, these pods stay pending forever).
If I want to run FluentBit in Fargate, I'll use a sidecar container for it.
## Root cause
At the bottom of [DaemonSet QuickStart](https://github.com/aws-samples/amazon-cloudwatch-container-insights/blob/master/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluent-bit-quickstart.yaml) there's the pod spec for FluentBit that has these tolerations:
```
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
- operator: "Exists"
effect: "NoExecute"
- operator: "Exists"
effect: "NoSchedule"
```
The last 2 tolerations have no `key` and are thus blanket tolerations, triggering the scheduling on (pre-existing) Fargate nodes.
Note that the pod spec for the CloudWatch agent in the same yaml doesn't have any tolerations.
Contributor guide
Research direction
Open k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluent-bit-quickstart.yaml and inspect the FluentBit pod tolerations alongside the CloudWatch agent spec. Confirm the resulting DaemonSet does not remain pending on pre-existing EKS Fargate nodes, while the intended node tolerations still work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100