[EKS/Fargate] [Logging]: No logs with log group/stream templates
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
This is my current (and working) EKS fargate logging CM:
```
---
kind: ConfigMap
apiVersion: v1
metadata:
name: aws-logging
namespace: aws-observability
data:
flb_log_cw: "true"
filters.conf: |
[FILTER]
Name parser
Match *
Key_name log
Parser crio
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
Labels Off
Annotations Off
Buffer_Size 0
Kube_Meta_Cache_TTL 300s
output.conf: |
[OUTPUT]
Name cloudwatch_logs
Match *
region eu-north-1
log_group_name my-logs
log_stream_prefix from-fluent-bit-
log_retention_days 60
auto_create_group true
parsers.conf: |
[PARSER]
Name crio
Format Regex
Regex ^(?
Following the docs at https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch I tried adding the the following configuration to the output in order to separate the logs by namespace:
```
log_group_template eks-logs-$kubernetes['namespace_name']
log_stream_template $kubernetes['pod_name'].$kubernetes['container_name']
```
However when I do, no logs appear anymore in CloudWatch (adding either of these configs on their own causes the issue). The "default" log group no longer gets any logs, and no new log group is created. Additionally I don't get any logs from the flb agent. The describe pod command however says they are enabled after restarting a deployment:
```
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal LoggingEnabled 96s fargate-scheduler Successfully enabled logging for pod
```
**Which service(s) is this request for?**
EKS Fargate Logging
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
See above
**Are you currently working around this issue?**
Currently forced to send all logs in the cluster to the same log group, and with less readable log stream names.
**Additional context**
Kubernetes v1.23
**Attachments**
n/a
Contributor guide
Research direction
Start with the EKS Fargate logging ConfigMap, especially output.conf, and the linked Fluent Bit CloudWatch output documentation. Reproduce the failure with log_group_template and log_stream_template separately, then verify that logs reach CloudWatch and the configured group and stream names are created.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, observability-sre
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100