aws / aws/containers-roadmap

[EKS/Fargate] [Logging]: No logs with log group/stream templates

Open
#1,854 4 comments 13 reactions 0 assignees View on GitHub
EKS Fargate Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
This is my current (and working) EKS fargate logging CM:
```
---
kind: ConfigMap
apiVersion: v1
metadata:
name: aws-logging
namespace: aws-observability
data:
flb_log_cw: "true"
filters.conf: |
[FILTER]
Name parser
Match *
Key_name log
Parser crio
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
Labels Off
Annotations Off
Buffer_Size 0
Kube_Meta_Cache_TTL 300s
output.conf: |
[OUTPUT]
Name cloudwatch_logs
Match *
region eu-north-1
log_group_name my-logs
log_stream_prefix from-fluent-bit-
log_retention_days 60
auto_create_group true
parsers.conf: |
[PARSER]
Name crio
Format Regex
Regex ^(?

Following the docs at https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch I tried adding the the following configuration to the output in order to separate the logs by namespace:

```
log_group_template eks-logs-$kubernetes['namespace_name']
log_stream_template $kubernetes['pod_name'].$kubernetes['container_name']
```

However when I do, no logs appear anymore in CloudWatch (adding either of these configs on their own causes the issue). The "default" log group no longer gets any logs, and no new log group is created. Additionally I don't get any logs from the flb agent. The describe pod command however says they are enabled after restarting a deployment:
```
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal LoggingEnabled 96s fargate-scheduler Successfully enabled logging for pod
```

**Which service(s) is this request for?**
EKS Fargate Logging

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
See above

**Are you currently working around this issue?**
Currently forced to send all logs in the cluster to the same log group, and with less readable log stream names.

**Additional context**
Kubernetes v1.23

**Attachments**
n/a

Contributor guide

Open the contributing guide

Research direction

Start with the EKS Fargate logging ConfigMap, especially output.conf, and the linked Fluent Bit CloudWatch output documentation. Reproduce the failure with log_group_template and log_stream_template separately, then verify that logs reach CloudWatch and the configured group and stream names are created.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, observability-sre
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.