[aws-for-fluent-bit] logRetentionPeriod parameter is not working for existing logGroups
- Dominant language
- Mustache
- Stars
- 1.3k
- Forks
- 1.1k
- Avg merge
- 22m
- Merged PRs (30d)
- 5
Description
I have deployed aws-for-fluent-bit with logRetention set to 90 days using argo.
Below is the argo application yaml
```
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: aws-for-fluent-bit
namespace: argocd
spec:
destination:
namespace: fluent-bit
server: https://kubernetes.default.svc
project: admin
source:
chart: aws-for-fluent-bit
helm:
values: |
cloudWatchLogs:
region: us-west-2
logRetentionDays: 90
serviceAccount:
create: true
name: aws-for-fluent-bit
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam:::role/role-for-fluent_bit-us-west-2
repoURL: https://aws.github.io/eks-charts
targetRevision: 0.1.28
syncPolicy:
automated:
selfHeal: true
syncOptions:
- Replace=true
- CreateNamespace=true
```
This deploys fine but all the older logGroups still have `never` as retention period .
Below is the fluent-bit.conf.
```
[SERVICE]
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_PORT 2020
Health_Check On
HC_Errors_Count 5
HC_Retry_Failure_Count 5
HC_Period 5
Parsers_File /fluent-bit/parsers/parsers.conf
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
DB /var/log/flb_kube.db
Parser docker
Docker_Mode On
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc.cluster.local:443
Merge_Log On
Merge_Log_Key data
Keep_Log On
K8S-Logging.Parser On
K8S-Logging.Exclude On
Buffer_Size 32k
[OUTPUT]
Name cloudwatch_logs
Match *
region us-west-2
log_group_name /aws/eks/fluentbit-cloudwatch/logs
log_group_template /aws/eks/fluentbit-cloudwatch/workload/$kubernetes['namespace_name']
log_stream_prefix fluentbit-
log_stream_template $kubernetes['pod_name'].$kubernetes['container_name']
auto_create_group true
log_retention_days 90
```
The policy attached to this role has required permissions as well
```
{
"Effect" : "Allow",
"Action" : [
"cloudwatch:PutMetricData",
"ec2:DescribeVolumes",
"ec2:DescribeTags",
"logs:PutLogEvents",
"logs:DescribeLogStreams",
"logs:DescribeLogGroups",
"logs:CreateLogStream",
"logs:CreateLogGroup",
"logs:PutRetentionPolicy"
],
"Resource" : "*"
}
```
Contributor guide
Research direction
Start with the aws-for-fluent-bit chart configuration represented in the Argo application YAML and the fluent-bit.conf output settings. Compare retention behavior for newly created and existing CloudWatch log groups, then verify that existing groups receive the configured 90-day retention period with the listed IAM permissions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, observability-sre
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100