aws / aws/eks-charts

aws-for-fluent-bit errors when disabling firehose, kinesis, and elastisearch plugins

Open
#751 2 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Mustache
Stars
1.3k
Forks
1.1k
Avg merge
22m
Merged PRs (30d)
5

Description

**Describe the bug**
When using aws-for-fluent-bit helm chart, I disable everything but the cloudwatch plugin. This makes the container dump out errors for the other services.

```
aws-for-fluent-bit:
enabled: true
source:
repoURL: https://aws.github.io/eks-charts
targetRevision: 0.1.16
chart: aws-for-fluent-bit
helm:
cloudwatch:
enabled: true
region: us-east-1
firehose:
enabled: false
kinesis:
enabled: false
elasticsearch:
enabled: false
tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Exists"
effect: "NoSchedule"
- operator: "Exists"
effect: "NoExecute"
- operator: "Exists"
effect: "NoSchedule"
serviceAccount:
create: true
name: aws-for-fluent-bit-sa
annotations:
eks.amazonaws.com/role-arn:
```

logs I get (redacted):
```
time="2022-05-11T12:20:07Z" level=error msg="[kinesis 0] PutRecords failed with AccessDeniedException: User: is not authorized to perform: kinesis:PutRecords on resource: because no identity-based policy allows the kinesis:PutRecords action\n\tstatus code: 400, request id: \n"
time="2022-05-11T12:20:07Z" level=error msg="[kinesis 0] AccessDeniedException: User: is not authorized to perform: kinesis:PutRecords on resource: because no identity-based policy allows the kinesis:PutRecords action\n\tstatus code: 400, request id: \n"
[2022/05/11 12:20:07] [ warn] [engine] chunk '1-1652271597.188086791.flb' cannot be retried: task_id=22, input=tail.0 > output=firehose.1
[2022/05/11 12:20:07] [ warn] [engine] chunk '1-1652271597.188086791.flb' cannot be retried: task_id=22, input=tail.0 > output=kinesis.2
[2022/05/11 12:20:07] [ warn] [engine] failed to flush chunk '1-1652271597.188086791.flb', retry in 10 seconds: task_id=22, input=tail.0 > output=es.3 (out_id=3)
[2022/05/11 12:20:09] [ warn] [engine] chunk '1-1652271281.422029229.flb' cannot be retried: task_id=21, input=tail.0 > output=es.3
[2022/05/11 12:20:09] [ info] [input] tail.0 resume (storage buf overlimit 22/128)
[2022/05/11 12:20:09] [ warn] [input] tail.0 paused (mem buf overlimit)
time="2022-05-11T12:20:11Z" level=error msg="[firehose 0] PutRecordBatch failed with AccessDeniedException: User: is not authorized to perform: firehose:PutRecordBatch on resource: because no identity-based policy allows the firehose:PutRecordBatch action\n\tstatus code: 400, request id: "
time="2022-05-11T12:20:11Z" level=error msg="[firehose 0] AccessDeniedException: User: is not authorized to perform: firehose:PutRecordBatch on resource: because no identity-based policy allows the firehose:PutRecordBatch action\n\tstatus code: 400, request id: \n"
```

**Steps to reproduce**
Disable the firehose, kinesis, and elastisearch plugins and look at pod logs.

**Expected outcome**
The firehose, kinesis, and elastisearch plugins are disabled with no pod log errors

**Environment**

* Chart name: aws-for-fluent-bit
* Chart version: 0.1.16
* Kubernetes version: 1.21
* Using EKS (yes/no), if so version? Yes 1.21

**Additional Context**:

Contributor guide

Open the contributing guide

Research direction

Inspect the aws-for-fluent-bit Helm chart's plugin enablement values and rendered Fluent Bit configuration. Reproduce the issue with the listed values and compare the rendered outputs. Done when disabled firehose, kinesis, and elasticsearch plugins are absent from the configuration and pod logs show no errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, helm, kubernetes
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.