aws / aws/eks-charts

[aws-for-fluent-bit] kinesis_streams Does not support compression and aggregation

Open
#1,206 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Mustache
Stars
1.3k
Forks
1.1k
Avg merge
22m
Merged PRs (30d)
5

Description

Hi Team,

While using the below config I'm getting error that aggregation and compression is not supported.

I'm using `chart_revision: 0.1.35`

```
kinesis_streams:
enabled: true
match: '*'
region: us-east-1
stream: input-stream
role_arn: "arn:aws:iam::xxxxx:role/writer-role"
time_key: eventTime
time_key_format: "%Y-%m-%dT%H:%M:%S.%9NZ"
sts_endpoint: "https://sts.us-east-1.amazonaws.com"
partition_key: PROGRAM
extraOutputs: |
aggregation true
compression zlib
```

## Error

```
[error] [config] kinesis_streams: unknown configuration property 'compression'.
The following properties are allowed: region, stream, time_key, time_key_format, role_arn, endpoint, sts_endpoint, external_id, log_key, and auto_retry_requests.

[2025/03/21 18:43:31] [error] [config] kinesis_streams: unknown configuration property 'aggregation'. The following properties are allowed: region, stream, time_key, time_key_format, role_arn, endpoint, sts_endpoint, external_id, log_key, and auto_retry_requests.
[2025/03/21 18:43:31] [ help] try the command: /fluent-bit/bin/fluent-bit -o kinesis_streams -h

```

1. How I can use compression and aggregation in kinesis_streams?
2. Is there any documentation that tells what options can be used under extraOutputs?

Contributor guide

Open the contributing guide

Research direction

Start with the kinesis_streams configuration at chart_revision 0.1.35 and the Fluent Bit `-o kinesis_streams -h` output shown in the report. Verify whether extraOutputs is documented or supported, then document the available options and the status or limitation of compression and aggregation. Done means users can determine how to configure these settings or understand that they are unsupported.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, helm, yaml
Domain
cloud, documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.