[ECS] [fluentbit]: handle [STREAM_TASK] imported from S3 appropriately
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
Currently init script treats downloaded or local config files containing SQL stream configurations as a normal files to include. Just like parsers this is a wrong approach. Although the container does not crash, `@INCLUDE` does not treat any `[STREAM_TASK]` as a useful config - this has to be set as a startup argument of fluent-bit or provided as streams_file config in the service section.
Ideally the init Go script should treat files containing string `[STREAM_TASK]` as an exception similar to `[PARSER]` and add it to the `[SERVICE]` section.
### Original Configuration
Original configuration before migration to S3 loading:
```
[SERVICE]
Daemon Off
Flush 5
Log_Level error
Parsers_File custom-parsers.conf
Streams_File custom-streams.conf
[FILTER]
name multiline
Match *
Buffer True
multiline.key_content log
mode partial_message
[FILTER]
Name modify
Match *
Add system_name myserv
[FILTER]
Name parser
Match *
Key_Name log
Parser json_parser
Preserve_Key On
Reserve_Data True
[OUTPUT]
Name s3
Match logs.s3
region ${AWS_REGION}
bucket ${BUCKET_NAME}
s3_key_format /${AWS_ACCOUNT_ID}/${AWS_REGION}/${SERVICE}/%Y/%m/%d/%H/$UUID.json
canned_acl bucket-owner-full-control
auto_retry_requests true
upload_timeout ${UPLOAD_TIMEOUT}
total_file_size ${TOTAL_FILE_SIZE}
use_put_object On
[OUTPUT]
Name datadog
Match logs.datadog
Host http-intake.logs.datadoghq.eu
TLS on
compress gzip
apikey ${DD_API_KEY}
dd_service ${DD_SERVICE}
dd_source ${SOURCE}
dd_message_key log
dd_tags ${DD_TAG_LIST}
```
**custom-streams.conf**
```
[STREAM_TASK]
Name datadog_logs
Exec CREATE STREAM datadog WITH (tag='logs.datadog') AS SELECT * from TAG:'*myserv*' WHERE extra['target'] = 'datadog';
[STREAM_TASK]
Name s3_logs
Exec CREATE STREAM s3 WITH (tag='logs.s3') AS SELECT * from TAG:'*myserv*' WHERE @record.contains(container_id) AND NOT @record.contains(extra['target']);
```
**custom-parsers.conf**
```
[PARSER]
Name docker
Format json
Time_Keep On
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
[PARSER]
Name json
Format json
[PARSER]
Name json_parser
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
Decode_Field escaped_utf8 log_processed do_next
Decode_Field_As json log_processed
```
### New configuration
In the new setup I uploaded all the files to S3 and added `aws_fluent_bit_init_s3_1` to `aws_fluent_bit_init_s3_3` variables with full object ARNs.
The parsers file is included correctly into the command. The streams file is used as `@INCLUDE`. I changed the main config file like this:
```
[SERVICE]
Daemon Off
Flush 5
Log_Level error
#Parsers_File custom-parsers.conf
#Streams_File custom-streams.conf
...
```
The setup works when I uncomment and change `Streams_File` to `/init/fluent-bit-init-s3-files/custom-streams.conf`. However, this is kind of a guessing game. Ideally the file should be properly handled by the init script.
### Fluent Bit Version Info
AWS for FluentBit init-2.34.1
FluentBit v1.19
### Cluster Details
ECS, sidecar deployment of FluentBit
**Which service(s) is this request for?**
ECS (Fluentbit)
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
Ideally the init Go script should treat files containing string [STREAM_TASK] as an exception similar to [PARSER] and add it to the [SERVICE] section.
**Are you currently working around this issue?**
Not working around, we use files baked into the container and for each service it's a different Dockerfile + tag.
**Additional context**
Nothing extra
**Attachments**
Original issue: https://github.com/aws/aws-for-fluent-bit/issues/1028
Contributor guide
Assessment
This issue has not been assessed yet.