elastic / elastic/beats

[Filebeat] AWS CloudWatch Input add multiline support

Open
#23,575 9 comments 7 reactions 0 assignees View on GitHub
enhancement Team:Integrations
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

Mirroring request for S3 multiline input, https://github.com/elastic/beats/issues/23350

Describe the enhancement:
At the moment Filebeat AWS CloudWatch Input doesn't offer multiline support

Describe a specific use case for the enhancement or feature:
In AWS Cloudwatch streaming to SQS there might are occasions for log files that hold multiline log lines like for instance AWS Hadoop or other Java products.

We currently have AWS SQS logs in Cloudwatch that need multiline processing in order to get into ELK pipeline.

https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-s3.html
https://github.com/elastic/beats/blob/master/x-pack/filebeat/input/s3/config.go#L12
Would it be possible to add multiline support please like there exist for logs input already??
https://www.elastic.co/guide/en/beats/filebeat/master/multiline-examples.html
https://github.com/elastic/beats/blob/master/filebeat/input/log/config.go#L34
https://github.com/elastic/beats/blob/master/filebeat/input/log/config.go#L70

Contributor guide

Open the contributing guide

Research direction

Compare the requested CloudWatch input behavior with the S3 multiline request in issue 23350. Start with x-pack/filebeat/input/s3/config.go and the multiline-related sections of filebeat/input/log/config.go, then review the linked multiline examples. Done means the AWS CloudWatch input supports multiline processing for logs such as Hadoop or Java output and its behavior is covered by the relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.