[Filebeat] AWS CloudWatch Input add multiline support
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
Mirroring request for S3 multiline input, https://github.com/elastic/beats/issues/23350
Describe the enhancement:
At the moment Filebeat AWS CloudWatch Input doesn't offer multiline support
Describe a specific use case for the enhancement or feature:
In AWS Cloudwatch streaming to SQS there might are occasions for log files that hold multiline log lines like for instance AWS Hadoop or other Java products.
We currently have AWS SQS logs in Cloudwatch that need multiline processing in order to get into ELK pipeline.
https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-s3.html
https://github.com/elastic/beats/blob/master/x-pack/filebeat/input/s3/config.go#L12
Would it be possible to add multiline support please like there exist for logs input already??
https://www.elastic.co/guide/en/beats/filebeat/master/multiline-examples.html
https://github.com/elastic/beats/blob/master/filebeat/input/log/config.go#L34
https://github.com/elastic/beats/blob/master/filebeat/input/log/config.go#L70
Contributor guide
Research direction
Compare the requested CloudWatch input behavior with the S3 multiline request in issue 23350. Start with x-pack/filebeat/input/s3/config.go and the multiline-related sections of filebeat/input/log/config.go, then review the linked multiline examples. Done means the AWS CloudWatch input supports multiline processing for logs such as Hadoop or Java output and its behavior is covered by the relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100