elastic / elastic/integrations
[AWS lambda_logs]: Support for logs shipped to S3
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
### Integration Name
AWS [aws]
### Dataset Name
_No response_
### Integration Version
v0.112.0
### Agent Version
8.17.4
### OS Version and Architecture
Kubernetes
### User Goal
Ingest AWS Lambda Logs that are Shipped using the following scenario https://docs.aws.amazon.com/lambda/latest/dg/logging-with-s3.html (not Cloudwatch logs). When they get to the S3 bucket, the logs are compressed using ZSTD https://facebook.github.io/zstd/ and appear with a `.zst` suffix. We haven't found a way to change that in the AWS tools configurations. Can we add a way to decompress those files at the beginning of the pipeline ? We've opened an issue on the beats component : https://github.com/elastic/beats/issues/45292
We are using https://github.com/elastic/integrations/blob/main/packages/aws/data_stream/lambda_logs/elasticsearch/ingest_pipeline/aws-lambda-json.yml but since the logs are not decompressed we get binary information in the `message` and the pipeline cannot extract the info.
### Existing Features
We are working with the https://github.com/elastic/integrations/blob/main/packages/aws/data_stream/lambda_logs/elasticsearch/ingest_pipeline/aws-lambda-json.yml existing pipeline.
### What did you see?
### Anything else?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.