elastic / elastic/integrations

[AWS lambda_logs]: Support for logs shipped to S3

Open
#14,490 1 comment 0 reactions 0 assignees View on GitHub
Integration:aws needs:triage Stalled Team:obs-ds-hosted-services
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 17h
Merged PRs (30d)
225

Description

### Integration Name

AWS [aws]

### Dataset Name

_No response_

### Integration Version

v0.112.0

### Agent Version

8.17.4

### OS Version and Architecture

Kubernetes

### User Goal

Ingest AWS Lambda Logs that are Shipped using the following scenario https://docs.aws.amazon.com/lambda/latest/dg/logging-with-s3.html (not Cloudwatch logs). When they get to the S3 bucket, the logs are compressed using ZSTD https://facebook.github.io/zstd/ and appear with a `.zst` suffix. We haven't found a way to change that in the AWS tools configurations. Can we add a way to decompress those files at the beginning of the pipeline ? We've opened an issue on the beats component : https://github.com/elastic/beats/issues/45292

We are using https://github.com/elastic/integrations/blob/main/packages/aws/data_stream/lambda_logs/elasticsearch/ingest_pipeline/aws-lambda-json.yml but since the logs are not decompressed we get binary information in the `message` and the pipeline cannot extract the info.

### Existing Features

We are working with the https://github.com/elastic/integrations/blob/main/packages/aws/data_stream/lambda_logs/elasticsearch/ingest_pipeline/aws-lambda-json.yml existing pipeline.

### What did you see?

Image

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.