elastic / elastic/elastic-serverless-forwarder

ESF incorrectly sets region for VPC flow logs from multi-region S3 bucket

Open
#741 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
38
Forks
50
PR merge metrics
No merged PRs in 30d

Description

- Version: 1.14.0

## Context

I am creating an issue for a user who reported it using a private channel.

> All VPC flow log data, regardless of originating AWS region, is stored in a single bucket in the `us-east-1` region.

In this scenario, the user consolidates VPC logs from multiple regions in a single S3 bucket and uses ESF to ingest all VPC logs from the S3 bucket.

ESF creates documents with the region of the S3 bucket instead of the region of the S3 object.

## Current behavior

My understanding of the region problem is that the user:

- Is sending all the VPC flow logs in a single S3 bucket; for example, a bucket named `my-vpcflow-logs` hosted on the region `eu-north-1`.
- Each VPC Flow log file is stored in an object key like the following:

```text
AWSLogs/123/vpcflowlogs/eu-central-11/2024/07/11/123_vpcflowlogs_eu-central-1_fl-0cea9cbf050c152d5_20240711T0000Z_123eabd0.log.gz
```

- When ESF processes the `AWSLogs/123/vpcflowlogs/eu-central-11/2024/07/11/123_vpcflowlogs_eu-central-1_fl-0cea9cbf050c152d5_20240711T0000Z_123eabd0.log.gz` object, it creates a document with the following fields:

```json
{
"cloud": {
"provider": "aws",
"region": "eu-north-1",
"account": {
"id": "123"
}
}
}
```

## Expected behavior

- However, since the `AWSLogs/123/vpcflowlogs/eu-central-11/2024/07/11/123_vpcflowlogs_eu-central-1_fl-0cea9cbf050c152d5_20240711T0000Z_123eabd0.log.gz` comes from the `eu-central-1` region, the user expects the document to have the following content:

```json
{
"cloud": {
"provider": "aws",
"region": "eu-central-1",
"account": {
"id": "123"
}
}
}
```

## Notes

In the current version, ESF [uses](https://github.com/elastic/elastic-serverless-forwarder/blob/cce7939c921e6034b060308fd531d253d73cf609/handlers/aws/s3_sqs_trigger.py#L104-L105) the region from the notification S3 published in the SQS queue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.