elastic / elastic/elastic-serverless-forwarder
ESF incorrectly sets region for VPC flow logs from multi-region S3 bucket
- Dominant language
- Python
- Stars
- 38
- Forks
- 50
- PR merge metrics
- No merged PRs in 30d
Description
- Version: 1.14.0
## Context
I am creating an issue for a user who reported it using a private channel.
> All VPC flow log data, regardless of originating AWS region, is stored in a single bucket in the `us-east-1` region.
In this scenario, the user consolidates VPC logs from multiple regions in a single S3 bucket and uses ESF to ingest all VPC logs from the S3 bucket.
ESF creates documents with the region of the S3 bucket instead of the region of the S3 object.
## Current behavior
My understanding of the region problem is that the user:
- Is sending all the VPC flow logs in a single S3 bucket; for example, a bucket named `my-vpcflow-logs` hosted on the region `eu-north-1`.
- Each VPC Flow log file is stored in an object key like the following:
```text
AWSLogs/123/vpcflowlogs/eu-central-11/2024/07/11/123_vpcflowlogs_eu-central-1_fl-0cea9cbf050c152d5_20240711T0000Z_123eabd0.log.gz
```
- When ESF processes the `AWSLogs/123/vpcflowlogs/eu-central-11/2024/07/11/123_vpcflowlogs_eu-central-1_fl-0cea9cbf050c152d5_20240711T0000Z_123eabd0.log.gz` object, it creates a document with the following fields:
```json
{
"cloud": {
"provider": "aws",
"region": "eu-north-1",
"account": {
"id": "123"
}
}
}
```
## Expected behavior
- However, since the `AWSLogs/123/vpcflowlogs/eu-central-11/2024/07/11/123_vpcflowlogs_eu-central-1_fl-0cea9cbf050c152d5_20240711T0000Z_123eabd0.log.gz` comes from the `eu-central-1` region, the user expects the document to have the following content:
```json
{
"cloud": {
"provider": "aws",
"region": "eu-central-1",
"account": {
"id": "123"
}
}
}
```
## Notes
In the current version, ESF [uses](https://github.com/elastic/elastic-serverless-forwarder/blob/cce7939c921e6034b060308fd531d253d73cf609/handlers/aws/s3_sqs_trigger.py#L104-L105) the region from the notification S3 published in the SQS queue.
Contributor guide
Assessment
This issue has not been assessed yet.