grafana / grafana/lambda-promtail

lambda-promtail regex fails for vpc flow logs with hourly partition

Open
#9 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
11
Forks
42
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
When AWS VPC Flow logs are partitioned by every hour, rather than the default 24 hours, [it adds an 'hour' folder](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-s3-path.html)
> If you enable hourly partitions, the files are delivered to the following location.
> `bucket-and-optional-prefix/AWSLogs/account_id/vpcflowlogs/region/year/month/day/hour/`

and this causes the [regex to fail](https://github.com/grafana/loki/blob/63967b76102a9f85234526413a411ea2b63d9d1c/tools/lambda-promtail/lambda-promtail/s3.go#L83C49-L83C290), and result in `{"errorMessage":"type of S3 event could not be determined for object ...`

**To Reproduce**
Steps to reproduce the behavior:
1. Deploy `lambda-promtail`,
1. Create S3 bucket,
2. Create VPC flow log, with hourly partitioning:
```hcl
resource "aws_flow_log" "example" {
log_destination = aws_s3_bucket.example.arn
log_destination_type = "s3"
traffic_type = "ALL"
vpc_id = aws_vpc.example.id
destination_options {
file_format = "parquet"
per_hour_partition = true
}
}
```
3. Observe `lambda-promtail` logs

**Expected behavior**
Either:
* [update docs](https://grafana.com/docs/loki/latest/send-data/lambda-promtail/) to highlight unsupported hourly partition
* update regex

**Environment:**
- Infrastructure: AWS Lambda
- Deployment tool: Terraform

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.