grafana / grafana/lambda-promtail

lambda-promtail: ALB connection logs are not properly recognized by the `defaultFilenameRegex` expression

Open
#23 8 comments 0 reactions 0 assignees View on GitHub
triage
Dominant language
Go
Stars
11
Forks
42
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**

AWS released connection logging for Application Load Balancers alongside their new mTLS functionality: https://aws.amazon.com/blogs/aws/mutual-authentication-for-application-load-balancer-to-reliably-verify-certificate-based-client-identities/ in November 2023.

When one activates those logs they are shipped to the ELB log bucket alongside the already existing access logs. However connection log files have a different file name structure than access logs where [they are prefixed with the string "conn_log_".](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-connection-logs.html#connection-log-file-format)

⚠️ Unfortunately the AWS documentation states the prefix ends with a period, however in reality an underscore is used!

The `defaultFilenameRegex` does not cover those logs currently which is why below error is thrown:

`level=error caller=main.go:269 err="error processing event: type of S3 event could not be determined for object \"AWSLogs/347414xxxxxx/elasticloadbalancing/eu-central-1/2024/02/07/conn_log_347414xxxxxx_elasticloadbalancing_eu-central-1_app.xxxxxxxxx.5ede558f4b0e97a9_20240207T0955Z_10.43.xxx.xxx_b38hdhc6.log.gz\""`

https://github.com/grafana/loki/blob/46c6118eb2a2d56406042431dfbed288df6e924b/tools/lambda-promtail/lambda-promtail/s3.go#L78

**To Reproduce**
Steps to reproduce the behavior:
1. Enable connection logs on an Application Load Balancer.
2. Start scraping ELB logs from the referenced bucket using lambda-promtail.

**Expected behavior**
Connection log files should be recognized as valid `s3_lb` log files and handled by the parser. This could be achieve by adjusting the regex like so: `AWSLogs\/(?P\d+)\/(?P[a-zA-Z0-9_\-]+)\/(?P[\w-]+)\/(?P\d+)\/(?P\d+)\/(?P\d+)\/(?:conn_log_)?\d+\_(?:elasticloadbalancing|vpcflowlogs)\_\w+-\w+-\d_(?:(?Papp|net)\.*?)?(?P[a-zA-Z0-9\-]+)`

**Environment:**
- Infrastructure: AWS, AWS Lambda, AWS S3
- Deployment tool: Terraform

**Screenshots, Promtail config, or terminal output**

`level=error caller=main.go:269 err="error processing event: type of S3 event could not be determined for object \"AWSLogs/347414xxxxxx/elasticloadbalancing/eu-central-1/2024/02/07/conn_log_347414xxxxxx_elasticloadbalancing_eu-central-1_app.xxxxxxxxx.5ede558f4b0e97a9_20240207T0955Z_10.43.xxx.xxx_b38hdhc6.log.gz\""`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.