elastic / elastic/integrations

[Amazon S3]: Grok Parsing Error with Valid S3 Object Key Characters

Open
#13,617 1 comment 0 reactions 0 assignees View on GitHub
Integration:aws needs:triage Stalled Team:obs-ds-hosted-services
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 17h
Merged PRs (30d)
225

Description

### Integration Name

AWS [aws]

### Dataset Name

aws.s3access

### Integration Version

v2.45.2

### Agent Version

8.17.4

### Agent Output Type

elasticsearch

### Elasticsearch Version

8.17.4

### OS Version and Architecture

elastic-agent-wolfi:8.17.4

### Software/API Version

_No response_

### Error Message

```
Processor 'grok' with tag '' failed with message 'Provided Grok expressions do not match field value: [28bdbd9102b76c3da07c44e094b16b382f90f94be71626f4d9589b1309d2d7ef my-bucket [31/Mar/2025:22:33:24 +0000] 18.68.18.10 97cb88763cd5f6a666a34ac23fec99f53137a309c78d86459aa421f2c82e82877e77c206a23ad9e41ecc07e3c586e292 MD9TCAF28XVJT7H7 REST.GET.OBJECT path/thumbnails/75*75/image-1.jpeg "GET /path/thumbnails/75*75/image-1.jpeg HTTP/1.1" 200 - 8642 8642 103 102 "-" "Amazon CloudFront" - 79o5Ho+xysFrip1ZRvYqOd2rhlOcCxkbPMPJ88s7YaM/k797fr+9ZCXsLvKs3yW7nhOta+g6yAI= SigV4 TLS_AES_128_GCM_SHA256 AuthHeader my-bucket.s3.amazonaws.com TLSv1.3 - -]'
```

### Event Original

```
28bdbd9102b76c3da07c44e094b16b382f90f94be71626f4d9589b1309d2d7ef my-bucket [02/Apr/2025:23:10:13 +0000] 64.252.71.106 97cb88763cd5f6a666a34ac23fec99f53137a309c78d86459aa421f2c82e82877e77c206a23ad9e41ecc07e3c586e292 0FMAC0C5SEXE32V4 REST.GET.OBJECT path/thumbnails/75*75/image-1.jpeg "GET /path/thumbnails/75*75/image-1.jpeg HTTP/1.1" 200 - 8208 8208 69 68 "-" "Amazon CloudFront" - bOe41Qirrpdy0aueV7z6WG1YGLVKBg64L0ghi4go4PRTOrxJxEStT196gst2/CURe0Lar7wQE0YEDWfjiiqhWZ7AO4Dgf0ileq6xldSGrTk= SigV4 TLS_AES_128_GCM_SHA256 AuthHeader my-bucket.s3.amazonaws.com TLSv1.3 - -
```

### What did you do?

I created a Case [01850523] with Support. The finding was that there is a problem with parsing the bucket key because of the `*` character in the name. We didn't really have any other option around this besides editing the first Grok processor step in the `logs-aws.s3access-*` pipeline by updating the pattern definition for "S3KEY" to allow for the `*` character as follows:
```
"S3KEY": """[a-zA-Z0-9\/_\.\*\-%+]+"""
```

![Image](https://github.com/user-attachments/assets/797b2ecf-1fff-4634-804c-e6902b2bb634)

### What did you see?

After updating the processor, the bucket access logs were successfully being parsed - the solution worked!

### What did you expect to see?

The `*` should have been a supported character all along because it is included in AWS's Safe Characters for S3 Object names: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html#object-key-guidelines

I would expect that all characters in the Safe Characters list are supported.

![Image](https://github.com/user-attachments/assets/7268ce6e-5de5-4d38-92a8-4c2423a21ab7)

### Anything else?

I filed this as a Bug instead of a Feature Enhancement because the `*` is a supported character by S3.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.