elastic / elastic/integrations

[aws.s3access] Grok fails on S3 Compute checksum records (4-part operation, UUID request ID)

Open Beginner friendly
#20,443 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 18h
Merged PRs (30d)
182

Description

### Integration Name

AWS [packages/aws]

### Dataset Name

aws.s3access

### Integration Version

(all, incl. 7.0.0)

### Problem

Two `pattern_definitions` in the `aws.s3access` ingest pipeline are too narrow to match log records that AWS explicitly documents. Both cause the whole grok to fail, so the affected events are indexed with `event.kind: pipeline_error` instead of being parsed.

Both are visible in [`data_stream/s3access/elasticsearch/ingest_pipeline/default.yml`](https://github.com/elastic/integrations/blob/main/packages/aws/data_stream/s3access/elasticsearch/ingest_pipeline/default.yml#L43-L51):

```yaml
S3REQUESTID: "[a-zA-Z0-9]+"
S3OPERATION: "%{WORD}.%{WORD}.%{WORD}"
```

#### 1. `S3OPERATION` cannot match four-part operations

The pattern has exactly three `WORD` slots. The [S3 server access log format docs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/LogFormat.html) document `S3.COMPUTE.OBJECT.CHECKSUM` for the *Compute checksum* operation, which has four components. Since `\w+` cannot match a `.`, a four-part operation is structurally unmatchable — no amount of backtracking helps.

The **Operation** field docs list it directly:

> **Example entry**
> ```
> REST.PUT.OBJECT
> S3.COMPUTE.OBJECT.CHECKSUM
> ```

#### 2. `S3REQUESTID` does not allow hyphens

For *Compute checksum* job requests, the **Request ID** field carries the associated job ID, which is a UUID. From the same docs:

> A string generated by Amazon S3 to uniquely identify each request. For **Compute checksum** job requests, the **Request ID** field displays the associated job ID.

The documented example record uses `e5042925-b524-4b3b-a869-f3881e78ff3a`. `[a-zA-Z0-9]+` rejects the hyphens.

### Reproducer

The AWS docs include a full example record for the Compute checksum operation, which fails on both counts:

```
7cd47ef2be amzn-s3-demo-bucket [06/Feb/2019:00:00:38 +0000] - 79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be e5042925-b524-4b3b-a869-f3881e78ff3a S3.COMPUTE.OBJECT.CHECKSUM example-object - - - - 1048576 - - - - - bPf7qjG4XwYdPgDQTl72GW/uotRhdPz2UryEyAFLDSRmKrakUkJCYLtAw6fdANcrsUYc1M/kIulXM1u5vZQT5g== - - - - - - - -
```

The two causes are independent — either one alone is enough to make the record fail to parse.

### Expected

Compute-checksum records parse into `aws.s3access.*` fields like any other record, rather than landing in the error path.

### Anything else?

Same class of problem as #20417 (`S3REQUESTER` too narrow for CloudFront OAI requesters), which is fixed in #20442. These two were found while reviewing that fix and are deliberately kept out of that PR to keep it scoped to the issue it closes.

Contributor guide

Open the contributing guide

Research direction

Start in packages/aws/data_stream/s3access/elasticsearch/ingest_pipeline/default.yml at the S3REQUESTID and S3OPERATION pattern definitions. Compare them with the AWS Compute checksum example record, then validate that the record parses into aws.s3access fields instead of event.kind: pipeline_error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, elasticsearch
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.