elastic / elastic/integrations
[aws.s3access] Grok fails on S3 Compute checksum records (4-part operation, UUID request ID)
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 182
Description
### Integration Name
AWS [packages/aws]
### Dataset Name
aws.s3access
### Integration Version
(all, incl. 7.0.0)
### Problem
Two `pattern_definitions` in the `aws.s3access` ingest pipeline are too narrow to match log records that AWS explicitly documents. Both cause the whole grok to fail, so the affected events are indexed with `event.kind: pipeline_error` instead of being parsed.
Both are visible in [`data_stream/s3access/elasticsearch/ingest_pipeline/default.yml`](https://github.com/elastic/integrations/blob/main/packages/aws/data_stream/s3access/elasticsearch/ingest_pipeline/default.yml#L43-L51):
```yaml
S3REQUESTID: "[a-zA-Z0-9]+"
S3OPERATION: "%{WORD}.%{WORD}.%{WORD}"
```
#### 1. `S3OPERATION` cannot match four-part operations
The pattern has exactly three `WORD` slots. The [S3 server access log format docs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/LogFormat.html) document `S3.COMPUTE.OBJECT.CHECKSUM` for the *Compute checksum* operation, which has four components. Since `\w+` cannot match a `.`, a four-part operation is structurally unmatchable — no amount of backtracking helps.
The **Operation** field docs list it directly:
> **Example entry**
> ```
> REST.PUT.OBJECT
> S3.COMPUTE.OBJECT.CHECKSUM
> ```
#### 2. `S3REQUESTID` does not allow hyphens
For *Compute checksum* job requests, the **Request ID** field carries the associated job ID, which is a UUID. From the same docs:
> A string generated by Amazon S3 to uniquely identify each request. For **Compute checksum** job requests, the **Request ID** field displays the associated job ID.
The documented example record uses `e5042925-b524-4b3b-a869-f3881e78ff3a`. `[a-zA-Z0-9]+` rejects the hyphens.
### Reproducer
The AWS docs include a full example record for the Compute checksum operation, which fails on both counts:
```
7cd47ef2be amzn-s3-demo-bucket [06/Feb/2019:00:00:38 +0000] - 79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be e5042925-b524-4b3b-a869-f3881e78ff3a S3.COMPUTE.OBJECT.CHECKSUM example-object - - - - 1048576 - - - - - bPf7qjG4XwYdPgDQTl72GW/uotRhdPz2UryEyAFLDSRmKrakUkJCYLtAw6fdANcrsUYc1M/kIulXM1u5vZQT5g== - - - - - - - -
```
The two causes are independent — either one alone is enough to make the record fail to parse.
### Expected
Compute-checksum records parse into `aws.s3access.*` fields like any other record, rather than landing in the error path.
### Anything else?
Same class of problem as #20417 (`S3REQUESTER` too narrow for CloudFront OAI requesters), which is fixed in #20442. These two were found while reviewing that fix and are deliberately kept out of that PR to keep it scoped to the issue it closes.
Contributor guide
Research direction
Start in packages/aws/data_stream/s3access/elasticsearch/ingest_pipeline/default.yml at the S3REQUESTID and S3OPERATION pattern definitions. Compare them with the AWS Compute checksum example record, then validate that the record parses into aws.s3access fields instead of event.kind: pipeline_error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, elasticsearch
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100