elastic / elastic/beats

[filebeat] [aws-s3] `LastModified` is always Go time zero value when using SQS

Open
#45,566 2 comments 0 reactions 0 assignees View on GitHub
Stalled Team:obs-ds-hosted-services
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

When using the `aws-s3` input with SQS notifications, the `LastModified` field from S3 events is always parsed as the Go time zero value (`time.Time{}`), which leads to issues like duplicate IDs. https://github.com/elastic/beats/blob/8ff04e9cbe0c44dd1864a491982292998200f298/x-pack/filebeat/input/awss3/s3_objects.go#L462

A quick fix I tried that worked:

```diff
diff --git a/x-pack/filebeat/input/awss3/s3_objects.go b/x-pack/filebeat/input/awss3/s3_objects.go
index e120348329..5bbb9182eb 100644
--- a/x-pack/filebeat/input/awss3/s3_objects.go
+++ b/x-pack/filebeat/input/awss3/s3_objects.go
@@ -234,6 +234,9 @@ func (p *s3ObjectProcessor) download() (obj *s3DownloadedObject, err error) {
if getObjectOutput == nil {
return nil, fmt.Errorf("empty response from s3 get object: %w", err)
}
+ if getObjectOutput.LastModified != nil {
+ p.s3Obj.S3.Object.LastModified = *getObjectOutput.LastModified
+ }
s3RequestURL := getObjectOutput.ResultMetadata.Get(s3RequestURLMetadataKey)
if s3RequestURLAsString, ok := s3RequestURL.(string); ok {
p.s3RequestURL = s3RequestURLAsString
```

not sure if this breaks anything or creates unexpected behaviors, needs a bit more debugging

Contributor guide

Open the contributing guide

Research direction

Start in x-pack/filebeat/input/awss3/s3_objects.go, especially s3ObjectProcessor.download() and the LastModified handling near the referenced line. Reproduce the SQS notification path and compare the S3 event value with the GetObject response. Done means LastModified is preserved for SQS events without introducing unexpected behavior or duplicate IDs.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.