[filebeat] [aws-s3] `LastModified` is always Go time zero value when using SQS
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
When using the `aws-s3` input with SQS notifications, the `LastModified` field from S3 events is always parsed as the Go time zero value (`time.Time{}`), which leads to issues like duplicate IDs. https://github.com/elastic/beats/blob/8ff04e9cbe0c44dd1864a491982292998200f298/x-pack/filebeat/input/awss3/s3_objects.go#L462
A quick fix I tried that worked:
```diff
diff --git a/x-pack/filebeat/input/awss3/s3_objects.go b/x-pack/filebeat/input/awss3/s3_objects.go
index e120348329..5bbb9182eb 100644
--- a/x-pack/filebeat/input/awss3/s3_objects.go
+++ b/x-pack/filebeat/input/awss3/s3_objects.go
@@ -234,6 +234,9 @@ func (p *s3ObjectProcessor) download() (obj *s3DownloadedObject, err error) {
if getObjectOutput == nil {
return nil, fmt.Errorf("empty response from s3 get object: %w", err)
}
+ if getObjectOutput.LastModified != nil {
+ p.s3Obj.S3.Object.LastModified = *getObjectOutput.LastModified
+ }
s3RequestURL := getObjectOutput.ResultMetadata.Get(s3RequestURLMetadataKey)
if s3RequestURLAsString, ok := s3RequestURL.(string); ok {
p.s3RequestURL = s3RequestURLAsString
```
not sure if this breaks anything or creates unexpected behaviors, needs a bit more debugging
Contributor guide
Research direction
Start in x-pack/filebeat/input/awss3/s3_objects.go, especially s3ObjectProcessor.download() and the LastModified handling near the referenced line. Reproduce the SQS notification path and compare the S3 event value with the GetObject response. Done means LastModified is preserved for SQS events without introducing unexpected behavior or duplicate IDs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100