[Filebeat][input/aws-s3]: backup_to_bucket_arn and delete_after_backup do not work
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
`backup_to_bucket_arn` and `delete_after_backup` options do not work. The filebeat keeps processing files from the `bucket_arn` over and over again, and they are never deleted and placed in `delete_after_backup` bucket.
This was tested with version `8.15.3`.
### Example
Filebeat configuration:
```yaml
filebeat.inputs:
- type: aws-s3
enabled: true
session_token: x
access_key_id: x
secret_access_key: x
bucket_arn: arn:aws:s3:::constanca-test-s3-backup
backup_to_bucket_arn: arn:aws:s3:::constanca-test-s3-backup-delete
delete_after_backup: true
number_of_workers: 5
bucket_list_interval: 20s
expand_event_list_from_field: Records
```
Logs that prove filebeat finds the files:
```
{"log.level":"debug","@timestamp":"2024-11-26T08:44:57.564Z","log.logger":"input.aws-s3.s3","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*s3ObjectProcessor).ProcessS3Object","file.name":"awss3/s3_objects.go","file.line":129},"message":"Begin S3 object processing.","service.name":"filebeat","id":"6F32504842815259","bucket_arn":"constanca-test-s3-backup","object_key":"export.ndjson","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-26T08:44:57.564Z","log.logger":"input.aws-s3.s3","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*s3ObjectProcessor).ProcessS3Object","file.name":"awss3/s3_objects.go","file.line":129},"message":"Begin S3 object processing.","service.name":"filebeat","id":"6F32504842815259","bucket_arn":"constanca-test-s3-backup","object_key":"gopher.png","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-26T08:44:57.673Z","log.logger":"input.aws-s3.s3","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*s3ObjectProcessor).ProcessS3Object.func1","file.name":"awss3/s3_objects.go","file.line":137},"message":"End S3 object processing.","service.name":"filebeat","id":"6F32504842815259","bucket_arn":"constanca-test-s3-backup","object_key":"export.ndjson","elapsed_time_ns":108787233,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-26T08:44:57.775Z","log.logger":"input.aws-s3.s3","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*s3ObjectProcessor).ProcessS3Object.func1","file.name":"awss3/s3_objects.go","file.line":137},"message":"End S3 object processing.","service.name":"filebeat","id":"6F32504842815259","bucket_arn":"constanca-test-s3-backup","object_key":"gopher.png","elapsed_time_ns":210936613,"ecs.version":"1.6.0"}
```
However, the file remains in `bucket_arn`:

And `backup_to_bucket_arn` remains empty:

Contributor guide
Research direction
Start with awss3/s3_objects.go, especially the ProcessS3Object path named in the logs, and reproduce the supplied aws-s3 configuration on Filebeat 8.15.3. Trace handling of backup_to_bucket_arn and delete_after_backup. Done means processed objects are copied to the backup bucket, deleted from the source when requested, and not processed repeatedly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100