elastic / elastic/beats

[Filebeat][input/aws-s3]: backup_to_bucket_arn and delete_after_backup do not work

Open
#41,784 3 comments 1 reaction 0 assignees View on GitHub
bug Team:obs-ds-hosted-services
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

`backup_to_bucket_arn` and `delete_after_backup` options do not work. The filebeat keeps processing files from the `bucket_arn` over and over again, and they are never deleted and placed in `delete_after_backup` bucket.

This was tested with version `8.15.3`.

### Example

Filebeat configuration:

```yaml
filebeat.inputs:
- type: aws-s3
enabled: true
session_token: x
access_key_id: x
secret_access_key: x
bucket_arn: arn:aws:s3:::constanca-test-s3-backup
backup_to_bucket_arn: arn:aws:s3:::constanca-test-s3-backup-delete
delete_after_backup: true
number_of_workers: 5
bucket_list_interval: 20s
expand_event_list_from_field: Records
```

Logs that prove filebeat finds the files:

```
{"log.level":"debug","@timestamp":"2024-11-26T08:44:57.564Z","log.logger":"input.aws-s3.s3","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*s3ObjectProcessor).ProcessS3Object","file.name":"awss3/s3_objects.go","file.line":129},"message":"Begin S3 object processing.","service.name":"filebeat","id":"6F32504842815259","bucket_arn":"constanca-test-s3-backup","object_key":"export.ndjson","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-26T08:44:57.564Z","log.logger":"input.aws-s3.s3","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*s3ObjectProcessor).ProcessS3Object","file.name":"awss3/s3_objects.go","file.line":129},"message":"Begin S3 object processing.","service.name":"filebeat","id":"6F32504842815259","bucket_arn":"constanca-test-s3-backup","object_key":"gopher.png","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-26T08:44:57.673Z","log.logger":"input.aws-s3.s3","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*s3ObjectProcessor).ProcessS3Object.func1","file.name":"awss3/s3_objects.go","file.line":137},"message":"End S3 object processing.","service.name":"filebeat","id":"6F32504842815259","bucket_arn":"constanca-test-s3-backup","object_key":"export.ndjson","elapsed_time_ns":108787233,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-26T08:44:57.775Z","log.logger":"input.aws-s3.s3","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*s3ObjectProcessor).ProcessS3Object.func1","file.name":"awss3/s3_objects.go","file.line":137},"message":"End S3 object processing.","service.name":"filebeat","id":"6F32504842815259","bucket_arn":"constanca-test-s3-backup","object_key":"gopher.png","elapsed_time_ns":210936613,"ecs.version":"1.6.0"}
```

However, the file remains in `bucket_arn`:
![Image](https://github.com/user-attachments/assets/55a11cd7-7581-4921-9393-866bf02ead55)

And `backup_to_bucket_arn` remains empty:
![Image](https://github.com/user-attachments/assets/3a53e42e-7dfd-41cb-acc0-0657dbe178b9)

Contributor guide

Open the contributing guide

Research direction

Start with awss3/s3_objects.go, especially the ProcessS3Object path named in the logs, and reproduce the supplied aws-s3 configuration on Filebeat 8.15.3. Trace handling of backup_to_bucket_arn and delete_after_backup. Done means processed objects are copied to the backup bucket, deleted from the source when requested, and not processed repeatedly.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.