elastic / elastic/logstash

Using the S3 input plugin, duplicate data collection occurs

Open
#17,840 2 comments 0 reactions 0 assignees View on GitHub
enhancement status:needs-triage
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

My Logstash version is 7.17.7. When deploying multiple Logstash nodes to simultaneously collect data from OBS buckets at the same time, there is a probabilistic occurrence of data duplication issues. Is there any way to solve this problem?

My configuration like this.
`input {
s3 {
access_key_id => "xxx"
codec => "plain"
secret_access_key => "xxx"
region => "xxx"
bucket => "obs-test"
prefix => "test/log"
interval => 3
delete => true
endpoint => "xxx"
temporary_directory => "xxx"
}
}

output {
file {
path => ["xxx"]
}
}`

I used delete => true, but in a multi-node scenario, the file might not have been deleted yet and could be read simultaneously by other nodes.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.