Using the S3 input plugin, duplicate data collection occurs
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 63
Description
My Logstash version is 7.17.7. When deploying multiple Logstash nodes to simultaneously collect data from OBS buckets at the same time, there is a probabilistic occurrence of data duplication issues. Is there any way to solve this problem?
My configuration like this.
`input {
s3 {
access_key_id => "xxx"
codec => "plain"
secret_access_key => "xxx"
region => "xxx"
bucket => "obs-test"
prefix => "test/log"
interval => 3
delete => true
endpoint => "xxx"
temporary_directory => "xxx"
}
}
output {
file {
path => ["xxx"]
}
}`
I used delete => true, but in a multi-node scenario, the file might not have been deleted yet and could be read simultaneously by other nodes.
Contributor guide
Assessment
This issue has not been assessed yet.