Azure / Azure/azure-diagnostics-tools
Logstash Azure Blob plugin doesn't see all the files
- Dominant language
- Ruby
- Stars
- 101
- Forks
- 93
- PR merge metrics
- No merged PRs in 30d
Description
Hi folks,
I've started using Logstash Azure Blob plugin recently, all seemed to be pretty well in TEST, but when it came to PROD with thousands of jsons I'm getting into troubles.
The issue is - logstash can't read ALL the jsons stored on a blobl storage account, it picks them up randomly and only few of them, I'm running logstash in a debug mode and can't see any errors except the one appearing from time to time:
(412) A lease >ID was specified, but the lease for the blob has expired
however is still running and getting some of files into Elastic
Here is my config:
```
input
{
azureblob
{
storage_account_name => "errorlogsstorage"
storage_access_key => "mykeyhere"
container => "errorlogs"
codec => "json"
interval => "1"
registry_create_policy => "start_over"
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["http://10.78.0.75:9200"]
index => "errorlogs"
codec => json
}
}
```
Maybe anyone has experienced that same and could share a solution ? Any inputs are much appreciated
Thanks !
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.