Azure / Azure/azure-diagnostics-tools

Logstash Azure Blob plugin doesn't see all the files

Open
#222 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Ruby
Stars
101
Forks
93
PR merge metrics
No merged PRs in 30d

Description

Hi folks,
I've started using Logstash Azure Blob plugin recently, all seemed to be pretty well in TEST, but when it came to PROD with thousands of jsons I'm getting into troubles.
The issue is - logstash can't read ALL the jsons stored on a blobl storage account, it picks them up randomly and only few of them, I'm running logstash in a debug mode and can't see any errors except the one appearing from time to time:
(412) A lease >ID was specified, but the lease for the blob has expired
however is still running and getting some of files into Elastic
Here is my config:
```
input
{
azureblob
{
storage_account_name => "errorlogsstorage"
storage_access_key => "mykeyhere"
container => "errorlogs"
codec => "json"
interval => "1"
registry_create_policy => "start_over"
}
}

output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["http://10.78.0.75:9200"]
index => "errorlogs"
codec => json
}
}
```

Maybe anyone has experienced that same and could share a solution ? Any inputs are much appreciated
Thanks !

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.