Azure / Azure/azure-diagnostics-tools
logstash-input-azureblob (0.9.13) lease issues with data/registry
- Dominant language
- Ruby
- Stars
- 101
- Forks
- 93
- PR merge metrics
- No merged PRs in 30d
Description
Hi all,
I'm using logstash-input-azureblob (0.9.13) to process NSG Flow Logs from our Azure environment. Initially when I set it up, there were no issues and logs were being processed fine. However, at some point data stopped being processed. Logstash has the following errors in the log file which indicate that the issue is lease related:
```
[2018-12-20T11:24:22,648][ERROR][logstash.inputs.logstashinputazureblob] Oh My, An error occurred. LeaseLost (412): A lease ID was specified, but the lease for the blob has expired.
RequestId:c3034529-001e-002c-0656-98c448000000
Time:2018-12-20T11:24:10.5736280Z: ["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/azure-core-0.1.15/lib/azure/core/http/retry_policy.rb:58:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/azure-core-0.1.15/lib/azure/core/http/http_request.rb:110:in `block in with_filter'", "org/jruby/RubyMethod.java:111:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/azure-core-0.1.15/lib/azure/core/http/signer_filter.rb:28:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/azure-core-0.1.15/lib/azure/core/http/http_request.rb:110:in `block in with_filter'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/azure-core-0.1.15/lib/azure/core/service.rb:36:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/azure-core-0.1.15/lib/azure/core/filtered_service.rb:34:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/azure-core-0.1.15/lib/azure/core/signed_service.rb:41:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/azure-storage-0.15.0.preview/lib/azure/storage/service/storage_service.rb:62:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/azure-storage-0.15.0.preview/lib/azure/storage/blob/blob_service.rb:62:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/azure-storage-0.15.0.preview/lib/azure/storage/blob/block.rb:118:in `create_block_blob'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:479:in `save_registry'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:460:in `create_registry'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:357:in `register_for_read'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:165:in `process'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:151:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:409:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:403:in `block in start_input'"] {:exception=>#LeaseLostA lease ID was specified, but the lease for the blob has expired.\nRequestId:c3034529-001e-002c-0656-98c448000000\nTime:2018-12-20T11:24:10.5736280Z" @url=# @request=# @request_headers={"User-Agent"=>"logstash-input-azureblob/0.9.11; Azure-Storage/0.15.0-preview (Ruby 2.3.3-p0; Linux linux)", "x-ms-date"=>"Thu, 20 Dec 2018 11:23:52 GMT", "x-ms-version"=>"2016-05-31", "DataServiceVersion"=>"1.0;NetFx", "MaxDataServiceVersion"=>"3.0;NetFx", "Content-Type"=>"application/atom+xml; charset=utf-8", "x-ms-blob-type"=>"BlockBlob", "x-ms-lease-id"=>"*****", "x-ms-blob-content-type"=>"text/plain; charset=UTF-8", "Content-Length"=>"100981", "Content-MD5"=>"*****", "Authorization"=>"SharedKey *****:*****"} @ssl=# @response=# @response_headers={"content-length"=>"243", "content-type"=>"application/xml", "server"=>"Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0", "x-ms-request-id"=>"c3034529-001e-002c-0656-98c448000000", "x-ms-version"=>"2016-05-31", "date"=>"Thu, 20 Dec 2018 11:24:10 GMT", "connection"=>"close"} @status=412>>, @uri=#>, @uri: #, @description: "A lease ID was specified, but the lease for the blob has expired.\nRequestId:c3034529-001e-002c-0656-98c448000000\nTime:2018-12-20T11:24:10.5736280Z", @type: "LeaseLost">}
```
Checking the data/registry lease it says "expired". I've tried acquiring and releasing the lease which puts it in "Available" state then restarting logstash.
Still no data comes through then checking the lease state of data/registry after a few minutes it says "Expired" again. However, the above error doesn't appear again - there's just no data being processed.
There is definitely new data in the blob storage.
Other things I've tried:
* deleting "data/registry". It gets re-created after restarting logstash but still no data processed
* reinstalling logstash-input-azureblob
Any assistance would be much appreciated!
Gen.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.