elastic / elastic/logstash

Sending Data from azure blob storage to ES using logstash ?

Open
#7,748 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

- Version: Logstash 2.4.0
- Operating System: WIN 10
- Config File (if you have sensitive info, please remove it):
```
input
{
azureblob
{
storage_account_name => "mystorageaccount"
storage_access_key => "VGhpcyBpcyBhIGZha2Uga2V5Lg=="
container => "mycontainer"
}
}
output {
elasticsearch { hosts => ["localhost:9200"]
}
}
```
- Sample Data:
[2017-04-26 20:39:54,633][TRACE][index.search.slowlog.query] [data-0] [content-apr-2017][0] took[82.1micros], took_millis[0], types[publishedarticle], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[{"query":{"bool":{"must":[{"terms":{"List.Id":[316271282]}}]}}}], extra_source[],
[2017-04-26 20:39:54,633][TRACE][index.search.slowlog.query] [data-0] [content-apr-2017][0] took[82.1micros], took_millis[0], types[publishedarticle], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[{"query":{"bool":{"must":[{"terms":{"List.Id":[316271292]}}]}}}], extra_source[],
[2017-04-26 20:39:54,633][TRACE][index.search.slowlog.query] [data-0] [content-apr-2017][0] took[82.1micros], took_millis[0], types[publishedarticle], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[{"query":{"bool":{"must":[{"terms":{"List.Id":[316271382]}}]}}}], extra_source[],

- Steps to Reproduce:
1) Create a storage account in azure and copied raw data .txt file to a container into that account.
2) With logstash azureblob [plugin](https://github.com/Azure/azure-diagnostics-tools/tree/master/Logstash/logstash-input-azureblob) sending it to local ES cluster using above logstash config file.

I am getting error like this in logstash

Caught exception while locking {:exception=>#BlobTypeNotSupportedPage blob is not supported for this account type.\nRequestId:a1be676a-0001-0050-3fb9-002df1000000\nTime:2017-07-19T18:06:18.1830830Z" @url=# @request=# @request_headers={"User-Agent"=>"Azure-Core/0.1.7", "x-ms-date"=>"Wed, 19 Jul 2017 18:06:17 GMT", "x-ms-version"=>"2014-02-14", "DataServiceVersion"=>"1.0;NetFx", "MaxDataServiceVersion"=>"2.0;NetFx", "Content-Type"=>"application/atom+xml; charset=utf-8", "x-ms-blob-type"=>"PageBlob", "Content-Length"=>"0", "x-ms-blob-content-length"=>"512", "x-ms-sequence-number"=>"0", "Authorization"=>"SharedKey azureexplorerblob:FXkteDw28TnQ7LuQRdRy+DmmLxzQZtHrce9E2lthLoI="} @ssl=# @response=# @response_headers={"content-length"=>"238", "content-type"=>"application/xml", "server"=>"Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0", "x-ms-request-id"=>"a1be676a-0001-0050-3fb9-002df1000000", "x-ms-version"=>"2014-02-14", "date"=>"Wed, 19 Jul 2017 18:06:17 GMT", "connection"=>"close"} @status=400>>, @uri=#>, @uri: #, @description: "Page blob is not supported for this account type.\nRequestId:a1be676a-0001-0050-3fb9-002df1000000\nTime:2017-07-19T18:06:18.1830830Z", @type: "BlobTypeNotSupported">, :level=>:error}

But when i changed the storage account type from blob to general storage when i executed the first time , i can see the output send to ES . When i did the same again it is not sending any output to ES and also didn't showing any error in logstash too(i.e.Even i kept `stdout { codec => rubydebug }` in output ).

How can i resolve this and send data to ES ?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.