Azure / Azure/azure-diagnostics-tools

One logstash input for Azure blob is slower than other

Open
#214 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Ruby
Stars
101
Forks
93
PR merge metrics
No merged PRs in 30d

Description

Issue with **Logstash input for Azure blob**

I have one instance of logstash for reading data from blob storage. Although logs are in the same container I have 2 major folder structure for logs from two different processes. Blob structure is something like this
Blob
- Container
- Process1/Year/Month/Day/Hour/LogFile
- Process2/Year/Month/Day/Hour/LogFile

My logstash blob config looks like this

`azureblob
{
storage_account_name => 'folder1'
storage_access_key => ''
container => 'logs'
id => 'jobs1'
blob_list_page_size => 150
file_chunk_size_bytes => 8088608
registry_create_policy => 'resume'
path_filters => 'folder1/2020 /**/*.csv'
}

azureblob
{
storage_account_name => 'folder2'
storage_access_key => ''
container => 'logs'
id => 'jobs1'
blob_list_page_size => 150
file_chunk_size_bytes => 8088608
registry_create_policy => 'resume'
path_filters => 'folder2/2020 /**/*.csv'
}`

Heap is around 3G and cpu usage is at 70-80%.

I run only one instance of logstash. Issue is logs from folder2 are processed much faster than logs from folder1. Folder2 is days ahead of folder1. ( This is catch up scenario. Am reading logs from start of this month) How do I debug this ?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.