Azure / Azure/azure-diagnostics-tools
One logstash input for Azure blob is slower than other
- Dominant language
- Ruby
- Stars
- 101
- Forks
- 93
- PR merge metrics
- No merged PRs in 30d
Description
Issue with **Logstash input for Azure blob**
I have one instance of logstash for reading data from blob storage. Although logs are in the same container I have 2 major folder structure for logs from two different processes. Blob structure is something like this
Blob
- Container
- Process1/Year/Month/Day/Hour/LogFile
- Process2/Year/Month/Day/Hour/LogFile
My logstash blob config looks like this
`azureblob
{
storage_account_name => 'folder1'
storage_access_key => ''
container => 'logs'
id => 'jobs1'
blob_list_page_size => 150
file_chunk_size_bytes => 8088608
registry_create_policy => 'resume'
path_filters => 'folder1/2020 /**/*.csv'
}
azureblob
{
storage_account_name => 'folder2'
storage_access_key => ''
container => 'logs'
id => 'jobs1'
blob_list_page_size => 150
file_chunk_size_bytes => 8088608
registry_create_policy => 'resume'
path_filters => 'folder2/2020 /**/*.csv'
}`
Heap is around 3G and cpu usage is at 70-80%.
I run only one instance of logstash. Issue is logs from folder2 are processed much faster than logs from folder1. Folder2 is days ahead of folder1. ( This is catch up scenario. Am reading logs from start of this month) How do I debug this ?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.