Azure / Azure/azure-diagnostics-tools

[logstash-input-azurewadtable] Allow to limit the processed event size

Open
#169 1 comment 0 reactions 0 assignees View on GitHub
logstash-input-azuretable
Dominant language
Ruby
Stars
101
Forks
93
PR merge metrics
No merged PRs in 30d

Description

Additional to `entity_count_to_process`, it would be useful to be able to specify the maximum size allowed of events processed by the input. The reason for this is that by default, the input requests 100 items per iteration. Now, in case there is an issue with the data in the input table, those items could be quite big (think GiBs). This would crash the whole Logstash instance because plugins are not isolated in Logstash. Our use case only involves small items (~KiB).

I am opening this feature request because he had a crash of one out of three Logstash instances and this instance was the only one running the logstash-input-azurewadtable input. We can not check the table in question because the entries have already been cleaned up automatically. It is also not clear if this plugin caused the crash. The idea with limiting the size is from Elastic support with whom this case has been discussed in detail.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the logstash-input-azurewadtable input and its existing entity_count_to_process handling. Trace how batches are requested and events are processed, then verify that a configured maximum event size prevents oversized items from being processed without affecting normal small items.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.