Azure / Azure/azure-diagnostics-tools
[logstash-input-azurewadtable] Allow to limit the processed event size
- Dominant language
- Ruby
- Stars
- 101
- Forks
- 93
- PR merge metrics
- No merged PRs in 30d
Description
Additional to `entity_count_to_process`, it would be useful to be able to specify the maximum size allowed of events processed by the input. The reason for this is that by default, the input requests 100 items per iteration. Now, in case there is an issue with the data in the input table, those items could be quite big (think GiBs). This would crash the whole Logstash instance because plugins are not isolated in Logstash. Our use case only involves small items (~KiB).
I am opening this feature request because he had a crash of one out of three Logstash instances and this instance was the only one running the logstash-input-azurewadtable input. We can not check the table in question because the entries have already been cleaned up automatically. It is also not clear if this plugin caused the crash. The idea with limiting the size is from Elastic support with whom this case has been discussed in detail.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the logstash-input-azurewadtable input and its existing entity_count_to_process handling. Trace how batches are requested and events are processed, then verify that a configured maximum event size prevents oversized items from being processed without affecting normal small items.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ruby
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100