Graylog2 / Graylog2/graylog2-server
Large lookup tables (CSV file adapter) can crash Graylog node
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## Expected Behavior
When creating a lookup adapter (example: CSV file adapter), it shouldn't exhaust the heap memory of the Graylog node, even if its backing contents don't fit into memory.
## Current Behavior
Creating a CSV file adapter backed by a large file (example: 250 MB) will ultimately exhaust the (heap) memory of the Graylog node if the file is larger than the available memory (or even smaller because the resident memory size is a multiple of the size on disk).
Refs https://community.graylog.org/t/graylog-2-4-3-crash-from-lookup-table/3942
## Possible Solution
Either issue a warning and deny creating a lookup adapter which is backed by contents not fitting into the available heap memory or, in case of the CSV file adapter, don't pre-load everything into memory from disk but use some paging mechanism instead.
## Steps to Reproduce (for bugs)
1. Start Graylog with 2 GB of heap memory
2. Create 250 MB large CSV file with lots of entries
3. Create lookup table backed by CSV file adapter configured to read the previously created file
4. ❓
5. Profit! 💸
## Your Environment
* Graylog Version: 2.4.x
* Elasticsearch Version: N/A
* MongoDB Version: N/A
Contributor guide
Assessment
This issue has not been assessed yet.