Graylog2 / Graylog2/graylog2-server

Large lookup tables (CSV file adapter) can crash Graylog node

Open
#4,532 4 comments 0 reactions 0 assignees View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior

When creating a lookup adapter (example: CSV file adapter), it shouldn't exhaust the heap memory of the Graylog node, even if its backing contents don't fit into memory.

## Current Behavior

Creating a CSV file adapter backed by a large file (example: 250 MB) will ultimately exhaust the (heap) memory of the Graylog node if the file is larger than the available memory (or even smaller because the resident memory size is a multiple of the size on disk).

Refs https://community.graylog.org/t/graylog-2-4-3-crash-from-lookup-table/3942

## Possible Solution

Either issue a warning and deny creating a lookup adapter which is backed by contents not fitting into the available heap memory or, in case of the CSV file adapter, don't pre-load everything into memory from disk but use some paging mechanism instead.

## Steps to Reproduce (for bugs)

1. Start Graylog with 2 GB of heap memory
2. Create 250 MB large CSV file with lots of entries
3. Create lookup table backed by CSV file adapter configured to read the previously created file
4. ❓
5. Profit! 💸

## Your Environment

* Graylog Version: 2.4.x
* Elasticsearch Version: N/A
* MongoDB Version: N/A

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.