Graylog2 / Graylog2/graylog2-server

Bloom Filters for Lookups would allow the usage of threat intellicence

Open
#11,348 0 comments 0 reactions 0 assignees View on GitHub
feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## What?

We would like to do lookups in our threat intel on some of the fields in the streams. With the data-Adapters in place at the lookups are not satisfying. The usage of CPU and Memory is to high.
Data types we would like to be able to do a lookup for: IPv4 and IPv6, Domains, URLs and emailadresses, maybe filehashes.
The result is binary: it is part of the bloomfilter or not. This would lead to a certain field with a certain value to be added to the message.

## Why?

The usage of memory for the caches is way to high for a big threat intel list, and the performance is not as good as it could be. A bloomfilter would be a probabilistic, but good way to handle it. In a second step a positive hit could be checked against the original list. This second step could be done in a pipline with standard methods but drastically decreased load, or within the filter-tooling.

## Your Environment
* Graylog Version: 4.1.3
* Operating System: ubuntu 20.04

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.