Graylog2 / Graylog2/graylog2-server
Bloom Filters for Lookups would allow the usage of threat intellicence
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## What?
We would like to do lookups in our threat intel on some of the fields in the streams. With the data-Adapters in place at the lookups are not satisfying. The usage of CPU and Memory is to high.
Data types we would like to be able to do a lookup for: IPv4 and IPv6, Domains, URLs and emailadresses, maybe filehashes.
The result is binary: it is part of the bloomfilter or not. This would lead to a certain field with a certain value to be added to the message.
## Why?
The usage of memory for the caches is way to high for a big threat intel list, and the performance is not as good as it could be. A bloomfilter would be a probabilistic, but good way to handle it. In a second step a positive hit could be checked against the original list. This second step could be done in a pipline with standard methods but drastically decreased load, or within the filter-tooling.
## Your Environment
* Graylog Version: 4.1.3
* Operating System: ubuntu 20.04
Contributor guide
Assessment
This issue has not been assessed yet.