new features, network aggregation

Hi all,

I would like to propose some features for nprobe that I would like to have them.
I'm using nprobe together with pfring and a SPAN port for creating flows going to ELK.
Nprobe does not populate network mask fields (%IPV4_DST_MASK), because it does not know the network that an IP belongs to. These fields are only populated when using xflow information from network devices.

So, I would like to propose nprobe the following:
Feature 1:

• nprobe to connect to a BGP peer and receive a full internet table
• nprobe to do a lookup for src and dst IP and populate fields %IPV4_DST_NETWORK, %IPV4_SRC_NETWORK.

Feature 2:

• Have the option to aggregate traffic based on "local IP" <--> "remote subnet" and send these flows to ELK and not flows of the form "local IP" <--> "remote IP". Store this info as %IPV4_SRC/DST_NETWORK and not as IP/MASK.

The case for feature 2, is that we are interested for individual IPs in our local network, in order to know individually for each server and each user the network behaviour, while for remote IPs (that don't belong to us), we are only interested for the remote subnets as a whole and not the remote IPs, per se.

In order to know our network behaviour, we are aggregating (now) this info (SRC/DST networks) from elasticsearch, which is very difficult when we don't have the "%IPV4_SRC/DST_NETWORK" as an individual field.

The AS fields (%SRC/DST_AS) are helping towards this direction, but since we are using the info to better balance our traffic across our upstream links, we need to also know about networks and not just ASes.

With feature 2, the flows (localIP<--->remotesubnet) send to elasticsearch would be less and the architecture would scale better.

Feature 3:
Have the option to filter which flows go to ELK. The filter should be able to rate limit flows of few packets.

The idea is that, the flows of >n packets per XX secs, are much more important, that the rest. For example, flows with 1packet per 60secs, could be rate limited to 100 flows/s and not compete with more important flows.

Cheers,
Spiros

No files, tests, or entry points are named. Start by reviewing nProbe's existing flow export and the handling of %IPV4_SRC/DST_NETWORK, then investigate how PF_RING input and ELK output are connected; done would require a scoped design and implementation for the requested BGP lookup, subnet aggregation, and flow filtering.