ntop / ntop/nProbe

Issues for converting pcap files to NetFlow

Open
#438 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Lua
Stars
1.8k
Forks
51
PR merge metrics
No merged PRs in 30d

Description

I am trying to use nprobe for converting pcap files to NetFlow on Windows. The pcap files I am using can be downloaded from
UNSW-NB15 IDS dataset pcap files in which, each folder includes 27 pcap files.

I am using below command options for nprobe:

./nprobe /c -n none  -i "D:\pcaps 17-2-2015\1.pcap" -V 9  -T "%IPV4_SRC_ADDR %IPV4_DST_ADDR %PROTOCOL_MAP
 %IN_PKTS %OUT_PKTS %IN_BYTES %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT 
%L4_DST_PORT %PROTOCOL %SRC_VLAN %L7_PROTO %L7_PROTO_NAME" --dump-path "D:\Flows" --dump-format t 
--csv-separator "," --max-log-lines 200000000

It works for some of these files (all pcap files have the same number of packets (more or less 3.5x10^6), and I have tried different things to get it to work for the rest.
For instance, when nprobe did not convert a pcap to NetFlow, I doubted the pcap file formats, opened them in WireShark and exported them again to pcap format, but still it did not work.
I even divided the pcap to smaller parts and tried to convert each part separately. If one part did not converted again, I divided it to smaller parts, until I reached to a pcap file with a single packet that could not be converted to NetFlow. This is the output of nprobe when not converting a pcap to NetFlow:

Running nProbe for Windows.
22/Jul/2020 15:41:16 [nprobe.c:4421] Valid nProbe Pro license found
22/Jul/2020 15:41:16 [nprobe.c:6436] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ?
22/Jul/2020 15:41:16 [nprobe.c:6439] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ?
22/Jul/2020 15:41:16 [nprobe.c:6526] Welcome to nProbe Pro v.8.7.191214 ($Revision: 4384 $) for Windows
22/Jul/2020 15:41:16 [nprobe.c:6536] Running on Windows
22/Jul/2020 15:41:16 [nprobe.c:6547] [LICENSE] nProbe SystemId: ................
22/Jul/2020 15:41:16 [nprobe.c:6584] Dumping flow files every 10 sec into directory D:\Flows
22/Jul/2020 15:41:16 [nprobe.c:6617] Sample rate [packet: 1][flow collection/export: 1/1]
22/Jul/2020 15:41:16 [modbusPlugin.c:104] [MODBUS] Idle flow timeout set to 120 sec
22/Jul/2020 15:41:16 [nprobe.c:9404] Welcome to nProbe v.8.7.191214 for Windows
22/Jul/2020 15:41:16 [nprobe.c:8385] Using NetFlow Packet Payload Len: 1472
22/Jul/2020 15:41:16 [nprobe.c:8420] Flow export type: bidirectional flows
22/Jul/2020 15:41:16 [plugin.c:1282] 0 plugin(s) enabled
22/Jul/2020 15:41:16 [nprobe.c:8855] Each flow is 98 bytes long
22/Jul/2020 15:41:16 [nprobe.c:8856] The # flows per packet has been set to 14
22/Jul/2020 15:41:16 [nprobe.c:8859] IP TOS is ignored
22/Jul/2020 15:41:16 [nprobe.c:8887] Non IPv4/v6 traffic is discarded according to the template
22/Jul/2020 15:41:16 [nprobe.c:9684] Flows ASs will not be computed (missing libmxminddb support)
22/Jul/2020 15:41:16 [export.c:543] Using TLV as serialization format

Initially, I thought that single packet was the source of the problem and filtered it out in the original pcap files expecting it should be fine now for converting by nprobe, but again the output was the same.

Just in case, this is the output for pcap files that are successfully converted to NetFlow (the original pcaps and/or divided pcaps):

Running nProbe for Windows.
22/Jul/2020 16:05:13 [nprobe.c:4421] Valid nProbe Pro license found
22/Jul/2020 16:05:13 [nprobe.c:6436] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ?
22/Jul/2020 16:05:13 [nprobe.c:6439] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ?
22/Jul/2020 16:05:13 [nprobe.c:6526] Welcome to nProbe Pro v.8.7.191214 ($Revision: 4384 $) for Windows
22/Jul/2020 16:05:13 [nprobe.c:6536] Running on Windows
22/Jul/2020 16:05:13 [nprobe.c:6547] [LICENSE] nProbe SystemId: ...............
22/Jul/2020 16:05:13 [nprobe.c:6584] Dumping flow files every 10 sec into directory D:\Flows
22/Jul/2020 16:05:13 [nprobe.c:6617] Sample rate [packet: 1][flow collection/export: 1/1]
22/Jul/2020 16:05:13 [modbusPlugin.c:104] [MODBUS] Idle flow timeout set to 120 sec
22/Jul/2020 16:05:13 [nprobe.c:9404] Welcome to nProbe v.8.7.191214 for Windows
22/Jul/2020 16:05:13 [nprobe.c:8385] Using NetFlow Packet Payload Len: 1472
22/Jul/2020 16:05:13 [nprobe.c:8420] Flow export type: bidirectional flows
22/Jul/2020 16:05:13 [plugin.c:1282] 0 plugin(s) enabled
22/Jul/2020 16:05:13 [nprobe.c:8855] Each flow is 98 bytes long
22/Jul/2020 16:05:13 [nprobe.c:8856] The # flows per packet has been set to 14
22/Jul/2020 16:05:13 [nprobe.c:8859] IP TOS is ignored
22/Jul/2020 16:05:13 [nprobe.c:8887] Non IPv4/v6 traffic is discarded according to the template
22/Jul/2020 16:05:13 [nprobe.c:9684] Flows ASs will not be computed (missing libmxminddb support)
22/Jul/2020 16:05:13 [export.c:543] Using TLV as serialization format
22/Jul/2020 16:05:25 [nprobe.c:10042] nProbe started successfully
22/Jul/2020 16:05:25 [nprobe.c:6683] Flushing active flows
22/Jul/2020 16:05:25 [engine.c:3392] About to flush hash (threadId 0)
22/Jul/2020 16:05:25 [engine.c:3396] Completed hash walk (thread 0)
22/Jul/2020 16:05:28 [nprobe.c:3237] Processed packets: 3526992 (max bucket search: 4)
22/Jul/2020 16:05:28 [nprobe.c:3220] Fragment queue length: 6
22/Jul/2020 16:05:28 [nprobe.c:3261] Flow export stats:      [0 bytes/0 pkts][0 flows/0 pkts sent]
22/Jul/2020 16:05:28 [nprobe.c:3267] Flow export drop stats: [0 bytes/0 pkts][0 flows]
22/Jul/2020 16:05:28 [nprobe.c:3272] Total flow stats:       [0 bytes/0 pkts][0 flows/0 pkts sent]
22/Jul/2020 16:05:28 [nprobe.c:3282] Total dumped to file:   [43306 flows]

Is there any specific requirement that a pcap file should have in order to be fine for nprobe to convert it to NetFlow?
Or is there any option preventing internal errors stopping nprobe from exporting to NetFlow?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the Windows command shown with a failing pcap and a successfully converted pcap, then compare their packet contents and nProbe output. Check whether the single-packet files still reproduce the failure. Done means identifying the pcap requirement or nProbe option that prevents conversion from stopping.

Written by the indexing model from the issue text.

Assessment

Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.