secdev / secdev/scapy

[enhancement] Async Pcap(Ng)Reader implementations

Open
#4,503 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
12.6k
Forks
2.2k
Avg merge
1d 4h
Merged PRs (30d)
56

Description

Hi, first time opening an issue hope I do everything right 😃 , I choose not to go for the issue template as it seemed out of context for this since it's not a bug but a proposal.

Intro

As for the current state of Scapy there is no async-aware implementation for the PcapReader/PcapNgReader classes which may instead be useful when parsing many large .pcap* files using Scapy.

Solution?

I am not too familiar with the codebase but I tried to put together a horribly hacked version of possible implementation: at https://github.com/calligraf0/scapy/tree/async-pcap-reader (there is also a junkyard.py file which contains a "benchmark" which currently still lags behind the multithread sync implementation)

The main limitations I stumbled across right now are:

  • this obviously requires a python version which supports asyncio
  • lack of gzip support (the code I smashed together currently does not handle gzip compressed files, as I couldn't find a valid "drop in replacement" for gzip which supports async)
  • pcap/cap only support (I only implemented the bare minimum, so no PcapNgReader for now)
  • ugly defer of header parsing on first packet read (because of async, I couldn't use aiofiles in the __init__ method, a possible solution could be using the AsyncMixin class / rework the way the file is handled?)
  • I am not a great programmer 😅 (that has no immediate solution 😝 )

Questions

  • Is this a valid/interesting enhancement proposal? If so, is the solution I PoC'ed together a viable way (providing an extra class AsyncPcapReader instead of modifying the RawPcapReader class to also support async)?
  • I am happy to try give this a better shot, are there any pointers on what to improve/rework?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing Scapy's PcapReader and PcapNgReader implementations, then compare the async-pcap-reader branch and its junkyard.py benchmark. Before coding, clarify the supported Python versions, gzip handling, PcapNgReader scope, and header-parsing approach with maintainers. Done criteria are not defined beyond an agreed async reader design and implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.