[enhancement] Async Pcap(Ng)Reader implementations
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 12.6k
- Forks
- 2.2k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 56
Description
Hi, first time opening an issue hope I do everything right 😃 , I choose not to go for the issue template as it seemed out of context for this since it's not a bug but a proposal.
Intro
As for the current state of Scapy there is no async-aware implementation for the PcapReader/PcapNgReader classes which may instead be useful when parsing many large .pcap* files using Scapy.
Solution?
I am not too familiar with the codebase but I tried to put together a horribly hacked version of possible implementation: at https://github.com/calligraf0/scapy/tree/async-pcap-reader (there is also a junkyard.py file which contains a "benchmark" which currently still lags behind the multithread sync implementation)
The main limitations I stumbled across right now are:
- this obviously requires a python version which supports asyncio
- lack of gzip support (the code I smashed together currently does not handle gzip compressed files, as I couldn't find a valid "drop in replacement" for gzip which supports async)
- pcap/cap only support (I only implemented the bare minimum, so no PcapNgReader for now)
- ugly defer of header parsing on first packet read (because of async, I couldn't use aiofiles in the
__init__method, a possible solution could be using the AsyncMixin class / rework the way the file is handled?) - I am not a great programmer 😅 (that has no immediate solution 😝 )
Questions
- Is this a valid/interesting enhancement proposal? If so, is the solution I PoC'ed together a viable way (providing an extra class AsyncPcapReader instead of modifying the RawPcapReader class to also support async)?
- I am happy to try give this a better shot, are there any pointers on what to improve/rework?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing Scapy's PcapReader and PcapNgReader implementations, then compare the async-pcap-reader branch and its junkyard.py benchmark. Before coding, clarify the supported Python versions, gzip handling, PcapNgReader scope, and header-parsing approach with maintainers. Done criteria are not defined beyond an agreed async reader design and implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100