intake / intake/intake-pcap

Suggested performance improvements

Open
#6 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2
Forks
4
PR merge metrics
No merged PRs in 30d

Description

The following are suggestions to improve performance or memory usage. See #5 for benchmarks.

**Packet Parsing**

- [ ] Cython/Numba
- [ ] replace pcapy/libpcap with native Python

**Bulk Loading**

- [ ] named tuple
- [ ] class with `__slots__`
- [ ] numpy complex dtype
- [ ] pre-allocated Pandas dataframe (estimate upper bound)

I also used pyflame when profiling #5. This could help when comparing any of the above suggestions. To see a breakdown of how much time is used during parsing and loading, I've included the flamegraph when loading a 1GB file with 3M+ packets:

![pyflame-pcap](https://user-images.githubusercontent.com/6038/35663875-58421f36-06d4-11e8-9f32-e1cc4b4a3acf.png)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.