intel / intel/hyperscan

Performance Edge Case

Open
#251 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
5.5k
Forks
816
Avg merge
4d 21h
Merged PRs (30d)
2

Description

We have found an interesting edge case in HyperScan 5.0.0 and 5.3.0 performance we are trying to understand it a bit better, form better patterns, and understand if there is a "bug" in play here or not.

The initial culprit pattern was (either streaming or block is equally bad):
`/\*{5}(\w+\*){20}/Hs`

The corpus: A simple HTTP response containing `*` 32000 times (`**********...`) in the body; the repeated character is one of the keys.

---

Removing this one pattern provided a HUGE speedups (10X in a set of about 115 patterns). In attempting to troubleshoot with patbench we discovered that simpler variations are even worse:
`/\*{5}(\w+\*){20}/Hs` => 10X speedup
`/\*{5}\w/Hs` => 30X speedup

---

Using the simpler variations we found the `\w` is likely partially the culprit here:

`/\*{5}[a-zA-Z]/Hs` => not slow to start with, patbench does not pick this regex
`/\*{5}[a-zA-Z0123]/Hs` => not slow to start with
`/\*{5}[a-zA-Z01234]/Hs` => 27X slower than the above

This extends to adding non-numerics to the character class:
`/\*{5}[a-zA-Z><)(/Hs` => fast
`/\*{5}[a-zA-Z><)("/Hs` => slow

---

This was all initially tested in the context of a larger (115) pattern set, but even isolating the single pattern:
`/\*{5}[a-zA-Z01234]/Hs` against the a PCAP file with `*` 32K times => 100-150Mbps
`/\*{5}[a-zA-Z0123]/Hs` against the a PCAP file with `*` 32K times => 1724Mbps

---

Further testing results:
`/\*\*[a-zA-Z01234]/Hs` => 100-150Mbps
`/\*[a-zA-Z01234]/Hs` => 3000-5000Mbps
`/\*\*[a-zA-Z0123]/Hs` => 20000-30000Mbps

---

Are we looking at expected behavior? It seems that there's some triggering case that causes HyperScan to drastically change behavior/mode/underlying engine. If this is known/expected, can someone who understands that explain to a pattern writer how to avoid triggering such a case?

We're happy to provide more input if needed to help run down why this "break point" exists.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.