The regex engine is about twenty times slower than DuckDB's on ClickBench q28
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- Avg merge
- 1h 31m
- Merged PRs (30d)
- 640
Description
The regex engine that landed in #826 gives the right answer and is about twenty times slower than DuckDB's on the one query that exercises it at scale.
## The measurement
ClickBench q28 at 1M rows, on `data/clickbench/1M/hits_0.parquet`, the same file and the same 596,103 non empty referers for both engines. One run each on a laptop that had other work on it, so read the ratio and not the milliseconds.
| engine | q28 at 1M |
| --- | --- |
| DuckDB 1.5.5 | 0.35 s |
| pandas, the `str.replace` pass on its own | 1.4 s |
| firepanda | 7.1 s |
The pattern is q28's, which is `^https?://(?:www\.)?([^/]+)/.*$`, compiled once for the column in every case. The answers agree exactly: two rows, four columns, the same average length to every digit, the same count, and the same digest on both text columns. So this is a cost question and not a correctness one.
Almost all of firepanda's 7.1 s is the regex pass. The rest of q28 is a group by on 596,103 rows into two groups with a mean, a count and a minimum over text, which the same port does in q27 in a fraction of that.
## Why this is worth its own issue
q28 is the only ClickBench query with a regular expression in it, so this costs one cell in one table and it would be easy to leave. Two reasons not to.
The first is that `text_hostname` exists. It is the same pattern written out by hand in Mojo, it is the fast path the URL kernel documents itself as, and it is much faster than the engine on exactly this input because it knows how long a row comes out before any byte moves and can size and copy in two parallel passes. So we have a measured upper bound on what this input can be done in, written in Mojo, in this repository. The gap between the engine and that kernel is the thing to size first, because it separates what the engine costs from what the work costs.
The second is that `.str.replace`, `.str.contains`, `.str.match` and `.str.extract` in the pandas front end all route through the same engine, and a user who reaches for any of them on a column of any size pays this. ClickBench q28 is the measurement that happens to exist, not the only thing affected.
## What is not known yet
Where the time goes has not been run down at all. Candidates worth ruling in or out before anything is changed:
- The program is stepped one byte at a time per row with no prefix check, so every row pays the full machine even when the first byte cannot start a match. `^https?` means a row not starting with `h` can be rejected on one byte.
- The output is built serially, one row appended after the last, where the hand written kernel computes every row's output length first and then fills in parallel. That is the two pass shape, and it is why the kernel can be parallel and the engine cannot.
- Captures are on for a replacement, which may cost on every row rather than on matching rows.
- Whether any of this is allocation rather than matching.
## Done when
There is a profile saying where the 7.1 s goes, a number for `text_hostname` on the same input for scale, and either a fix with a before and after on q28 at 1M or a written reason why the shape of the engine cannot get there. tamnd/firepanda-bench#39 carries the suite side and will report whatever number this ends at.
Contributor guide
Research direction
Profile ClickBench q28 at 1M rows using data/clickbench/1M/hits_0.parquet, then compare the regex pass with the text_hostname kernel on the same input. Measure where the 7.1 s goes, including matching, allocation, captures, and output construction; done means a profile, a same-input text_hostname number, and either a q28 before/after or a written reason the engine cannot reach it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100