Peak RSS is 2x polars and has not moved
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- PR merge metrics
- PR metrics pending
Description
Peak RSS on TPC-H sf1, 13900K, medians of three runs:
| engine | peak RSS max | median |
|---|---|---|
| firepanda | 4.54 GB | 3.88 GB |
| duckdb | 3.28 GB | 2.58 GB |
| pandas | 3.65 GB | 2.87 GB |
| polars | 2.85 GB | 1.95 GB |
We use twice the memory polars does and have not moved on it once while the time went from 1.179 s to 1.070 s. It is now a wider gap than the time gap, which is 1.09x.
This is one cause rather than a list of them. Every operator writes its whole output before the next one reads it, so a chain of five operators over a column writes that column down five times. polars does not, and the difference is not a constant factor on the peak, it is a constant factor on the traffic too, which is why it shows up in the time as well.
q6 is the cleanest measurement of it because it is the least interesting query. A scan, three predicates on `lineitem` and a sum. We move about 470 MB. polars streams about 120 MB. There is no algorithm in q6 to be better at, so the whole 2.64x is traffic. The correlation across the suite says the same thing: the queries we lose worst are the ones with the least work per byte, and q13, which does real work per byte, is 1.06x.
So the fix is the M2b work on #79 rather than a hunt for a leak, and this issue exists to hold the number rather than to propose a different plan. What has to be true before it closes:
- [ ] peak RSS median on sf1 under 2.5 GB, which is duckdb's number and one polars beats
- [ ] q6 moving under 200 MB, measured rather than reasoned about
- [ ] the same at sf5, where the gap should be worse and is currently unmeasured because #79 line 95 has us not running sf5 on q6, q8 and q10 at all
Two things are worth ruling out before assuming it is all structural, because both would be real defects rather than design:
- [ ] the RSS growth across repeated runs in the q1 smoke test, 2.85 then 4.38 then 5.21 GB, which is not what a stateless query should do and has never been explained
- [ ] the zero filled allocations that do not need to be, at `join/pairs.mojo:668`, `hash/lasting.mojo:329`, `io/read.mojo:1670`, `array/strings.mojo:592`, `io/arrow_import.mojo:355` and `:444`, and `kernel/cast.mojo:134`. These cost touch rather than peak, but a page written is a page resident.
Split out of #79 so the number has somewhere to live that is not a comment thread.
Contributor guide
Research direction
Start by reading the M2b work on #79 and running the q6 and q1 smoke measurements on sf1; sf5 is also required once the benchmark coverage allows it. Inspect the listed allocation sites, including join/pairs.mojo:668 and hash/lasting.mojo:329, while checking repeated-run RSS growth. Done means median sf1 peak RSS is under 2.5 GB, q6 is under 200 MB, and the same targets are measured at sf5.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100