More potential merge speedups
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
After #6931, [@gjoseph92 asked](https://github.com/dask/distributed/issues/6899#issuecomment-1223385032) "what's the next thing that pops out on the profile?".
Here are some speedscope profiles on the client (not that interesting, here for completeness), scheduler, and worker; both showing native call frames and in the `nofilter` case not filtering out any Python interpreter stack frames.
A reminder of the benchmark. I have eight workers, merging two distributed dataframes and keep the total number of rows per worker fixed but vary how many chunks it is split into. Here's the state of play
| chunks_per_worker | rows_per_chunk | before #6931 | after #6931 |
|-------------------|----------------|--------|-------|
| 100 | 50000 | 75s | 48s |
| 10 | 500000 | 9s | 9s |
| 1 | 5000000 | 8s | 8s |
The profiles record for 300 seconds, which is about 6 iterations of the merge (a bit less on the worker because profiling has about a 10% overhead there)
- [client with filter](https://www.speedscope.app/#profileURL=https%3A%2F%2Fgistcdn.githack.com%2Fwence-%2Ff67d545e5241a0e33ee3b80e54b2ee8f%2Fraw%2F2feef311a7a09ac839f258f0859245a8f68a0dc3%2Fclient-filter.json)
- [client no filter](https://www.speedscope.app/#profileURL=https%3A%2F%2Fgistcdn.githack.com%2Fwence-%2Ff67d545e5241a0e33ee3b80e54b2ee8f%2Fraw%2F2feef311a7a09ac839f258f0859245a8f68a0dc3%2Fclient-nofilter.json)
- [scheduler with filter](https://www.speedscope.app/#profileURL=https%3A%2F%2Fgistcdn.githack.com%2Fwence-%2Ff67d545e5241a0e33ee3b80e54b2ee8f%2Fraw%2F2feef311a7a09ac839f258f0859245a8f68a0dc3%2Fscheduler-filter.json)
- [scheduler no filter](https://www.speedscope.app/#profileURL=https%3A%2F%2Fgistcdn.githack.com%2Fwence-%2Ff67d545e5241a0e33ee3b80e54b2ee8f%2Fraw%2F2feef311a7a09ac839f258f0859245a8f68a0dc3%2Fscheduler-nofilter.json)
- [worker with filter](https://www.speedscope.app/#profileURL=https%3A%2F%2Fgistcdn.githack.com%2Fwence-%2Ff67d545e5241a0e33ee3b80e54b2ee8f%2Fraw%2F2feef311a7a09ac839f258f0859245a8f68a0dc3%2Fworker-filter.json)
- [worker no filter](https://www.speedscope.app/#profileURL=https%3A%2F%2Fgistcdn.githack.com%2Fwence-%2Ff67d545e5241a0e33ee3b80e54b2ee8f%2Fraw%2F2feef311a7a09ac839f258f0859245a8f68a0dc3%2Fworker-nofilter.json)
In this case, on the worker we can see a bit over 10% of the total runtime is spent in `sizeof_pandas_dataframe`. Surely this should be cheaper, so we go spelunking...
Applying
```patch
diff --git a/dask/sizeof.py b/dask/sizeof.py
index f31b0660e..a36874778 100644
--- a/dask/sizeof.py
+++ b/dask/sizeof.py
@@ -141,10 +141,12 @@ def register_pandas():
@sizeof.register(pd.DataFrame)
def sizeof_pandas_dataframe(df):
p = sizeof(df.index)
- for name, col in df.items():
- p += col.memory_usage(index=False)
- if col.dtype == object:
- p += object_size(col._values)
+ mgr = df._mgr
+ blocks = mgr.blocks
+ n = len(df)
+ for i in mgr.blknos:
+ dtype = blocks[i].dtype
+ p += n * dtype.itemsize
return int(p) + 1000
@sizeof.register(pd.Series)
```
Which I _do not_ recommend is merged:
| chunks_per_worker | rows_per_chunk | after #6931 | sizeof-opt |
|-------------------|----------------|-------------|------------|
| 100 | 50000 | 48 s | 39s |
| 10 | 500000 | 9s | 9s |
| 1 | 5000000 | 8s | 8s |
Not bad...
Here are the profiles (this time sampling for 1000s) on the worker
You'll have to download these since loading via the profileURL option of speedscope doesn't appear to work in this case (probably they're too long).
- [worker with filter sizeof-opt](https://gist.github.com/wence-/f67d545e5241a0e33ee3b80e54b2ee8f/raw/a29a36e03ef949602f264b0c9a3c06a76ff6ccd7/worker-filter-sizeof-opt.json)
- [worker no filter sizeof-opt](https://gist.github.com/wence-/f67d545e5241a0e33ee3b80e54b2ee8f/raw/a29a36e03ef949602f264b0c9a3c06a76ff6ccd7/worker-nofilter-sizeof-opt.json)
I am less confident that this is in and of itself a no-brainer improvement. That said, in the large task limit, where every bit of fighting against Amdahl helps, chipping away systematically probably will be useful.
Contributor guide
Assessment
This issue has not been assessed yet.