dask / dask/dask-benchmarks

Benchmark for graph construction

Open
#41 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
18
Forks
19
PR merge metrics
No merged PRs in 30d

Description

While exploring scheduler improvements it was recently discovered that removing redundant string concatenations has a measurable performance increase. Tom noted that there currently were no isolated benchmarks which [tested graph construction](https://github.com/dask/dask/pull/6137#issuecomment-624865967)

We could do something similar to what was suggested in https://github.com/dask/dask/pull/6137

```python
In [2]: ddf_d = timeseries(start='2000-01-01', end='2002-01-01', partition_freq='1d')

In [3]: %timeit shuffle(ddf_d, "id", shuffle="tasks")
67.2 ms ± 2.54 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

Contributor guide

Open the contributing guide

Research direction

The issue names no benchmark file; start by reviewing the repository's existing ASV benchmarks and reproduce the supplied Python example for Dask graph construction. Done means an isolated benchmark measures graph construction, including the shuffle example, so performance changes such as removing redundant string concatenations can be compared.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.