Create doc page to collect benchmarks using dirtyduck data, generate initial benchmarks
@nanounanue is already working on this.
Since Mar 1, 2018.
- Dominant language
- Jupyter Notebook
- Stars
- 201
- Forks
- 62
- PR merge metrics
- No merged PRs in 30d
Description
Using the open dataset and experiment config in https://github.com/dssg/dirtyduck we should start to collect benchmarks when new features are introduced. Although collecting such benchmarks is an ongoing process that cant be covered by one unit of work, this issue tracks the creation of such a page to collect them with some examples. Some ideas for different benchmarks we could produce on master: s3 vs FS storage, single threaded vs n processes in a multicore experiment, small/medium/large grids.
Given that memory is often an issue, it could make sense to try and measure max memory usage as part of this benchmark, not just runtime. I'm not entirely sure how to do this, but if nothing else the memory/swap available on the test instance should be noted (along with other relevant instane info, of course)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.