dropbox / dropbox/stopwatch

Support to manually aggregate reports? (e.g. after joining parallel jobs?)

Open
#13 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
66
Forks
14
PR merge metrics
No merged PRs in 30d

Description

## Problem

I'd like to aggregate reports across multiple processes, e.g. using `multiprocessing.Pool`.
(In my case, it's a manual worker dispatch based on `mp.Process` - [see here](https://github.com/EricCousineau-TRI/repro/blob/54494a5c5154f19e693e4862fbaa79cddcd78d6f/python/process_util_example/process_util.py#L149) for an old port of the code).

## Motivating Example
In my current code, I do something like this:
```py
def worker(values):
for value in values:
out, timing = do_work(value)
yield out, timing

def main():
values = range(n) # etc.
results = parallel_work(worker, values)
outs, timings = zip(*results)
# Print aggregation of "timings" reports.
```

More concretely, here's the example code (doesn't have all deps, but it communicates the intent):
https://github.com/EricCousineau-TRI/repro/blob/54494a5c5154f19e693e4862fbaa79cddcd78d6f/drake_stuff/multibody_plant_prototypes/generate_poses_sink_clutter.py#L507-L516

## Request

Is there an easy way to aggregate results themselves using public API?

Currently, it looks like aggregation is done internally:
https://github.com/dropbox/stopwatch/blob/94f59aa2134774e06281cf0d02057bb15c1a2f2d/stopwatch.py#L303-L306

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.