TimelyDataflow / TimelyDataflow/differential-dataflow
Profiling CPU usage of differential programs
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3k
- Forks
- 211
- Avg merge
- 10h 42m
- Merged PRs (30d)
- 34
Description
I would like to understand where my differential program spends time. Is my best bet to just use a generic CPU profiler or is there some sort of API for this in timely/differential? Using a generic profiler does not seem very promising, as the particular program I am looking at has thousands of operators, so it will be hard to correlate call stacks with specific operators.
In the past, the tracing API proved immensely useful in optimizing the memory footprint of my programs (#113). I was hoping to use something similar for CPU profiling.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the tracing API discussed in issue #113 and the differential-dataflow profiling needs described here. Compare that approach with generic CPU profiling, then seek maintainer agreement on the API or instrumentation scope and define completion criteria before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100