Strategies for profiling functions in ipyparallel
- Dominant language
- Jupyter Notebook
- Stars
- 2.6k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
I have a function that is going slow and I would like to figure out what is holding it up. I have profiled the underlying non-parallel function and it goes about as fast as it can on data of a size which it can handle. However, it really needs to work on blocks of data in parallel to get done in a reasonable amount of time and to work on data of any significance. So, I really need to profile it while it is running in parallel with `ipyparallel`. Are there any suggested ways to go about this? Is it possible to use some existing tool like `line_profiler`? If not, what other tools might be look at? Does `ipyparallel` have any tricks for doing this sort of thing?
Contributor guide
Research direction
The issue names ipyparallel, line_profiler, and parallel worker functions but identifies no files, tests, or entry points. Review the existing profiling and parallel-execution guidance, determine whether line_profiler or another tool supports this workflow, and document a reproducible recommendation for profiling functions while they run in parallel.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, performance
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100