ipython / ipython/ipyparallel

Strategies for profiling functions in ipyparallel

Open
#87 1 comment 1 reaction 0 assignees View on GitHub
question
Dominant language
Jupyter Notebook
Stars
2.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

I have a function that is going slow and I would like to figure out what is holding it up. I have profiled the underlying non-parallel function and it goes about as fast as it can on data of a size which it can handle. However, it really needs to work on blocks of data in parallel to get done in a reasonable amount of time and to work on data of any significance. So, I really need to profile it while it is running in parallel with `ipyparallel`. Are there any suggested ways to go about this? Is it possible to use some existing tool like `line_profiler`? If not, what other tools might be look at? Does `ipyparallel` have any tricks for doing this sort of thing?

Contributor guide

Open the contributing guide

Research direction

The issue names ipyparallel, line_profiler, and parallel worker functions but identifies no files, tests, or entry points. Review the existing profiling and parallel-execution guidance, determine whether line_profiler or another tool supports this workflow, and document a reproducible recommendation for profiling functions while they run in parallel.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, performance
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.