Trying to understand latency/slowdown of cell execution in subsequent cells.
- Dominant language
- Jupyter Notebook
- Stars
- 13.3k
- Forks
- 5.8k
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 7
Description
I access my jupyter notebook remotely and have consistently noticed that my cells seem to take longer and longer to execute as more cells are run. I'm trying to understand what is going on and if anything can be done to alleviate it.
As an example:
In the first screenshot, I run the three cells after a kernel restart and it's nice and snappy.
In the second screenshot, I've loaded 6 small pandas dataframes (60k rows 150ish columns), trained a small random forest regressor (10 estimators) in scikit learn, and then run the same three cells. Not only has execution time increased but the printed times are an additional 3 or 4 additional seconds apart.
As my later cells get more and more complex, this slowdown seems to increase substantially. Any idea what might be going on?
Contributor guide
Research direction
Start by reproducing the slowdown in a remote Jupyter notebook: compare the three-cell execution timings after a kernel restart with timings after loading six small pandas dataframes and training the small scikit-learn random forest. Investigate why execution time increases as later cells become more complex; done means identifying the cause and documenting an actionable way to alleviate it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, pandas, scikit-learn
- Domain
- data, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100