jupyter / jupyter/notebook

Trying to understand latency/slowdown of cell execution in subsequent cells.

Open
#3,224 29 comments 4 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
13.3k
Forks
5.8k
Avg merge
6d 11h
Merged PRs (30d)
7

Description

I access my jupyter notebook remotely and have consistently noticed that my cells seem to take longer and longer to execute as more cells are run. I'm trying to understand what is going on and if anything can be done to alleviate it.

As an example:
screen shot 2018-01-17 at 2 51 11 am
In the first screenshot, I run the three cells after a kernel restart and it's nice and snappy.

screen shot 2018-01-17 at 2 50 39 am
In the second screenshot, I've loaded 6 small pandas dataframes (60k rows 150ish columns), trained a small random forest regressor (10 estimators) in scikit learn, and then run the same three cells. Not only has execution time increased but the printed times are an additional 3 or 4 additional seconds apart.

As my later cells get more and more complex, this slowdown seems to increase substantially. Any idea what might be going on?

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the slowdown in a remote Jupyter notebook: compare the three-cell execution timings after a kernel restart with timings after loading six small pandas dataframes and training the small scikit-learn random forest. Investigate why execution time increases as later cells become more complex; done means identifying the cause and documenting an actionable way to alleviate it.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, pandas, scikit-learn
Domain
data, machine-learning, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.