jupyter / jupyter/notebook

Kernel restarts halfway through training with no error message

Open
#3,697 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
13.3k
Forks
5.8k
Avg merge
6d 11h
Merged PRs (30d)
7

Description

I am training 2 models using StratifiedKFolds cross validation from sklearn with 10 folds. The first model trains perfectly with no errors. However when training the second model the first 2 folds make it through with no error but on the third iteration the kernel restarts with no error message like this:

```
[I 13:31:37.030 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports
[D 13:31:37.031 NotebookApp] Starting kernel: ['/home/aidan/.virtualenvs/deep3.5/bin/python3', '-m', 'ipykernel_launcher', '-f', '/run/user/1000/jupyter/kernel-6f298253-4ce8-472b-aa48-f25d50a6a9cc.json']
```
This is the code I run which causes an error when calling `model.fit`:

```python3
for i, (train_split, val_split) in enumerate(skf):
K.clear_session()
X, y, X_val, y_val = X_train[train_split], y_train[train_split], X_train[val_split], y_train[val_split]
checkpoint = ModelCheckpoint('best_2d_%d.h5'%i, monitor='val_loss', verbose=1, save_best_only=True)
early = EarlyStopping(monitor="val_loss", mode="min", patience=5)
tb = TensorBoard(log_dir='./logs/' + PREDICTION_FOLDER_2D + '/fold_%i'%i, write_graph=True)
callbacks_list = [checkpoint, early, tb]
print("#"*50)
print("Fold: ", i, "(", len(train_split), "files )")
model = get_2d_conv_model(config_2d)
print("Constructed model fitting with files:")
history = model.fit(X, y, validation_data=(X_val, y_val), callbacks=callbacks_list,
batch_size=64, epochs=config_2d.max_epochs)
```

I'm confused as to where to start given the lack of error message. Has anyone experienced anything similar or know where I can get an error message when this happens again?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the restart from the shown Python notebook loop, beginning at model.fit and the KernelRestarter log. Compare the third-fold run with the earlier folds and capture any available kernel or notebook diagnostics; done means identifying a reproducible cause or documenting the missing error path clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, keras, python, scikit-learn
Domain
devtools, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.