Kernel restarts halfway through training with no error message
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 13.3k
- Forks
- 5.8k
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 7
Description
I am training 2 models using StratifiedKFolds cross validation from sklearn with 10 folds. The first model trains perfectly with no errors. However when training the second model the first 2 folds make it through with no error but on the third iteration the kernel restarts with no error message like this:
```
[I 13:31:37.030 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports
[D 13:31:37.031 NotebookApp] Starting kernel: ['/home/aidan/.virtualenvs/deep3.5/bin/python3', '-m', 'ipykernel_launcher', '-f', '/run/user/1000/jupyter/kernel-6f298253-4ce8-472b-aa48-f25d50a6a9cc.json']
```
This is the code I run which causes an error when calling `model.fit`:
```python3
for i, (train_split, val_split) in enumerate(skf):
K.clear_session()
X, y, X_val, y_val = X_train[train_split], y_train[train_split], X_train[val_split], y_train[val_split]
checkpoint = ModelCheckpoint('best_2d_%d.h5'%i, monitor='val_loss', verbose=1, save_best_only=True)
early = EarlyStopping(monitor="val_loss", mode="min", patience=5)
tb = TensorBoard(log_dir='./logs/' + PREDICTION_FOLDER_2D + '/fold_%i'%i, write_graph=True)
callbacks_list = [checkpoint, early, tb]
print("#"*50)
print("Fold: ", i, "(", len(train_split), "files )")
model = get_2d_conv_model(config_2d)
print("Constructed model fitting with files:")
history = model.fit(X, y, validation_data=(X_val, y_val), callbacks=callbacks_list,
batch_size=64, epochs=config_2d.max_epochs)
```
I'm confused as to where to start given the lack of error message. Has anyone experienced anything similar or know where I can get an error message when this happens again?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the restart from the shown Python notebook loop, beginning at model.fit and the KernelRestarter log. Compare the third-fold run with the earlier folds and capture any available kernel or notebook diagnostics; done means identifying a reproducible cause or documenting the missing error path clearly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, keras, python, scikit-learn
- Domain
- devtools, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100