AnswerDotAI / AnswerDotAI/nbdev

nbdev_prepare and nbdev_test hang if I use the parallel library with loky as the backend

Open
#1,365 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Jupyter Notebook
Stars
5.3k
Forks
513
Avg merge
2d 30m
Merged PRs (30d)
8

Description

(First, thanks for nbdev. This project is great!)

# Minimally reproducible example
This code in my notebook causes `nbdev_test` to hang indefinitely:

```python
with parallel_backend("loky"):
def g(y):
return y + 1

Parallel(n_jobs=2)(delayed(g)(y) for y in [1, 2, 3, 4])
```

But specifying "threading" as the backend works:
```python
with parallel_backend("threading"):
def g(y):
return y + 1

Parallel(n_jobs=2)(delayed(g)(y) for y in [1, 2, 3, 4])
```

The other thing that allows this to work is passing --n_workers=0 to nbdev_test.
Note that "loky" is the default backend, so not specifying a backend when using parallel also fails.

This is a Mac M1
I'm on Python 3.9.18
nbdev 2.3.12
fastcore 1.5.29
Tried upgrading loky and joblib to 3.4.1 and 1.3.2 just to make sure that wasn't the issue. (It wasn't)

It's clearly related to the use of parallel in nbdev_test, but that's as far as I got:
https://github.com/fastai/nbdev/blob/4af4d479c78880f4a18af4254b119f8af8b3a8a4/nbdev/test.py#L90-L91C6

(I'm posting this here in case others are using Parallel and are stymied when nbdev_test (or nbdev_prepare) stops working.
If said people stumble across this, note that setting the parallel backend to threading makes your code _MUCH_ slower due to the Python GIL).

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the minimal example with nbdev_test and nbdev_prepare, then inspect nbdev/test.py around lines 90-91 where parallel execution is mentioned. Compare the loky, threading, and --n_workers=0 cases on the reported environment. Done means the reproducible notebook completes without hanging while preserving normal parallel execution.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter, python
Domain
testing-qa, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.