AnswerDotAI / AnswerDotAI/nbdev

nbdev_test fails when notebook imports polars package due to `set_start_method` conflict

Open
#1,185 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
5.3k
Forks
513
Avg merge
2d 30m
Merged PRs (30d)
8

Description

# Failure and repro

Trivial notebook/nbdev repo with `import polars as pl` fails ([repro CI run](https://github.com/daniel-vainsencher/nbdev-polars-mp-bug-repro/actions/runs/3329229804/jobs/5506182947)) with error message:
```
ImportError: Polars only works with python multiprocessing method set to: `mp.set_start_method("spawn")` or `mp.set_start_method("forkserver")`before importing polars. See: https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
```
and the stack confirms (lines 324-5) the method is set to `fork`.

This reproduces locally (Linux) under nbdev_test, but not when run in a jupyter-notebook, thus I am assuming the difference is nbdev_test or a dep.

# Context

[Polars explanation](https://pola-rs.github.io/polars-book/user-guide/howcani/multiprocessing.html) of their assertion seems to suggest `fork` is not a uniformly appropriate choice.

I have not found nbdev_test UI to easily override the method (which would be a reasonable workaround).

# My investigation so far

Candidate deciders of the method/callers of `set_start_method` include [nbdev_test](https://github.com/fastai/nbdev/blob/656b7bc26c8760f839a6106250f071e8e096397c/nbdev/test.py#L87) (which seems to implicitly allow 'fork'), or [fastcore](https://github.com/fastai/fastcore/blob/91cdf9930914d0043b958cbdc040ab39db29a762/fastcore/parallel.py#L19) (but CI is not on darwin) and [shell](https://github.com/fastai/execnb/blob/master/nbs/02_shell.ipynb) (but again, not on darwin).

I did not find justification for the different rules being applied to set the start method, and am not familiar enough with multiprocessing to propose a wise patch. But, as a relative newcomer to nbdev who is loving both it and polars, I thank everyone working on both and hope we can fix the conflict.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.