AnswerDotAI / AnswerDotAI/nbdev
nbdev_test fails when notebook imports polars package due to `set_start_method` conflict
- Dominant language
- Jupyter Notebook
- Stars
- 5.3k
- Forks
- 513
- Avg merge
- 2d 30m
- Merged PRs (30d)
- 8
Description
# Failure and repro
Trivial notebook/nbdev repo with `import polars as pl` fails ([repro CI run](https://github.com/daniel-vainsencher/nbdev-polars-mp-bug-repro/actions/runs/3329229804/jobs/5506182947)) with error message:
```
ImportError: Polars only works with python multiprocessing method set to: `mp.set_start_method("spawn")` or `mp.set_start_method("forkserver")`before importing polars. See: https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
```
and the stack confirms (lines 324-5) the method is set to `fork`.
This reproduces locally (Linux) under nbdev_test, but not when run in a jupyter-notebook, thus I am assuming the difference is nbdev_test or a dep.
# Context
[Polars explanation](https://pola-rs.github.io/polars-book/user-guide/howcani/multiprocessing.html) of their assertion seems to suggest `fork` is not a uniformly appropriate choice.
I have not found nbdev_test UI to easily override the method (which would be a reasonable workaround).
# My investigation so far
Candidate deciders of the method/callers of `set_start_method` include [nbdev_test](https://github.com/fastai/nbdev/blob/656b7bc26c8760f839a6106250f071e8e096397c/nbdev/test.py#L87) (which seems to implicitly allow 'fork'), or [fastcore](https://github.com/fastai/fastcore/blob/91cdf9930914d0043b958cbdc040ab39db29a762/fastcore/parallel.py#L19) (but CI is not on darwin) and [shell](https://github.com/fastai/execnb/blob/master/nbs/02_shell.ipynb) (but again, not on darwin).
I did not find justification for the different rules being applied to set the start method, and am not familiar enough with multiprocessing to propose a wise patch. But, as a relative newcomer to nbdev who is loving both it and polars, I thank everyone working on both and hope we can fix the conflict.
Contributor guide
Assessment
This issue has not been assessed yet.