AnswerDotAI / AnswerDotAI/nbdev

nbdev_test fails when notebook imports polars package due to `set_start_method` conflict

Aperta
#1,185 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Jupyter Notebook
Stelle
5.3k
Fork
513
Merge medio
2g 30m
PR unite (30g)
8

Descrizione

# Failure and repro

Trivial notebook/nbdev repo with `import polars as pl` fails ([repro CI run](https://github.com/daniel-vainsencher/nbdev-polars-mp-bug-repro/actions/runs/3329229804/jobs/5506182947)) with error message:
```
ImportError: Polars only works with python multiprocessing method set to: `mp.set_start_method("spawn")` or `mp.set_start_method("forkserver")`before importing polars. See: https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
```
and the stack confirms (lines 324-5) the method is set to `fork`.

This reproduces locally (Linux) under nbdev_test, but not when run in a jupyter-notebook, thus I am assuming the difference is nbdev_test or a dep.

# Context

[Polars explanation](https://pola-rs.github.io/polars-book/user-guide/howcani/multiprocessing.html) of their assertion seems to suggest `fork` is not a uniformly appropriate choice.

I have not found nbdev_test UI to easily override the method (which would be a reasonable workaround).

# My investigation so far

Candidate deciders of the method/callers of `set_start_method` include [nbdev_test](https://github.com/fastai/nbdev/blob/656b7bc26c8760f839a6106250f071e8e096397c/nbdev/test.py#L87) (which seems to implicitly allow 'fork'), or [fastcore](https://github.com/fastai/fastcore/blob/91cdf9930914d0043b958cbdc040ab39db29a762/fastcore/parallel.py#L19) (but CI is not on darwin) and [shell](https://github.com/fastai/execnb/blob/master/nbs/02_shell.ipynb) (but again, not on darwin).

I did not find justification for the different rules being applied to set the start method, and am not familiar enough with multiprocessing to propose a wise patch. But, as a relative newcomer to nbdev who is loving both it and polars, I thank everyone working on both and hope we can fix the conflict.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia da nbdev/test.py alla riga 87 e dall’esecuzione CI di repro collegata, quindi segui la configurazione di multiprocessing attraverso fastcore/parallel.py alla riga 19 e il notebook shell di execnb. Conferma quale componente seleziona fork e prova un percorso nbdev_test compatibile con il notebook polars. Il lavoro è completato quando repro passa con nbdev_test senza interrompere il comportamento esistente di multiprocessing.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
jupyter-notebook, python
Ambito
backend, testing-qa
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
30/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.