ipython / ipython/ipykernel

v6.3.0 introduced pkg_resources side effects

Aperta
#824 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
bug
Lingua principale
Python
Stelle
734
Fork
412
Merge medio
1g 5h
PR unite (30g)
8

Descrizione

`ipykernel` v6.3.0, I believe 470145ca7d103bdf6081bcfde09933ab81a56a5b specifically, introduced an indirect import on the `pkg_resources` module, via `debugpy.server.api`.

During this import, `debugpy` imports `pkg_resources` ([here](https://github.com/microsoft/debugpy/blob/b1ad4d66f1db9e2efac7d3df6a5fbb922d1435de/src/debugpy/_vendored/pydevd/pydevd_plugins/__init__.py#L5)).

The `pkg_resources` module has some [import-time side-effects](https://setuptools.pypa.io/en/latest/pkg_resources.html#workingset-objects) that cause problems in complex notebooks:
> It’s important to note that the global working_set object is initialized from sys.path **when pkg_resources is first imported**, but is only updated if you do all future sys.path manipulation via pkg_resources APIs.

I am filing this with `ipykernel` because I believe it's the place we start caring about the `sys.path` manipulations - it is somewhat common for notebooks to include `!pip install foo` steps, which impacts `sys.path` without using the `pkg_resources` APIs. However, I'd like to discuss where a fix _should_ live.

I'm part of the TensorFlow docs team, and we have an increasing number of notebooks that have a "restart runtime" requirement ([e.g.](https://colab.research.google.com/github/tensorflow/tfx/blob/master/docs/tutorials/data_validation/tfdv_basic.ipynb#scrollTo=s_NXX5GaSiZx)). I don't think fixing every notebook is the right solution, and we could apply a fix in our own infrastructure, but I would favour a fix at the root, if possible.

When we fix it in the notebook, we reload the module ([e.g.](https://colab.research.google.com/github/tensorflow/quantum/blob/master/docs/tutorials/mnist.ipynb#scrollTo=4Ql5PW-ACO0J&line=3&uniqifier=1)), however we should unload the module completely if we fix this before any notebook code, in case the notebook uses `pkg_resources`. If we were to do this, say, in `ipykernel`, would it impact the notebook kernel or other environment factors? e.g. in the use of pydevd or debugging?

In case it helps, [this notebook](https://gist.github.com/markmcd/04ee48ec7d4058c57ab6c40858cf4ddf) can be used to reproduce the failure. `ipykernel<6.3.0` works, but `ipykernel==6.3.0` (and at head) fail.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Start with commit 470145ca7d103bdf6081bcfde09933ab81a56a5b and trace the import through debugpy.server.api to debugpy/_vendored/pydevd/pydevd_plugins/__init__.py. Run the linked reproduction notebook against ipykernel<6.3.0, ipykernel==6.3.0, and current head while observing sys.path and pkg_resources. Done means the reproducer no longer fails without a notebook-specific workaround, with any impact on pydevd debugging understood.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
jupyter-notebook, python
Ambito
backend
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.