ipython / ipython/ipykernel

Make kernel and shell cleanup easier.

Aperta
#221 3 commenti 2 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
734
Fork
412
Merge medio
1g 5h
PR unite (30g)
8

Descrizione

[Related: #219]

This is a feature request, aimed at benefiting applications that:

1. embed an IPython kernel, and
2. have good unit tests.

We have such an application, and we're finding it painful to reset the main Python process to a sensible state after running a unit test that creates, uses, and then destroys an IPython kernel (specifically, an `InternalIPKernel`). The main issues we're experiencing are thread leaks and file descriptor leaks; currently our test suite leaks 22 file descriptors (related to the ZMQ sockets being used) per IPython-using unit test; which leads to the OS running out of file handles early on in the test suite. More details below.

It would be very useful to have a reasonably simple, documented way to release OS resources that have been acquired by a kernel / shell / frontend client.

I guess this is rather a niche use-case (I'd hope that anyone doing (1) above is also doing (2), but that's probably a bit naïve :-), so I understand if it's not something that the IPython devs want to pursue, but it seemed worth at least registering that some of us have encountered this need, and documenting some of the specific issues encountered. (Also, I can't help wondering how IPython's own test suites avoid running into these sort of issues.)

For reference, here are some of the specific issues that we're running into:

1. After tests that create a kernel, shell and qtconsole client, we end up with a singleton `ZMQInteractiveShell` object. That object has a `configurables` attribute (inherited from `IPython.core.InteractiveShell`) that's often appended to, but AFAICT never has items removed from it. In particular, that list includes methods related to each front end created over the course of the test suite, and so keeps all those front end objects and all their associated resources alive.

2. The ZMQ sockets rely on reference-counting (via a `__del__` method) for freeing the underlying Unix sockets; most don't seem to be closed explicitly at any point. So when objects are being kept alive, those sockets are kept alive, too.

3. PySide plays particularly badly with reference-count based cleanup, since PySide habitually creates objects with artificially inflated reference counts, which then become uncollectable. This is biting us particularly with the qtconsole. This is very much a PySide bug (and we'll probably eventually migrate to PyQt precisely to avoid this sort of problem), but having a way to do explicit cleanup within IPython would at least allow us to work around that bug.

4. Under Python 2 (which unfortunately is what we're stuck on for the forseeable future for this app), some objects are truly uncollectable, ending up in `gc.garbage`. The `ScriptMagics` object seems to be one such: we end up with a cycle containing those objects, and the fact that they have a `__del__` method means that they never get collected (and also that the cleanup that the `__del__` method is supposed to perform never happens). And again, that uncollectable garbage is keeping the `ZMQInteractiveShell` object alive.

5. So the upshot of the above is that the `ZMQInteractiveShell` object keeps many Python objects and OS resources alive. If we want to fix that, our options at this point are: (1) try to clean up that shell, or (2) try to remove all references to the shell. (1) seems very involved, and I guess is the point of this issue: it would be great to have some kind of one-line method call that cleans everything up. I tried to go with (2), but that's also painful: there are direct and indirect references to the shell in many places: various singletons, module-level variables, atexit handlers, sys-module attributes. To give some idea, here's a snippet from our current `tearDown` code:

```
# Reset IPython global instances
IPKernelApp.clear_instance()
IPythonKernel.clear_instance()
ZMQInteractiveShell.clear_instance()
InlineBackend.clear_instance()
IOLoop.clear_instance()

del IPython.utils.io.stdout.session
del IPython.utils.io.stderr.session
del IPython.utils.io.stdout.stream
del IPython.utils.io.stderr.stream

del IPython.utils.io.stdout
del IPython.utils.io.stderr

# Reset global zmq context (calling destroy does not do it !
zmq.Context._instance = None

# Reset exception hook. (Is there something in the IPython machinery
# that does this?)
sys.displayhook = sys.__displayhook__
sys.excepthook = sys.__excepthook__

import matplotlib.backends.qt_editor.formlayout

del matplotlib.backends.qt_editor.formlayout.STDERR.session

# Remove atexit handlers.
import atexit
while atexit._exithandlers:
handler = atexit._exithandlers.pop()
del handler
```

Any help with doing this sort of cleanup (even if it's just documentation on how to do it properly, or pointers to other places where it's done) would be very valuable.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.