agronholm / agronholm/pythonfutures

Raising an exception that is unable to be unpickled causes hang in ProcessPoolExecutor

Ouverte
#30 9 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
auto-migrated awaiting input Priority-Medium Type-Defect
Langage dominant
Python
Étoiles
236
Forks
46
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

```
What steps will reproduce the problem?
1. In the function submitted to a ProcessPoolExecutor, raise a custom exception
class that takes more than one argument to __init__.

What is the expected output? What do you see instead?
I expect a call to future.result() to not hang.

What version of the product are you using? On what operating system?
I'm using ver 2.1.6 on python 2.7 on Gentoo Linux.

Please provide any additional information below.
I have attached a patch to address the issue and a test case for it. Without
the patch, the new test case hangs. With the patch, it passes.

This is needed because of the issue raised in
http://bugs.python.org/issue1692335. An exception class that takes multiple
arguments to __init__ can be pickled but it raises a TypeError when being
unpickled:

In [1]: class MyError(Exception):
...: def __init__(self, arg1, arg2):
...: super(MyError, self).__init__(
...: 'arg1 = {}, arg2 = {}'.format(arg1, arg2))
...:

In [2]: import pickle

In [3]: p = pickle.dumps(MyError('arg1val', 'arg2val'))

In [4]: pickle.loads(p)
---------------------------------------------------------------------------

TypeError: __init__() takes exactly 3 arguments (2 given)

So if a child process raises an exception like this, it gets pickled and put in
the result queue just fine. However, in _queue_management_worker, the call to
result_queue.get(block=True) will raise an uncaught TypeError when it tries to
unpickle the exception. So then the queue management just breaks.

My proposed patch attempts to catch this condition before putting the exception
in the result queue and create a new exception that will be able to be
unpickled but still contains information from the original exception.
```

Original issue reported on code.google.com by `tbea...@gmail.com` on 30 Sep 2014 at 2:23

Attachments:
- [pickling_exception.patch](https://storage.googleapis.com/google-code-attachments/pythonfutures/issue-30/comment-0/pickling_exception.patch)

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.