agronholm / agronholm/pythonfutures

Raising an exception that is unable to be unpickled causes hang in ProcessPoolExecutor

Aberta
#30 9 comentários 0 reações 0 responsáveis Ver no GitHub
auto-migrated awaiting input Priority-Medium Type-Defect
Linguagem predominante
Python
Estrelas
236
Forks
46
Métricas de merge de PRs
Nenhum PR com merge em 30d

Descrição

```
What steps will reproduce the problem?
1. In the function submitted to a ProcessPoolExecutor, raise a custom exception
class that takes more than one argument to __init__.

What is the expected output? What do you see instead?
I expect a call to future.result() to not hang.

What version of the product are you using? On what operating system?
I'm using ver 2.1.6 on python 2.7 on Gentoo Linux.

Please provide any additional information below.
I have attached a patch to address the issue and a test case for it. Without
the patch, the new test case hangs. With the patch, it passes.

This is needed because of the issue raised in
http://bugs.python.org/issue1692335. An exception class that takes multiple
arguments to __init__ can be pickled but it raises a TypeError when being
unpickled:

In [1]: class MyError(Exception):
...: def __init__(self, arg1, arg2):
...: super(MyError, self).__init__(
...: 'arg1 = {}, arg2 = {}'.format(arg1, arg2))
...:

In [2]: import pickle

In [3]: p = pickle.dumps(MyError('arg1val', 'arg2val'))

In [4]: pickle.loads(p)
---------------------------------------------------------------------------

TypeError: __init__() takes exactly 3 arguments (2 given)

So if a child process raises an exception like this, it gets pickled and put in
the result queue just fine. However, in _queue_management_worker, the call to
result_queue.get(block=True) will raise an uncaught TypeError when it tries to
unpickle the exception. So then the queue management just breaks.

My proposed patch attempts to catch this condition before putting the exception
in the result queue and create a new exception that will be able to be
unpickled but still contains information from the original exception.
```

Original issue reported on code.google.com by `tbea...@gmail.com` on 30 Sep 2014 at 2:23

Attachments:
- [pickling_exception.patch](https://storage.googleapis.com/google-code-attachments/pythonfutures/issue-30/comment-0/pickling_exception.patch)

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.