Exceptions with non-string arguments and pickling in multi-process context
- Dominant language
- C++
- Stars
- 3.9k
- Forks
- 1k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 47
Description
### What happened?
I have this code that compiles an invalid program in subprocess. The raised exception `iree.compiler.tools.CompilerToolError` can not be correctly unpickled.
I am not sure if this is a bug in Python 3.10.9. It sure looks strange.
### Steps to reproduce your issue
```python
import iree.compiler
from concurrent.futures import ProcessPoolExecutor
def compile():
iree.compiler.compile_str(
"I'm a little teapot but not a valid program",
target_backends=iree.compiler.tools.DEFAULT_TESTING_BACKENDS)
with ProcessPoolExecutor() as executor:
executor.submit(compile).result()
```
The output is
```
concurrent.futures.process._RemoteTraceback:
'''
Traceback (most recent call last):
File "/home/petkantchin/ws/shark/shark_venv/lib/python3.10/site-packages/iree/compiler/tools/binaries.py", line 72, in __init__
errs = process.stderr.decode("utf-8")
AttributeError: 'str' object has no attribute 'stderr'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/concurrent/futures/process.py", line 387, in wait_result_broken_or_wakeup
result_item = result_reader.recv()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 251, in recv
return _ForkingPickler.loads(buf.getbuffer())
File "/home/petkantchin/ws/shark/shark_venv/lib/python3.10/site-packages/iree/compiler/tools/binaries.py", line 74, in __init__
errs = str(process.stderr) # Decode error or other: best we can do.
AttributeError: 'str' object has no attribute 'stderr'
'''
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/petkantchin/ws/shark/experiment/compile_in_subprocess.py", line 10, in
executor.submit(compile).result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
```
I was expecting to get an exception telling me that the MLIR program I am trying to compile is invalid.
During unpickling it tries to recreate the exception with the message string, but `CompilerToolError.__init__` accepts `subprocess.CompletedProcess`.
### What component(s) does this issue relate to?
Python
### Version information
Git commit e5d71f5a0b2635cfaf5153364cc699434ac92eb7.
### Additional context
Here is another self-contained example that recreates the issue.
```python
from concurrent.futures import ProcessPoolExecutor
class CustomException(Exception):
def __init__(self, x):
super().__init__(x.some_member)
class SomeClass:
def __init__(self, some_member):
self.some_member = some_member
def raise_exception():
x = SomeClass('foo')
raise CustomException(x)
with ProcessPoolExecutor() as executor:
executor.submit(raise_exception).result()
```
```
concurrent.futures.process._RemoteTraceback:
'''
Traceback (most recent call last):
File "/usr/lib/python3.10/concurrent/futures/process.py", line 387, in wait_result_broken_or_wakeup
result_item = result_reader.recv()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 251, in recv
return _ForkingPickler.loads(buf.getbuffer())
File "/home/petkantchin/ws/shark/experiment/exception_in_subprocess.py", line 5, in __init__
super().__init__(x.some_member)
AttributeError: 'str' object has no attribute 'some_member'
'''
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/petkantchin/ws/shark/experiment/exception_in_subprocess.py", line 16, in
executor.submit(raise_exception).result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
```
Contributor guide
Assessment
This issue has not been assessed yet.