[C++][Python] Exception ignored in: 'pyarrow._substrait._create_named_table_provider'
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Refactor Substrait.run_query() in C++/PyArrow so that it does not suppress user-defined Exceptions.
In PyArrow, Substrait can take in a `table_provider` object, which can map a table name in the Substrait plan to a table object in memory. This `table_provider` is user-defined python code, which can throw an exception. The underlying C++ implementation currently suppresses user-defined expressions and will instead report a more generic error[1]:
```
ARROW_ASSIGN_OR_RAISE(acero::Declaration source_decl,
named_table_provider(table_names, *base_schema));
if (!source_decl.IsValid()) {
return Status::Invalid("Invalid NamedTable Source");
}
```
When upgrading to Cython 3, the user-defined exception is now raised by default because cdef functions that are not extern now safely propagate Python exceptions by default. To mask raising this exception, we have to explicitly add `noexcept` to this Cython function[2]:
```
cdef CDeclaration _create_named_table_provider(
dict named_args, const std_vector[c_string]& names, const CSchema& schema
) noexcept:
```
This is the pytest output seen when suppressing the python user-defined Exception:
```
pyarrow/tests/test_substrait.py::test_named_table_invalid_table_name
/opt/homebrew/Caskroom/mambaforge/base/envs/arrow-dev/lib/python3.11/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning: Exception ignored in: 'pyarrow._substrait._create_named_table_provider'
Traceback (most recent call last):
File "/Users/dane/code/arrow/python/pyarrow/tests/test_substrait.py", line 250, in table_provider
raise Exception("Unrecognized table name")
Exception: Unrecognized table name
warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
```
[1] https://github.com/apache/arrow/blob/main/cpp/src/arrow/engine/substrait/relation_internal.cc#L422-L427
[2] https://github.com/apache/arrow/blob/main/python/pyarrow/_substrait.pyx#L29
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.