apache / apache/arrow

[C++][Python] Exception ignored in: 'pyarrow._substrait._create_named_table_provider'

Open
#37,235 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Refactor Substrait.run_query() in C++/PyArrow so that it does not suppress user-defined Exceptions.

In PyArrow, Substrait can take in a `table_provider` object, which can map a table name in the Substrait plan to a table object in memory. This `table_provider` is user-defined python code, which can throw an exception. The underlying C++ implementation currently suppresses user-defined expressions and will instead report a more generic error[1]:

```
ARROW_ASSIGN_OR_RAISE(acero::Declaration source_decl,
named_table_provider(table_names, *base_schema));

if (!source_decl.IsValid()) {
return Status::Invalid("Invalid NamedTable Source");
}
```

When upgrading to Cython 3, the user-defined exception is now raised by default because cdef functions that are not extern now safely propagate Python exceptions by default. To mask raising this exception, we have to explicitly add `noexcept` to this Cython function[2]:

```
cdef CDeclaration _create_named_table_provider(
dict named_args, const std_vector[c_string]& names, const CSchema& schema
) noexcept:
```

This is the pytest output seen when suppressing the python user-defined Exception:
```
pyarrow/tests/test_substrait.py::test_named_table_invalid_table_name
/opt/homebrew/Caskroom/mambaforge/base/envs/arrow-dev/lib/python3.11/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning: Exception ignored in: 'pyarrow._substrait._create_named_table_provider'

Traceback (most recent call last):
File "/Users/dane/code/arrow/python/pyarrow/tests/test_substrait.py", line 250, in table_provider
raise Exception("Unrecognized table name")
Exception: Unrecognized table name

warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
```

[1] https://github.com/apache/arrow/blob/main/cpp/src/arrow/engine/substrait/relation_internal.cc#L422-L427
[2] https://github.com/apache/arrow/blob/main/python/pyarrow/_substrait.pyx#L29

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.