apache / apache/arrow

[C++] Substrait: add mappings for starts_with, ends_with and match_substring

Open
#50,988 0 comments 0 reactions 1 assignee Claimed by @jonasdedden View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

`starts_with`, `ends_with` and `match_substring` have no Substrait mapping, so expressions using them cannot be serialized.

```python
import pyarrow as pa
import pyarrow.compute as pc
from pyarrow.substrait import serialize_expressions

schema = pa.schema([pa.field("cat", pa.string())])
serialize_expressions([pc.starts_with(pc.field("cat"), "al")], ["f"], schema)
```

```
pyarrow.lib.ArrowNotImplementedError: No conversion function exists to convert
the Arrow function starts_with to a Substrait call
```

Same for `pc.ends_with` and `pc.match_substring`. Comparisons, `isin` and arithmetic all serialize fine, so this looks like a per-function gap rather than a structural one.

Substrait defines `starts_with`, `ends_with` and `contains` in `functions_string.yaml`, which cover all three.

The signatures differ: the Substrait functions take the pattern as a second argument, while the Arrow kernels are unary and carry it in `MatchSubstringOptions`. So the mapping needs the second argument to be a string literal. Substrait's `case_sensitivity` option maps onto `ignore_case`, except for `CASE_INSENSITIVE_ASCII` which has no Arrow equivalent.

This matters for engines that ingest a PyArrow filter through Substrait, where an unmappable function turns into a hard failure rather than a fallback.

### Component(s)

C++, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.