apache / apache/arrow

[Python] `extract_regex` and `extract_regex_span` only extract the first match

Open
#48,953 1 comment 0 reactions 1 assignee Claimed by @shashbha14 View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Testing out the new span extractor discussed from #44615 (and the associated PR).
Trying to replicate pandas `findall`, but it is only matching the first result. This prevents common whitespace tokenization patterns from working, entirely.

Assuming a string array `s`:

```python
# PyArrow
patt = r'\b(?(?:\#[\w\d]+)|(?:\w\/\w)|(?:\w[\w\'\d]+))\b'
arr = pc.extract_regex(s,patt)

# Pandas
token_patt = re.compile(r'\b(?:(?:\#[\w\d]+)|(?:\w\/\w)|(?:\w[\w\'\d]+))\b', flags=re.M|re.DOTALL|re.IGNORECASE)
s.to_pandas().str.findall(token_patt)
```
The latter example gets all of the tokens from a sample array of sentences, while the former only selects the first token.
More interestingly, we would like to use the span extractor (since the equivalent pandas does not exist):

```python
# PyArrow
arr = pc.extract_regex_span(s,patt)

# Pandas
spans, tokens = zip(*(
zip(*[
(m.span(), m[0].lower())
for m in token_patt.finditer(doc)
])
for doc in s.to_pylist()
))
```

Is this just a limitation of RE2? I know it doesn't support a global `\g` flag. Interestingly, google's AI says something along the lines of

> `extract_regex` (Find All Matches):
>
> This function inherently finds all matches by default (similar to a global search) and returns structs with capture groups.
> Example: `pyarrow.compute.extract_regex(strings, pattern=r"(\d+)")` extracts all sequences of digits.

If that's incorrect/hallucination, maybe we can add some explicit note in the docs that this is not global, until a solution is made?

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.