[Python] `extract_regex` and `extract_regex_span` only extract the first match
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Testing out the new span extractor discussed from #44615 (and the associated PR).
Trying to replicate pandas `findall`, but it is only matching the first result. This prevents common whitespace tokenization patterns from working, entirely.
Assuming a string array `s`:
```python
# PyArrow
patt = r'\b(?(?:\#[\w\d]+)|(?:\w\/\w)|(?:\w[\w\'\d]+))\b'
arr = pc.extract_regex(s,patt)
# Pandas
token_patt = re.compile(r'\b(?:(?:\#[\w\d]+)|(?:\w\/\w)|(?:\w[\w\'\d]+))\b', flags=re.M|re.DOTALL|re.IGNORECASE)
s.to_pandas().str.findall(token_patt)
```
The latter example gets all of the tokens from a sample array of sentences, while the former only selects the first token.
More interestingly, we would like to use the span extractor (since the equivalent pandas does not exist):
```python
# PyArrow
arr = pc.extract_regex_span(s,patt)
# Pandas
spans, tokens = zip(*(
zip(*[
(m.span(), m[0].lower())
for m in token_patt.finditer(doc)
])
for doc in s.to_pylist()
))
```
Is this just a limitation of RE2? I know it doesn't support a global `\g` flag. Interestingly, google's AI says something along the lines of
> `extract_regex` (Find All Matches):
>
> This function inherently finds all matches by default (similar to a global search) and returns structs with capture groups.
> Example: `pyarrow.compute.extract_regex(strings, pattern=r"(\d+)")` extracts all sequences of digits.
If that's incorrect/hallucination, maybe we can add some explicit note in the docs that this is not global, until a solution is made?
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.