apache / apache/arrow

GroupBy first/last throw ArrowNotImplementedError: Using ordered aggregator...not supported

Open
#45,446 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

When trying to use first/last aggregations with TableGroupBy, PyArrow throws `ArrowNotImplementedError`:

```python
In [48]: import pyarrow as pa

In [49]: tbl = pa.Table.from_pydict({"key": range(3), "val": ["foo", "bar", "baz"]})

In [50]: pa.TableGroupBy(tbl1, "key").aggregate([("val", "first")])
ArrowNotImplementedError Traceback (most recent call last)
Cell In[50], line 1
----> 1 pa.TableGroupBy(tbl, "key").aggregate([("val", "first")])

File ~/miniforge3/envs/scratchpad/lib/python3.13/site-packages/pyarrow/table.pxi:6560, in pyarrow.lib.TableGroupBy.aggregate()

File ~/miniforge3/envs/scratchpad/lib/python3.13/site-packages/pyarrow/acero.py:410, in _group_by(table, aggregates, keys, use_threads)
404 def _group_by(table, aggregates, keys, use_threads=True):
406 decl = Declaration.from_sequence([
407 Declaration("table_source", TableSourceNodeOptions(table)),
408 Declaration("aggregate", AggregateNodeOptions(aggregates, keys=keys))
409 ])
--> 410 return decl.to_table(use_threads=use_threads)

File ~/miniforge3/envs/scratchpad/lib/python3.13/site-packages/pyarrow/_acero.pyx:590, in pyarrow._acero.Declaration.to_table()

File ~/miniforge3/envs/scratchpad/lib/python3.13/site-packages/pyarrow/error.pxi:155, in pyarrow.lib.pyarrow_internal_check_status()

File ~/miniforge3/envs/scratchpad/lib/python3.13/site-packages/pyarrow/error.pxi:92, in pyarrow.lib.check_status()

ArrowNotImplementedError: Using ordered aggregator in multiple threaded execution is not supported
```

Interestingly, the first_last aggregation returns without error:

```python
In [52]: pa.TableGroupBy(tbl, "key").aggregate([("val", "first_last")])
Out[52]:
pyarrow.Table
key: int64
val_first_last: struct
child 0, first: string
child 1, last: string
----
key: [[0,1,2]]
val_first_last: [
-- is_valid: all not null
-- child 0 type: string
["foo","bar","baz"]
-- child 1 type: string
["foo","bar","baz"]]
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.