apache / apache/arrow

[Python][Substrait] Acero consumer is unable to consume count function from substrait query plan

Open
#32,369 3 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

SQL
```java

SELECT
o_orderpriority,
count(*) AS order_count
FROM
orders
GROUP BY
o_orderpriority
```
The substrait plan generated from SQL, using Isthmus.

 

substrait count: 

 

Running the substrait plan with Acero returns this error:
```java

E   pyarrow.lib.ArrowInvalid: JsonToBinaryStream returned INVALID_ARGUMENT:(relations[0].root.input.aggregate.measures[0].measure) arguments: Cannot find field.  
```
 

From substrait query plan:

relations[0].root.input.aggregate.measures[0].measure
```java

"measure": {
"functionReference": 0,
"args": [],
"sorts": [],
"phase": "AGGREGATION_PHASE_INITIAL_TO_RESULT",
"outputType": {
"i64": {
"typeVariationReference": 0,
"nullability": "NULLABILITY_REQUIRED"
}
},
"invocation": "AGGREGATION_INVOCATION_ALL",
"arguments": []
}
```
```java

"extensions": [{
"extensionFunction": {
"extensionUriReference": 1,
"functionAnchor": 0,
"name": "count:opt"
}
}],
```
Count is a unary function and should be consumable, but isn't in this case.

**Reporter**: [Richard Tia](https://issues.apache.org/jira/browse/ARROW-17061) / @richtia
#### PRs and other links:
- [GitHub Pull Request #14123](https://github.com/apache/arrow/pull/14123)

**Note**: *This issue was originally created as [ARROW-17061](https://issues.apache.org/jira/browse/ARROW-17061). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Reproduce the SQL aggregation with the shown Substrait plan and inspect Acero's Substrait plan conversion around relations[0].root.input.aggregate.measures[0].measure. Compare the count:opt extension and its zero-argument measure with the consumer's expected representation. Done means the plan is accepted and the grouped count query executes without the Cannot find field error; PR #14123 is also relevant.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.