NVIDIA / NVIDIA/cudf

[Story] Eliminate fallbacks (single partition) for PDS-DS in the rapidsmpf streaming executor

Open
#21,749 0 comments 2 reactions 0 assignees View on GitHub
cudf-polars feature request rapids-mpf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

This story tracks every query that triggers fallback to single partition for PDS-DS. And missing feature support.

## Fallback Summary

### HStack / Select

| Query | Non-pointwise expression | Polars IR node | SQL equivalent |
|-------|--------------------------|----------------|----------------|
| Q12 | `.sum().over("i_class")` | `GroupedRollingWindow` | `SUM(...) OVER (PARTITION BY i_class)` |
| Q20 | `.sum().over("i_class")` | `GroupedRollingWindow` | `SUM(...) OVER (PARTITION BY i_class)` |
| Q36 | `.rank().over("partition_key")` | `GroupedRollingWindow` | `RANK() OVER (PARTITION BY ..., ORDER BY gross_margin)` |
| Q47 | `.over([...])` | `GroupedRollingWindow` | `SUM(...) OVER (PARTITION BY ...)` |
| Q49 | `.rank(method="min")` — no `.over()` | `UnaryFunction('rank')` | `RANK() OVER (ORDER BY ...)` — whole-frame, unpartitioned |
| Q51 | `.over(partition_by=..., order_by=...)` | `GroupedRollingWindow` | `SUM(...) OVER (PARTITION BY ..., ORDER BY d_date)` |
| Q53 | `.mean()` — no `.over()` | `Agg('mean', FRAME)` | `AVG(SUM(...)) OVER (PARTITION BY i_manufact_id)` — frame-level mean broadcast |
| Q57 | `.over([...])` | `GroupedRollingWindow` | `SUM(...) OVER (PARTITION BY ...)` |
| Q63 | `.mean().over("i_manager_id")` | `GroupedRollingWindow` | `AVG(SUM(...)) OVER (PARTITION BY i_manager_id)` |
| Q67 | `.over("i_category")` | `GroupedRollingWindow` | `SUM(...) OVER (PARTITION BY i_category)` |
| Q70 | `.over(["lochierarchy", "partition_key"])` | `GroupedRollingWindow` | `RANK() OVER (PARTITION BY ..., ORDER BY gross_margin)` |
| Q86 | `.over([...])` | `GroupedRollingWindow` | `RANK() OVER (PARTITION BY ..., ORDER BY gross_margin)` |
| Q89 | `.over([...])` | `GroupedRollingWindow` | `SUM(...) OVER (PARTITION BY ...)` |
| Q98 | `.sum().over("i_class")` | `GroupedRollingWindow` | `SUM(...) OVER (PARTITION BY i_class)` |

### GroupBy Aggregation Decomposition

| Query | Failing aggregation | Columns |
|-------|---------------------|---------|
| Q17 | `.std()` | `ss_quantity`, `sr_return_quantity`, `cs_quantity` |
| Q25 | `.std()` (param `agg=stddev_samp`) | `ss_net_profit`, `sr_net_loss`, `cs_net_profit` |
| Q29 | `.std()` (param `agg=stddev_samp`) | `ss_quantity`, `sr_return_quantity`, `cs_quantity` |
| Q35 | `.std()` (params `aggone=aggthree=stddev_samp`) | `cd_dep_count`, `cd_dep_employed_count`, `cd_dep_college_count` |
| Q39 | `.std()` | `inv_quantity_on_hand` (alongside the supported `.mean()`) |
| Q74 | `.std()` | `ss_net_paid` (store), `ws_net_paid` (web) |

### Conditional Join

| Query | Polars pattern | Condition | Scalar frame |
|-------|----------------|-----------|--------------|
| Q14 | `y.join(average_sales, how="cross").filter(sales > average_sales)` | `sales > average_sales` | `average_sales` — mean of all channels' quantity×price |
| Q23 | `customer_sales.join(threshold, how="cross").filter(ssales > threshold)` | `ssales > threshold` | `threshold` — 95% of max customer sales |
| Q24 | `ssales.join(threshold_table, how="cross").filter(paid > threshold)` | `paid > threshold` | `threshold_table` — 5% of mean netpaid |
| Q44 | `item_profits.join(benchmark, how="cross").filter(avg_profit > 0.9 * benchmark)` | `avg_profit > 0.9 * benchmark_profit` | `benchmark` — store avg profit for null-demo rows |

```
| Query | Status | Triggering Node | Cause |
|-------|--------|----------------|-------|
| Q01 | ✅ Pass | — | — |
| Q02 | ✅ Pass | — | — |
| Q03 | ✅ Pass | — | — |
| Q04 | ✅ Pass | — | — |
| Q05 | ✅ Pass | — | — |
| Q06 | ✅ Pass | — | — |
| Q07 | ✅ Pass | — | — |
| Q08 | ✅ Pass | — | — |
| Q09 | ✅ Pass | — | — |
| Q10 | ✅ Pass | — | — |
| Q11 | ✅ Pass | — | — |
| Q12 | ❌ Fallback | `HStack` | `.over("i_class")` — `GroupedRollingWindow` in `with_columns` |
| Q13 | ✅ Pass | — | — |
| Q14 | ❌ Fallback | `Join` | `ConditionalJoin`: cross join + `sales > average_sales` fused to ie_join |
| Q15 | ✅ Pass | — | — |
| Q16 | ❌ Fallback | `MapFunction` | `row_index` not supported for multiple partitions |
| Q17 | ❌ Fallback | `GroupBy` | `.std()` on `ss_quantity`, `sr_return_quantity`, `cs_quantity` |
| Q18 | ✅ Pass | — | — |
| Q19 | ✅ Pass | — | — |
| Q20 | ❌ Fallback | `HStack` | `.sum().over("i_class")` — `GroupedRollingWindow` in `with_columns` |
| Q21 | ✅ Pass | — | — |
| Q22 | ✅ Pass | — | — |
| Q23 | ❌ Fallback | `Join` | `ConditionalJoin`: cross join + `ssales > threshold` fused to ie_join |
| Q24 | ❌ Fallback | `Join` | `ConditionalJoin`: cross join + `paid > threshold` fused to ie_join |
| Q25 | ❌ Fallback | `GroupBy` | `.std()` on `ss_net_profit`, `sr_net_loss`, `cs_net_profit` (`agg=stddev_samp`) |
| Q26 | ✅ Pass | — | — |
| Q27 | ✅ Pass | — | — |
| Q28 | ❌ Fallback | `Select` | Non-pointwise `UnaryFunction` could not be decomposed |
| Q29 | ❌ Fallback | `GroupBy` | `.std()` on `ss_quantity`, `sr_return_quantity`, `cs_quantity` (`agg=stddev_samp`) |
| Q30 | ✅ Pass | — | — |
| Q31 | ✅ Pass | — | — |
| Q32 | ✅ Pass | — | — |
| Q33 | ✅ Pass | — | — |
| Q34 | ✅ Pass | — | — |
| Q35 | ❌ Fallback | `GroupBy` | `.std()` on `cd_dep_count`, `cd_dep_employed_count`, `cd_dep_college_count` (`aggone=aggthree=stddev_samp`) |
| Q36 | ❌ Fallback | `HStack` | `.rank().over("partition_key")` — `GroupedRollingWindow` in `with_columns` |
| Q37 | ✅ Pass | — | — |
| Q38 | ✅ Pass | — | — |
| Q39 | ❌ Fallback | `GroupBy` | `.std()` on `inv_quantity_on_hand` (alongside supported `.mean()`) |
| Q40 | ✅ Pass | — | — |
| Q41 | ✅ Pass | — | — |
| Q42 | ✅ Pass | — | — |
| Q43 | ✅ Pass | — | — |
| Q44 | ❌ Fallback | `Join` | `ConditionalJoin`: cross join + `avg_profit > 0.9 * benchmark` fused to ie_join |
| Q45 | ✅ Pass | — | — |
| Q46 | ✅ Pass | — | — |
| Q47 | ❌ Fallback | `HStack` | `.over([...])` — `GroupedRollingWindow` in `with_columns` |
| Q48 | ✅ Pass | — | — |
| Q49 | ❌ Fallback | `HStack` | `.rank(method="min")` (whole-frame, no `.over()`) — `UnaryFunction('rank')` in `with_columns` |
| Q50 | ✅ Pass | — | — |
| Q51 | ❌ Fallback | `HStack` | `.over(partition_by=..., order_by=...)` — `GroupedRollingWindow` in `with_columns` |
| Q52 | ✅ Pass | — | — |
| Q53 | ❌ Fallback | `HStack` | `.mean()` frame-level in `with_columns` (no `.over()`) — `Agg('mean', FRAME)` |
| Q54 | ✅ Pass | — | — |
| Q55 | ✅ Pass | — | — |
| Q56 | ✅ Pass | — | — |
| Q57 | ❌ Fallback | `HStack` | `.over([...])` — `GroupedRollingWindow` in `with_columns` |
| Q58 | ✅ Pass | — | — |
| Q59 | ✅ Pass | — | — |
| Q60 | ✅ Pass | — | — |
| Q61 | ✅ Pass | — | — |
| Q62 | ✅ Pass | — | — |
| Q63 | ❌ Fallback | `HStack` | `.mean().over("i_manager_id")` — `GroupedRollingWindow` in `with_columns` |
| Q64 | ✅ Pass | — | — |
| Q65 | ✅ Pass | — | — |
| Q66 | ✅ Pass | — | — |
| Q67 | ❌ Fallback | `HStack` | `.over("i_category")` — `GroupedRollingWindow` in `with_columns` |
| Q68 | ✅ Pass | — | — |
| Q69 | ✅ Pass | — | — |
| Q70 | ❌ Fallback | `HStack` | `.over(["lochierarchy", "partition_key"])` — `GroupedRollingWindow` in `with_columns` |
| Q71 | ✅ Pass | — | — |
| Q72 | ✅ Pass | — | — |
| Q73 | ✅ Pass | — | — |
| Q74 | ❌ Fallback | `GroupBy` | `.std()` on `ss_net_paid` / `ws_net_paid` (STDDEV_SAMP) |
| Q75 | ✅ Pass | — | — |
| Q76 | ✅ Pass | — | — |
| Q77 | ❌ Fallback | `Join` | Cross join (cartesian product) not supported for multiple partitions |
| Q78 | ✅ Pass | — | — |
| Q79 | ✅ Pass | — | — |
| Q80 | ✅ Pass | — | — |
| Q81 | ✅ Pass | — | — |
| Q82 | ✅ Pass | — | — |
| Q83 | ✅ Pass | — | — |
| Q84 | ✅ Pass | — | — |
| Q85 | ✅ Pass | — | — |
| Q86 | ❌ Fallback | `HStack` | `.over([...])` — `GroupedRollingWindow` in `with_columns` |
| Q87 | ✅ Pass | — | — |
| Q88 | ✅ Pass | — | — |
| Q89 | ❌ Fallback | `HStack` | `.over([...])` — `GroupedRollingWindow` in `with_columns` |
| Q90 | ✅ Pass | — | — |
| Q91 | ✅ Pass | — | — |
| Q92 | ✅ Pass | — | — |
| Q93 | ✅ Pass | — | — |
| Q94 | ✅ Pass | — | — |
| Q95 | ✅ Pass | — | — |
| Q96 | ✅ Pass | — | — |
| Q97 | ✅ Pass | — | — |
| Q98 | ❌ Fallback | `HStack` | `.sum().over("i_class")` — `GroupedRollingWindow` in `with_columns` |
| Q99 | ✅ Pass | — | — |
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.