NVIDIA / NVIDIA/cudf

[Story] Unsupported `polars.Expressions` in cudf-polars

Open
#23,151 0 comments 1 reaction 1 assignee Claimed by @mroeschke View on GitHub
cudf-polars feature request
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

In conjunction with developing an agent skill to implement unsupported `polars.Expression`s, https://github.com/rapidsai/cudf/pull/23078, the following table is the result of an accompanying agent investigation of `polars.Expressions` that are not supported in our single or multiple partition execution paths. This investigation was done with Polars expressions available as of 1.41.

Fill in the entries in the table with a link to the pull request that implements each expression:

| Polars expression (not implemented in cudf-polars) | Single-partition implementation | Multi-partition implementation |
|---|---|---|
| `append` | | |
| `bitwise_and` | | |
| `bitwise_or` | | |
| `bitwise_xor` | | |
| `bitwise_count_ones` | | Pointwise - not applicable |
| `bitwise_count_zeros` | | Pointwise - not applicable |
| `bitwise_leading_ones` | | Pointwise - not applicable |
| `bitwise_leading_zeros` | | Pointwise - not applicable |
| `bitwise_trailing_ones` | | Pointwise - not applicable |
| `bitwise_trailing_zeros` | | Pointwise - not applicable |
| `cumulative_eval` | | |
| `explode` | | |
| `flatten` | | |
| `map_batches` | | |
| `map_elements` | | Pointwise - not applicable |
| `sample` | | |
| `shuffle` | | |
| `cut` | | Pointwise - not applicable |
| `qcut` | | |
| `reshape` | | |
| `rolling_mean` | | |
| `rolling_sum` | | |
| `rolling_max` | | |
| `rolling_min` | | |
| `rolling_median` | | |
| `rolling_quantile` | | |
| `rolling_std` | | |
| `rolling_var` | | |
| `rolling_skew` | | |
| `rolling_kurtosis` | | |
| `rolling_rank` | | |
| `rolling_map` | | |
| `rolling_mean_by` | | |
| `rolling_sum_by` | | |
| `rolling_max_by` | | |
| `rolling_min_by` | | |
| `rolling_median_by` | | |
| `rolling_quantile_by` | | |
| `rolling_std_by` | | |
| `rolling_var_by` | | |
| `rolling_rank_by` | | |
| `agg_groups` | Deprecated in 1.35 https://docs.pola.rs/api/python/dev/reference/expressions/api/polars.Expr.agg_groups.html | |
| `approx_n_unique` | https://github.com/rapidsai/cudf/pull/23211 | |
| `unique_counts` | https://github.com/rapidsai/cudf/pull/23244 | |
| `arg_max` | https://github.com/rapidsai/cudf/pull/23212 | |
| `arg_min` | https://github.com/rapidsai/cudf/pull/23212 | |
| `arg_sort` | https://github.com/rapidsai/cudf/pull/23212 | |
| `arg_true` | https://github.com/rapidsai/cudf/pull/23284 | |
| `arg_unique` | https://github.com/rapidsai/cudf/pull/23212 | |
| `bottom_k_by` | https://github.com/rapidsai/cudf/pull/23213 | |
| `top_k_by` | https://github.com/rapidsai/cudf/pull/23213 | |
| `max_by` | https://github.com/rapidsai/cudf/pull/23224 | |
| `min_by` | https://github.com/rapidsai/cudf/pull/23224 | |
| `clip` | https://github.com/rapidsai/cudf/pull/23162 | Pointwise - not applicable |
| `cot` | https://github.com/rapidsai/cudf/pull/23191 | Pointwise - not applicable |
| `degrees` | https://github.com/rapidsai/cudf/pull/23210 | Pointwise - not applicable |
| `radians` | https://github.com/rapidsai/cudf/pull/23210 | Pointwise - not applicable |
| `log1p` | https://github.com/rapidsai/cudf/pull/23191 | Pointwise - not applicable |
| `sign` | https://github.com/rapidsai/cudf/pull/23250 | Pointwise - not applicable |
| `cum_count` | https://github.com/rapidsai/cudf/pull/23163 | |
| `diff` | https://github.com/rapidsai/cudf/pull/23225 | |
| `pct_change` | https://github.com/rapidsai/cudf/pull/23225 | |
| `drop_nans` | https://github.com/rapidsai/cudf/pull/23165 | |
| `entropy` | https://github.com/rapidsai/cudf/pull/23384 | |
| `skew` | https://github.com/rapidsai/cudf/pull/23353 | |
| `kurtosis` | https://github.com/rapidsai/cudf/pull/23353 | |
| `extend_constant` | https://github.com/rapidsai/cudf/pull/23167 | |
| `gather_every` | https://github.com/rapidsai/cudf/pull/23166 | |
| `hash` | https://github.com/rapidsai/cudf/pull/23313 | Pointwise - not applicable |
| `has_nulls` | https://github.com/rapidsai/cudf/pull/23168 | |
| `is_empty` | https://github.com/rapidsai/cudf/pull/23170 | |
| `is_close` | https://github.com/rapidsai/cudf/pull/23254 | Pointwise - not applicable |
| `hist` | https://github.com/rapidsai/cudf/pull/23254 | |
| `implode` | https://github.com/rapidsai/cudf/pull/23226 | |
| `item` | https://github.com/rapidsai/cudf/pull/23226 | |
| `index_of` | https://github.com/rapidsai/cudf/pull/23252 | |
| `search_sorted` | https://github.com/rapidsai/cudf/pull/23252 | |
| `interpolate` | | |
| `interpolate_by` | | |
| `mode` | https://github.com/rapidsai/cudf/pull/23226 | |
| `product` | https://github.com/rapidsai/cudf/pull/23169 | |
| `peak_max` | | |
| `peak_min` | | |
| `rechunk` | https://github.com/rapidsai/cudf/pull/23192 | Pointwise - not applicable |
| `reinterpret` | https://github.com/rapidsai/cudf/pull/23210 | Pointwise - not applicable |
| `to_physical` | https://github.com/rapidsai/cudf/pull/23313 | Pointwise - not applicable |
| `repeat_by` | https://github.com/rapidsai/cudf/pull/23227 | |
| `replace` | https://github.com/rapidsai/cudf/pull/23253 | Pointwise - not applicable |
| `replace_strict` | https://github.com/rapidsai/cudf/pull/23253 | Pointwise - not applicable |
| `reverse` | https://github.com/rapidsai/cudf/pull/23164 | |
| `rle` | | |
| `rle_id` | | |
| `round_sig_figs` | https://github.com/rapidsai/cudf/pull/23250 | Pointwise - not applicable |
| `truncate` | https://github.com/rapidsai/cudf/pull/23244 | Pointwise - not applicable |
| `rolling` (base time-indexed) | | |
| `quantile` (`interpolation="equiprobable"`) | ✅* | |
| `rank` (`method="random"`) | ✅* | |
| `over` (`mapping_strategy` ∈ {`join`,`explode`}) | ✅* | |
| `cum_sum` (`reverse=True`) | ✅* https://github.com/rapidsai/cudf/pull/23164 | |
| `cum_max` (`reverse=True`) | ✅* https://github.com/rapidsai/cudf/pull/23164 | |
| `cum_min` (`reverse=True`) | ✅* https://github.com/rapidsai/cudf/pull/23164 | |
| `cum_prod` (`reverse=True`) | ✅* https://github.com/rapidsai/cudf/pull/23164 | |
| `fill_null` (strategy / `limit=…`) | ✅* | |
| `forward_fill` | ✅ | |
| `backward_fill` | ✅ | |
| `all` | ✅ | |
| `any` | ✅ | |
| `std` | ✅ | |
| `var` | ✅ | |
| `median` | ✅ | |
| `sort` | ✅ | |
| `sort_by` | ✅ | |
| `gather` | ✅ | |
| `get` | ✅ | |
| `filter` | ✅ | |
| `where` | ✅ | |
| `shift` | ✅ | |
| `top_k` | ✅ | |
| `bottom_k` | ✅ | |
| `value_counts` | ✅ | |
| `is_duplicated` | ✅ | |
| `is_unique` | ✅ | |
| `is_first_distinct` | ✅ | |
| `is_last_distinct` | ✅ | |
| `str.extract_many` | | Pointwise - not applicable |
| `str.find_many` | | Pointwise - not applicable |
| `str.encode` | | Pointwise - not applicable |
| `str.decode` | | Pointwise - not applicable |
| `str.escape_regex` | https://github.com/rapidsai/cudf/pull/23171 | Pointwise - not applicable |
| `str.extract_all` | | Pointwise - not applicable |
| `str.normalize` | | Pointwise - not applicable |
| `str.split` | | Pointwise - not applicable |
| `str.explode` | | |
| `str.to_decimal` | https://github.com/rapidsai/cudf/pull/23243 | Pointwise - not applicable |
| `str.to_integer` | https://github.com/rapidsai/cudf/pull/23194 | Pointwise - not applicable |
| `str.to_time` | | Pointwise - not applicable |
| `str.contains` | ✅* https://github.com/rapidsai/cudf/pull/23240 | Pointwise - not applicable |
| `str.count_matches` | ✅* https://github.com/rapidsai/cudf/pull/23243 | Pointwise - not applicable |
| `str.extract` | ✅* https://github.com/rapidsai/cudf/pull/23243 | Pointwise - not applicable |
| `str.find` | ✅* https://github.com/rapidsai/cudf/pull/23240 | Pointwise - not applicable |
| `str.replace` | ✅* | Pointwise - not applicable |
| `str.replace_all` | ✅* | Pointwise - not applicable |
| `str.replace_many` | ✅* | Pointwise - not applicable |
| `str.slice` | ✅* | Pointwise - not applicable |
| `str.split_exact` | ✅* | Pointwise - not applicable |
| `str.strptime` | ✅* | Pointwise - not applicable |
| `str.to_date` | ✅* | Pointwise - not applicable |
| `str.to_datetime` | ✅* | Pointwise - not applicable |
| `str.strip_chars` | ✅* | Pointwise - not applicable |
| `str.strip_chars_start` | ✅* | Pointwise - not applicable |
| `str.strip_chars_end` | ✅* | Pointwise - not applicable |
| `str.head` | ✅* https://github.com/rapidsai/cudf/pull/23240 | Pointwise - not applicable |
| `str.tail` | ✅* https://github.com/rapidsai/cudf/pull/23240 | Pointwise - not applicable |
| `str.zfill` | ✅* https://github.com/rapidsai/cudf/pull/23193 | Pointwise - not applicable |
| `str.concat` | ✅ | |
| `str.join` | ✅ | |
| `dt.add_business_days` | | Pointwise - not applicable |
| `dt.is_business_day` | | Pointwise - not applicable |
| `dt.base_utc_offset` | | Pointwise - not applicable |
| `dt.convert_time_zone` | https://github.com/rapidsai/cudf/pull/23424 | Pointwise - not applicable |
| `dt.dst_offset` | | Pointwise - not applicable |
| `dt.replace_time_zone` | https://github.com/rapidsai/cudf/pull/23424 | Pointwise - not applicable |
| `dt.century` | https://github.com/rapidsai/cudf/pull/23153 | Pointwise - not applicable |
| `dt.millennium` | https://github.com/rapidsai/cudf/pull/23154 | Pointwise - not applicable |
| `dt.combine` | | Pointwise - not applicable |
| `dt.time` | | Pointwise - not applicable |
| `dt.date` | https://github.com/rapidsai/cudf/pull/23155 | Pointwise - not applicable |
| `dt.datetime` | Deprecated since 0.20.4 https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.dt.datetime.html | Pointwise - not applicable |
| `dt.with_time_unit` | Deprecated since 0.20.5 https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.dt.with_time_unit.html | Pointwise - not applicable |
| `dt.days_in_month` | https://github.com/rapidsai/cudf/pull/23156 | Pointwise - not applicable |
| `dt.quarter` | https://github.com/rapidsai/cudf/pull/23157 | Pointwise - not applicable |
| `dt.round` | https://github.com/rapidsai/cudf/pull/23158 | Pointwise - not applicable |
| `dt.offset_by` | | Pointwise - not applicable |
| `dt.replace` | | Pointwise - not applicable |
| `dt.total_days` | https://github.com/rapidsai/cudf/pull/18171 | Pointwise - not applicable |
| `dt.total_hours` | https://github.com/rapidsai/cudf/pull/18171 | Pointwise - not applicable |
| `dt.total_minutes` | https://github.com/rapidsai/cudf/pull/18171 | Pointwise - not applicable |
| `dt.total_seconds` | https://github.com/rapidsai/cudf/pull/18171 | Pointwise - not applicable |
| `dt.total_milliseconds` | https://github.com/rapidsai/cudf/pull/18171 | Pointwise - not applicable |
| `dt.total_microseconds` | https://github.com/rapidsai/cudf/pull/18171 | Pointwise - not applicable |
| `dt.total_nanoseconds` | https://github.com/rapidsai/cudf/pull/18171 | Pointwise - not applicable |
| `dt.truncate` (unit > day / non-`1`) | ✅* | Pointwise - not applicable |
| `list.agg` | | Pointwise - not applicable |
| `list.all` | | Pointwise - not applicable |
| `list.any` | | Pointwise - not applicable |
| `list.eval` | | Pointwise - not applicable |
| `list.filter` | | Pointwise - not applicable |
| `list.n_unique` | | Pointwise - not applicable |
| `list.reverse` | | Pointwise - not applicable |
| `list.unique` | | Pointwise - not applicable |
| `list.explode` | | |
| `list.arg_max` | | Pointwise - not applicable |
| `list.arg_min` | | Pointwise - not applicable |
| `list.concat` | | Pointwise - not applicable |
| `list.contains` | | Pointwise - not applicable |
| `list.count_matches` | | Pointwise - not applicable |
| `list.diff` | | Pointwise - not applicable |
| `list.drop_nulls` | | Pointwise - not applicable |
| `list.first` | | Pointwise - not applicable |
| `list.gather` | | Pointwise - not applicable |
| `list.gather_every` | | Pointwise - not applicable |
| `list.get` | | Pointwise - not applicable |
| `list.head` | | Pointwise - not applicable |
| `list.join` | | Pointwise - not applicable |
| `list.last` | | Pointwise - not applicable |
| `list.len` | | Pointwise - not applicable |
| `list.max` | | Pointwise - not applicable |
| `list.mean` | | Pointwise - not applicable |
| `list.median` | | Pointwise - not applicable |
| `list.min` | | Pointwise - not applicable |
| `list.sample` | | Pointwise - not applicable |
| `list.set_difference` | | Pointwise - not applicable |
| `list.set_intersection` | | Pointwise - not applicable |
| `list.set_symmetric_difference` | | Pointwise - not applicable |
| `list.set_union` | | Pointwise - not applicable |
| `list.shift` | | Pointwise - not applicable |
| `list.slice` | | Pointwise - not applicable |
| `list.sort` | | Pointwise - not applicable |
| `list.std` | | Pointwise - not applicable |
| `list.sum` | | Pointwise - not applicable |
| `list.tail` | | Pointwise - not applicable |
| `list.to_struct` | | Pointwise - not applicable |
| `list.var` | | Pointwise - not applicable |
| `list.to_array` | | Pointwise - not applicable |
| `arr.arg_max` | | Pointwise - not applicable |
| `arr.arg_min` | | Pointwise - not applicable |
| `arr.contains` | | Pointwise - not applicable |
| `arr.count_matches` | | Pointwise - not applicable |
| `arr.explode` | | |
| `arr.first` | | Pointwise - not applicable |
| `arr.get` | | Pointwise - not applicable |
| `arr.join` | | Pointwise - not applicable |
| `arr.last` | | Pointwise - not applicable |
| `arr.len` | | Pointwise - not applicable |
| `arr.max` | | Pointwise - not applicable |
| `arr.mean` | | Pointwise - not applicable |
| `arr.median` | | Pointwise - not applicable |
| `arr.min` | | Pointwise - not applicable |
| `arr.std` | | Pointwise - not applicable |
| `arr.sum` | | Pointwise - not applicable |
| `arr.var` | | Pointwise - not applicable |
| `arr.to_list` | | Pointwise - not applicable |
| `arr.to_struct` | | Pointwise - not applicable |
| `arr.agg` | | Pointwise - not applicable |
| `arr.all` | | Pointwise - not applicable |
| `arr.any` | | Pointwise - not applicable |
| `arr.eval` | | Pointwise - not applicable |
| `arr.head` | | Pointwise - not applicable |
| `arr.n_unique` | | Pointwise - not applicable |
| `arr.reverse` | | Pointwise - not applicable |
| `arr.shift` | | Pointwise - not applicable |
| `arr.slice` | | Pointwise - not applicable |
| `arr.sort` | | Pointwise - not applicable |
| `arr.tail` | | Pointwise - not applicable |
| `arr.unique` | | Pointwise - not applicable |
| `struct.with_fields` | | Pointwise - not applicable |
| `struct.unnest` | | |
| `cat.get_categories` | | |
| `cat.len_bytes` | | Pointwise - not applicable |
| `cat.len_chars` | | Pointwise - not applicable |
| `cat.slice` | | Pointwise - not applicable |
| `cat.starts_with` | | Pointwise - not applicable |
| `cat.ends_with` | | Pointwise - not applicable |
| `bin.contains` | | Pointwise - not applicable |
| `bin.starts_with` | | Pointwise - not applicable |
| `bin.ends_with` | | Pointwise - not applicable |
| `bin.get` | | Pointwise - not applicable |
| `bin.size` | | Pointwise - not applicable |
| `bin.encode` | | Pointwise - not applicable |
| `bin.reinterpret` | | Pointwise - not applicable |
| `bin.decode` | | Pointwise - not applicable |
| `bin.head` | | Pointwise - not applicable |
| `bin.tail` | | Pointwise - not applicable |
| `bin.slice` | | Pointwise - not applicable |
| `name.map_fields` | | Pointwise - not applicable |
| `pl.sum_horizontal` | https://github.com/rapidsai/cudf/pull/23427 | Pointwise - not applicable |
| `pl.min_horizontal` | https://github.com/rapidsai/cudf/pull/23427 | Pointwise - not applicable |
| `pl.max_horizontal` | https://github.com/rapidsai/cudf/pull/23160 | Pointwise - not applicable |
| `pl.mean_horizontal` | https://github.com/rapidsai/cudf/pull/23427 | Pointwise - not applicable |
| `pl.cum_sum_horizontal` | | Pointwise - not applicable |
| `pl.concat_list` | | Pointwise - not applicable |
| `pl.fold` | | Pointwise - not applicable |
| `pl.reduce` | | Pointwise - not applicable |
| `pl.cum_fold` | | Pointwise - not applicable |
| `pl.cum_reduce` | | Pointwise - not applicable |
| `pl.format` | | Pointwise - not applicable |
| `pl.coalesce` | https://github.com/rapidsai/cudf/pull/23159 | Pointwise - not applicable |
| `pl.arg_where` | https://github.com/rapidsai/cudf/pull/23284 | |
| `pl.int_range` | | |
| `pl.arange` | | |
| `pl.repeat` | https://github.com/rapidsai/cudf/pull/23227 | |
| `pl.struct` | ✅ | |

`✅*` marks expressions that already have a single-partition implementation but reject the specific parameter value(s) noted in parentheses. `pl.struct` is pointwise expression but is able to have a custom multiple partition implementation

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.