[Story] Unsupported `polars.Expressions` in cudf-polars
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
In conjunction with developing an agent skill to implement unsupported `polars.Expression`s, https://github.com/rapidsai/cudf/pull/23078, the following table is the result of an accompanying agent investigation of `polars.Expressions` that are not supported in our single or multiple partition execution paths. This investigation was done with Polars expressions available as of 1.41.
Fill in the entries in the table with a link to the pull request that implements each expression:
| Polars expression (not implemented in cudf-polars) | Single-partition implementation | Multi-partition implementation |
|---|---|---|
| `append` | | |
| `bitwise_and` | | |
| `bitwise_or` | | |
| `bitwise_xor` | | |
| `bitwise_count_ones` | | Pointwise - not applicable |
| `bitwise_count_zeros` | | Pointwise - not applicable |
| `bitwise_leading_ones` | | Pointwise - not applicable |
| `bitwise_leading_zeros` | | Pointwise - not applicable |
| `bitwise_trailing_ones` | | Pointwise - not applicable |
| `bitwise_trailing_zeros` | | Pointwise - not applicable |
| `cumulative_eval` | | |
| `explode` | | |
| `flatten` | | |
| `map_batches` | | |
| `map_elements` | | Pointwise - not applicable |
| `sample` | | |
| `shuffle` | | |
| `cut` | | Pointwise - not applicable |
| `qcut` | | |
| `reshape` | | |
| `rolling_mean` | | |
| `rolling_sum` | | |
| `rolling_max` | | |
| `rolling_min` | | |
| `rolling_median` | | |
| `rolling_quantile` | | |
| `rolling_std` | | |
| `rolling_var` | | |
| `rolling_skew` | | |
| `rolling_kurtosis` | | |
| `rolling_rank` | | |
| `rolling_map` | | |
| `rolling_mean_by` | | |
| `rolling_sum_by` | | |
| `rolling_max_by` | | |
| `rolling_min_by` | | |
| `rolling_median_by` | | |
| `rolling_quantile_by` | | |
| `rolling_std_by` | | |
| `rolling_var_by` | | |
| `rolling_rank_by` | | |
| `agg_groups` | Deprecated in 1.35 https://docs.pola.rs/api/python/dev/reference/expressions/api/polars.Expr.agg_groups.html | |
| `approx_n_unique` | https://github.com/rapidsai/cudf/pull/23211 | |
| `unique_counts` | https://github.com/rapidsai/cudf/pull/23244 | |
| `arg_max` | https://github.com/rapidsai/cudf/pull/23212 | |
| `arg_min` | https://github.com/rapidsai/cudf/pull/23212 | |
| `arg_sort` | https://github.com/rapidsai/cudf/pull/23212 | |
| `arg_true` | https://github.com/rapidsai/cudf/pull/23284 | |
| `arg_unique` | https://github.com/rapidsai/cudf/pull/23212 | |
| `bottom_k_by` | https://github.com/rapidsai/cudf/pull/23213 | |
| `top_k_by` | https://github.com/rapidsai/cudf/pull/23213 | |
| `max_by` | https://github.com/rapidsai/cudf/pull/23224 | |
| `min_by` | https://github.com/rapidsai/cudf/pull/23224 | |
| `clip` | https://github.com/rapidsai/cudf/pull/23162 | Pointwise - not applicable |
| `cot` | https://github.com/rapidsai/cudf/pull/23191 | Pointwise - not applicable |
| `degrees` | https://github.com/rapidsai/cudf/pull/23210 | Pointwise - not applicable |
| `radians` | https://github.com/rapidsai/cudf/pull/23210 | Pointwise - not applicable |
| `log1p` | https://github.com/rapidsai/cudf/pull/23191 | Pointwise - not applicable |
| `sign` | https://github.com/rapidsai/cudf/pull/23250 | Pointwise - not applicable |
| `cum_count` | https://github.com/rapidsai/cudf/pull/23163 | |
| `diff` | https://github.com/rapidsai/cudf/pull/23225 | |
| `pct_change` | https://github.com/rapidsai/cudf/pull/23225 | |
| `drop_nans` | https://github.com/rapidsai/cudf/pull/23165 | |
| `entropy` | https://github.com/rapidsai/cudf/pull/23384 | |
| `skew` | https://github.com/rapidsai/cudf/pull/23353 | |
| `kurtosis` | https://github.com/rapidsai/cudf/pull/23353 | |
| `extend_constant` | https://github.com/rapidsai/cudf/pull/23167 | |
| `gather_every` | https://github.com/rapidsai/cudf/pull/23166 | |
| `hash` | https://github.com/rapidsai/cudf/pull/23313 | Pointwise - not applicable |
| `has_nulls` | https://github.com/rapidsai/cudf/pull/23168 | |
| `is_empty` | https://github.com/rapidsai/cudf/pull/23170 | |
| `is_close` | https://github.com/rapidsai/cudf/pull/23254 | Pointwise - not applicable |
| `hist` | https://github.com/rapidsai/cudf/pull/23254 | |
| `implode` | https://github.com/rapidsai/cudf/pull/23226 | |
| `item` | https://github.com/rapidsai/cudf/pull/23226 | |
| `index_of` | https://github.com/rapidsai/cudf/pull/23252 | |
| `search_sorted` | https://github.com/rapidsai/cudf/pull/23252 | |
| `interpolate` | | |
| `interpolate_by` | | |
| `mode` | https://github.com/rapidsai/cudf/pull/23226 | |
| `product` | https://github.com/rapidsai/cudf/pull/23169 | |
| `peak_max` | | |
| `peak_min` | | |
| `rechunk` | https://github.com/rapidsai/cudf/pull/23192 | Pointwise - not applicable |
| `reinterpret` | https://github.com/rapidsai/cudf/pull/23210 | Pointwise - not applicable |
| `to_physical` | https://github.com/rapidsai/cudf/pull/23313 | Pointwise - not applicable |
| `repeat_by` | https://github.com/rapidsai/cudf/pull/23227 | |
| `replace` | https://github.com/rapidsai/cudf/pull/23253 | Pointwise - not applicable |
| `replace_strict` | https://github.com/rapidsai/cudf/pull/23253 | Pointwise - not applicable |
| `reverse` | https://github.com/rapidsai/cudf/pull/23164 | |
| `rle` | | |
| `rle_id` | | |
| `round_sig_figs` | https://github.com/rapidsai/cudf/pull/23250 | Pointwise - not applicable |
| `truncate` | https://github.com/rapidsai/cudf/pull/23244 | Pointwise - not applicable |
| `rolling` (base time-indexed) | | |
| `quantile` (`interpolation="equiprobable"`) | ✅* | |
| `rank` (`method="random"`) | ✅* | |
| `over` (`mapping_strategy` ∈ {`join`,`explode`}) | ✅* | |
| `cum_sum` (`reverse=True`) | ✅* https://github.com/rapidsai/cudf/pull/23164 | |
| `cum_max` (`reverse=True`) | ✅* https://github.com/rapidsai/cudf/pull/23164 | |
| `cum_min` (`reverse=True`) | ✅* https://github.com/rapidsai/cudf/pull/23164 | |
| `cum_prod` (`reverse=True`) | ✅* https://github.com/rapidsai/cudf/pull/23164 | |
| `fill_null` (strategy / `limit=…`) | ✅* | |
| `forward_fill` | ✅ | |
| `backward_fill` | ✅ | |
| `all` | ✅ | |
| `any` | ✅ | |
| `std` | ✅ | |
| `var` | ✅ | |
| `median` | ✅ | |
| `sort` | ✅ | |
| `sort_by` | ✅ | |
| `gather` | ✅ | |
| `get` | ✅ | |
| `filter` | ✅ | |
| `where` | ✅ | |
| `shift` | ✅ | |
| `top_k` | ✅ | |
| `bottom_k` | ✅ | |
| `value_counts` | ✅ | |
| `is_duplicated` | ✅ | |
| `is_unique` | ✅ | |
| `is_first_distinct` | ✅ | |
| `is_last_distinct` | ✅ | |
| `str.extract_many` | | Pointwise - not applicable |
| `str.find_many` | | Pointwise - not applicable |
| `str.encode` | | Pointwise - not applicable |
| `str.decode` | | Pointwise - not applicable |
| `str.escape_regex` | https://github.com/rapidsai/cudf/pull/23171 | Pointwise - not applicable |
| `str.extract_all` | | Pointwise - not applicable |
| `str.normalize` | | Pointwise - not applicable |
| `str.split` | | Pointwise - not applicable |
| `str.explode` | | |
| `str.to_decimal` | https://github.com/rapidsai/cudf/pull/23243 | Pointwise - not applicable |
| `str.to_integer` | https://github.com/rapidsai/cudf/pull/23194 | Pointwise - not applicable |
| `str.to_time` | | Pointwise - not applicable |
| `str.contains` | ✅* https://github.com/rapidsai/cudf/pull/23240 | Pointwise - not applicable |
| `str.count_matches` | ✅* https://github.com/rapidsai/cudf/pull/23243 | Pointwise - not applicable |
| `str.extract` | ✅* https://github.com/rapidsai/cudf/pull/23243 | Pointwise - not applicable |
| `str.find` | ✅* https://github.com/rapidsai/cudf/pull/23240 | Pointwise - not applicable |
| `str.replace` | ✅* | Pointwise - not applicable |
| `str.replace_all` | ✅* | Pointwise - not applicable |
| `str.replace_many` | ✅* | Pointwise - not applicable |
| `str.slice` | ✅* | Pointwise - not applicable |
| `str.split_exact` | ✅* | Pointwise - not applicable |
| `str.strptime` | ✅* | Pointwise - not applicable |
| `str.to_date` | ✅* | Pointwise - not applicable |
| `str.to_datetime` | ✅* | Pointwise - not applicable |
| `str.strip_chars` | ✅* | Pointwise - not applicable |
| `str.strip_chars_start` | ✅* | Pointwise - not applicable |
| `str.strip_chars_end` | ✅* | Pointwise - not applicable |
| `str.head` | ✅* https://github.com/rapidsai/cudf/pull/23240 | Pointwise - not applicable |
| `str.tail` | ✅* https://github.com/rapidsai/cudf/pull/23240 | Pointwise - not applicable |
| `str.zfill` | ✅* https://github.com/rapidsai/cudf/pull/23193 | Pointwise - not applicable |
| `str.concat` | ✅ | |
| `str.join` | ✅ | |
| `dt.add_business_days` | | Pointwise - not applicable |
| `dt.is_business_day` | | Pointwise - not applicable |
| `dt.base_utc_offset` | | Pointwise - not applicable |
| `dt.convert_time_zone` | https://github.com/rapidsai/cudf/pull/23424 | Pointwise - not applicable |
| `dt.dst_offset` | | Pointwise - not applicable |
| `dt.replace_time_zone` | https://github.com/rapidsai/cudf/pull/23424 | Pointwise - not applicable |
| `dt.century` | https://github.com/rapidsai/cudf/pull/23153 | Pointwise - not applicable |
| `dt.millennium` | https://github.com/rapidsai/cudf/pull/23154 | Pointwise - not applicable |
| `dt.combine` | | Pointwise - not applicable |
| `dt.time` | | Pointwise - not applicable |
| `dt.date` | https://github.com/rapidsai/cudf/pull/23155 | Pointwise - not applicable |
| `dt.datetime` | Deprecated since 0.20.4 https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.dt.datetime.html | Pointwise - not applicable |
| `dt.with_time_unit` | Deprecated since 0.20.5 https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.dt.with_time_unit.html | Pointwise - not applicable |
| `dt.days_in_month` | https://github.com/rapidsai/cudf/pull/23156 | Pointwise - not applicable |
| `dt.quarter` | https://github.com/rapidsai/cudf/pull/23157 | Pointwise - not applicable |
| `dt.round` | https://github.com/rapidsai/cudf/pull/23158 | Pointwise - not applicable |
| `dt.offset_by` | | Pointwise - not applicable |
| `dt.replace` | | Pointwise - not applicable |
| `dt.total_days` | https://github.com/rapidsai/cudf/pull/18171 | Pointwise - not applicable |
| `dt.total_hours` | https://github.com/rapidsai/cudf/pull/18171 | Pointwise - not applicable |
| `dt.total_minutes` | https://github.com/rapidsai/cudf/pull/18171 | Pointwise - not applicable |
| `dt.total_seconds` | https://github.com/rapidsai/cudf/pull/18171 | Pointwise - not applicable |
| `dt.total_milliseconds` | https://github.com/rapidsai/cudf/pull/18171 | Pointwise - not applicable |
| `dt.total_microseconds` | https://github.com/rapidsai/cudf/pull/18171 | Pointwise - not applicable |
| `dt.total_nanoseconds` | https://github.com/rapidsai/cudf/pull/18171 | Pointwise - not applicable |
| `dt.truncate` (unit > day / non-`1`) | ✅* | Pointwise - not applicable |
| `list.agg` | | Pointwise - not applicable |
| `list.all` | | Pointwise - not applicable |
| `list.any` | | Pointwise - not applicable |
| `list.eval` | | Pointwise - not applicable |
| `list.filter` | | Pointwise - not applicable |
| `list.n_unique` | | Pointwise - not applicable |
| `list.reverse` | | Pointwise - not applicable |
| `list.unique` | | Pointwise - not applicable |
| `list.explode` | | |
| `list.arg_max` | | Pointwise - not applicable |
| `list.arg_min` | | Pointwise - not applicable |
| `list.concat` | | Pointwise - not applicable |
| `list.contains` | | Pointwise - not applicable |
| `list.count_matches` | | Pointwise - not applicable |
| `list.diff` | | Pointwise - not applicable |
| `list.drop_nulls` | | Pointwise - not applicable |
| `list.first` | | Pointwise - not applicable |
| `list.gather` | | Pointwise - not applicable |
| `list.gather_every` | | Pointwise - not applicable |
| `list.get` | | Pointwise - not applicable |
| `list.head` | | Pointwise - not applicable |
| `list.join` | | Pointwise - not applicable |
| `list.last` | | Pointwise - not applicable |
| `list.len` | | Pointwise - not applicable |
| `list.max` | | Pointwise - not applicable |
| `list.mean` | | Pointwise - not applicable |
| `list.median` | | Pointwise - not applicable |
| `list.min` | | Pointwise - not applicable |
| `list.sample` | | Pointwise - not applicable |
| `list.set_difference` | | Pointwise - not applicable |
| `list.set_intersection` | | Pointwise - not applicable |
| `list.set_symmetric_difference` | | Pointwise - not applicable |
| `list.set_union` | | Pointwise - not applicable |
| `list.shift` | | Pointwise - not applicable |
| `list.slice` | | Pointwise - not applicable |
| `list.sort` | | Pointwise - not applicable |
| `list.std` | | Pointwise - not applicable |
| `list.sum` | | Pointwise - not applicable |
| `list.tail` | | Pointwise - not applicable |
| `list.to_struct` | | Pointwise - not applicable |
| `list.var` | | Pointwise - not applicable |
| `list.to_array` | | Pointwise - not applicable |
| `arr.arg_max` | | Pointwise - not applicable |
| `arr.arg_min` | | Pointwise - not applicable |
| `arr.contains` | | Pointwise - not applicable |
| `arr.count_matches` | | Pointwise - not applicable |
| `arr.explode` | | |
| `arr.first` | | Pointwise - not applicable |
| `arr.get` | | Pointwise - not applicable |
| `arr.join` | | Pointwise - not applicable |
| `arr.last` | | Pointwise - not applicable |
| `arr.len` | | Pointwise - not applicable |
| `arr.max` | | Pointwise - not applicable |
| `arr.mean` | | Pointwise - not applicable |
| `arr.median` | | Pointwise - not applicable |
| `arr.min` | | Pointwise - not applicable |
| `arr.std` | | Pointwise - not applicable |
| `arr.sum` | | Pointwise - not applicable |
| `arr.var` | | Pointwise - not applicable |
| `arr.to_list` | | Pointwise - not applicable |
| `arr.to_struct` | | Pointwise - not applicable |
| `arr.agg` | | Pointwise - not applicable |
| `arr.all` | | Pointwise - not applicable |
| `arr.any` | | Pointwise - not applicable |
| `arr.eval` | | Pointwise - not applicable |
| `arr.head` | | Pointwise - not applicable |
| `arr.n_unique` | | Pointwise - not applicable |
| `arr.reverse` | | Pointwise - not applicable |
| `arr.shift` | | Pointwise - not applicable |
| `arr.slice` | | Pointwise - not applicable |
| `arr.sort` | | Pointwise - not applicable |
| `arr.tail` | | Pointwise - not applicable |
| `arr.unique` | | Pointwise - not applicable |
| `struct.with_fields` | | Pointwise - not applicable |
| `struct.unnest` | | |
| `cat.get_categories` | | |
| `cat.len_bytes` | | Pointwise - not applicable |
| `cat.len_chars` | | Pointwise - not applicable |
| `cat.slice` | | Pointwise - not applicable |
| `cat.starts_with` | | Pointwise - not applicable |
| `cat.ends_with` | | Pointwise - not applicable |
| `bin.contains` | | Pointwise - not applicable |
| `bin.starts_with` | | Pointwise - not applicable |
| `bin.ends_with` | | Pointwise - not applicable |
| `bin.get` | | Pointwise - not applicable |
| `bin.size` | | Pointwise - not applicable |
| `bin.encode` | | Pointwise - not applicable |
| `bin.reinterpret` | | Pointwise - not applicable |
| `bin.decode` | | Pointwise - not applicable |
| `bin.head` | | Pointwise - not applicable |
| `bin.tail` | | Pointwise - not applicable |
| `bin.slice` | | Pointwise - not applicable |
| `name.map_fields` | | Pointwise - not applicable |
| `pl.sum_horizontal` | https://github.com/rapidsai/cudf/pull/23427 | Pointwise - not applicable |
| `pl.min_horizontal` | https://github.com/rapidsai/cudf/pull/23427 | Pointwise - not applicable |
| `pl.max_horizontal` | https://github.com/rapidsai/cudf/pull/23160 | Pointwise - not applicable |
| `pl.mean_horizontal` | https://github.com/rapidsai/cudf/pull/23427 | Pointwise - not applicable |
| `pl.cum_sum_horizontal` | | Pointwise - not applicable |
| `pl.concat_list` | | Pointwise - not applicable |
| `pl.fold` | | Pointwise - not applicable |
| `pl.reduce` | | Pointwise - not applicable |
| `pl.cum_fold` | | Pointwise - not applicable |
| `pl.cum_reduce` | | Pointwise - not applicable |
| `pl.format` | | Pointwise - not applicable |
| `pl.coalesce` | https://github.com/rapidsai/cudf/pull/23159 | Pointwise - not applicable |
| `pl.arg_where` | https://github.com/rapidsai/cudf/pull/23284 | |
| `pl.int_range` | | |
| `pl.arange` | | |
| `pl.repeat` | https://github.com/rapidsai/cudf/pull/23227 | |
| `pl.struct` | ✅ | |
`✅*` marks expressions that already have a single-partition implementation but reject the specific parameter value(s) noted in parentheses. `pl.struct` is pointwise expression but is able to have a custom multiple partition implementation
Contributor guide
Assessment
This issue has not been assessed yet.