h2oai / h2oai/datatable

Support user-defined aggregation and mapping functions

Open
#1,960 2 comments 0 reactions 0 assignees View on GitHub
design-doc new feature
Dominant language
C++
Stars
1.9k
Forks
164
Avg merge
7h 31m
Merged PRs (30d)
1

Description

A feature which is often used in pandas is `apply` (or `aggregate` or `transform`) that basically allow to do a mapping, aggregation or even a partial reduction operation.

[PySpark](https://databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-pyspark.html) basically introduced a way to define user-defined operations for groupby's and select operations:

| function type | Operation | Input → Output | Pandas equivalent |
|---------------|-------------|-----------------------|---------------------|
| `SCALAR` | Mapping | Series → Series | `df.transform(...)` |
| `GROUPED_MAP` | Group & Map | DataFrame → DataFrame | `df.apply(...)` |
| `GROUPED_AGG` | Reduce | Series → Scalar | `df.aggregate(...)` |

I would love to see something like this in Datatable. Maybe it would be possible to have a `udf` decorator such as:
```
@udf
def my_agg(x):
...
return ...
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.