Support user-defined aggregation and mapping functions
- Dominant language
- C++
- Stars
- 1.9k
- Forks
- 164
- Avg merge
- 7h 31m
- Merged PRs (30d)
- 1
Description
A feature which is often used in pandas is `apply` (or `aggregate` or `transform`) that basically allow to do a mapping, aggregation or even a partial reduction operation.
[PySpark](https://databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-pyspark.html) basically introduced a way to define user-defined operations for groupby's and select operations:
| function type | Operation | Input → Output | Pandas equivalent |
|---------------|-------------|-----------------------|---------------------|
| `SCALAR` | Mapping | Series → Series | `df.transform(...)` |
| `GROUPED_MAP` | Group & Map | DataFrame → DataFrame | `df.apply(...)` |
| `GROUPED_AGG` | Reduce | Series → Scalar | `df.aggregate(...)` |
I would love to see something like this in Datatable. Maybe it would be possible to have a `udf` decorator such as:
```
@udf
def my_agg(x):
...
return ...
```
Contributor guide
Assessment
This issue has not been assessed yet.