matrixorigin / matrixorigin/matrixone

[Subtask]: UDAF and UDTF

Open
#9,779 1 comment 0 reactions 1 assignee Claimed by @iamlinjunhong View on GitHub
attention/dev-design-required kind/subtask priority/p0 source/dev team/c2
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

### Parent Issue

#6581

### Detail of Subtask

User defined AGG function and user defined TABLE function are significantly different from scalar UDF. See #9778

UDAF and UDTF also need infrastructure support #96

### Describe implementation you've considered

Postgres started this stuff, see PostgreSQL for user defined agg and table function syntax. However it is rather clumsy.

https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-tabular-functions

Snowflake has UDTF, but not UDAF. UDAF can be implemented using UDTF, but very clumsy. Note that Snowflake UDTF can use WINDOW function syntax, over (...) partition by order by ...

Also note that snowflake does not implement/has api for merge/finalize, which pretty much limited the function for EVERY PARTITION defined in window spec must run in one CN, in one context (most importantly, one chunk of memory). So snowflake can parallelize UDTF across partitions of data but not within. BSP style parallel programming is also not possible.
But we can accept these limitations, probably good enough.

Note that UDTF can also be implemented in UDAF. Just wrap input rows and result rows in json envelope and we can use flatten to unnest json into tables/tuples. This could be less efficient than impl UDAF and UDTF separately for best performance but I think this is so much easier.

I suggest we impl UDAF over window spec.

### Additional information

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.