matrixorigin / matrixorigin/matrixone
[Subtask]: UDAF and UDTF
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
### Parent Issue
#6581
### Detail of Subtask
User defined AGG function and user defined TABLE function are significantly different from scalar UDF. See #9778
UDAF and UDTF also need infrastructure support #96
### Describe implementation you've considered
Postgres started this stuff, see PostgreSQL for user defined agg and table function syntax. However it is rather clumsy.
https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-tabular-functions
Snowflake has UDTF, but not UDAF. UDAF can be implemented using UDTF, but very clumsy. Note that Snowflake UDTF can use WINDOW function syntax, over (...) partition by order by ...
Also note that snowflake does not implement/has api for merge/finalize, which pretty much limited the function for EVERY PARTITION defined in window spec must run in one CN, in one context (most importantly, one chunk of memory). So snowflake can parallelize UDTF across partitions of data but not within. BSP style parallel programming is also not possible.
But we can accept these limitations, probably good enough.
Note that UDTF can also be implemented in UDAF. Just wrap input rows and result rows in json envelope and we can use flatten to unnest json into tables/tuples. This could be less efficient than impl UDAF and UDTF separately for best performance but I think this is so much easier.
I suggest we impl UDAF over window spec.
### Additional information
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.