apache / apache/arrow

PoC for UDFs

Open
#31,095 3 comments 0 reactions 0 assignees View on GitHub
Type: task
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

The idea is to do a proof of concept before developing the UDF API or required kernels as a PR. 

This PoC will be looking into a few ideas. 
- Using Project (ProjectNodeOptions) to execute a simple Python function which serves as a UDF
- Using Aggregate (AggregateNodeOptions) to execute a simple Python function which servers as a UDF

**Reporter**: [Vibhatha Lakmal Abeykoon](https://issues.apache.org/jira/browse/ARROW-15636) / @vibhatha

**Note**: *This issue was originally created as [ARROW-15636](https://issues.apache.org/jira/browse/ARROW-15636). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by examining the proposed ProjectNodeOptions and AggregateNodeOptions approaches for executing simple Python functions as UDFs. Compare the two proof-of-concept ideas and document whether either approach is viable before an API or required kernels are developed; the PoC findings define what done looks like.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend-api-design, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.