apache / apache/datafusion

Set-returning UDFs

Open
#1,604 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I'm looking at add an inverted index on my data and use it as the first step in more complicated data transformations. There's particular optimization that I think might be better done outside of the DataFusion engine.

**Describe the solution you'd like**
I'm imagining something like the following, where `inverted_index_search` returns an arbitrary number of rows.
```
SELECT inverted_index_search('some query');
```
```
+------+
| id |
+------+
| 1 |
| 2 |
+------+
```

I'd like to be able to write a udf (or udaf, or something new) that can return as many rows as it needs.

**Describe alternatives you've considered**
I looked at having my UDF return an array, but once I had the array I could not figure out how to unnest it. Issue https://github.com/apache/arrow-datafusion/issues/212 looks to cover that, but I think it might be better to just have the function return multiple rows directly.

**Additional context**
Apologies if I missed something that is already available for this.

Contributor guide

Open the contributing guide

Research direction

The issue names no implementation files or tests. Start by reviewing DataFusion's existing UDF and UDAF behavior, then read issue #212 about unnesting arrays. Done means a function such as inverted_index_search can return an arbitrary number of rows directly and the SQL example produces the requested result.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.