apache / apache/datafusion

Support Custom Function Registration with Catalog and Schema

Open
#15,363 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 11h
Merged PRs (30d)
362

Description

### Is your feature request related to a problem or challenge?

In my case, I would like to extend a default function's supporting signature. For example, the `avg` function in DataFusion doesn't support calculating the Duration average.
However, suppose I want to register a custom `avg` for Duration. In that case, it means that I need to re-implement the whole `avg` function because the new implementation will override the previous one when registered. I want to propose an approach to register the custom implementation and keep the original implementation at the same time.

### Describe the solution you'd like

In Postgres or DuckDB, their function definitions have catalog and schema. We can create a macro or UDF with the specific catalog or schema prefix. Then, we can set up the search_path to decide where to find the functions. If no matching candidates are found in the search path, we can find the candidate in the default path.

If we have similar features, I imagine I can follow these steps to extend an existing function easily:

1. Create a schema `self`.
2. Implement an `avg` function for Duration.
3. Register the `self.avg` function.
4. Set `search_path` to `self`.
5. Query the SQL `select avg(time_col1 - time_col2) from table1` (Invoke self.avg internally).
6. Query the SQL `select avg(int_col) from table1` (Invoke default avg internally).

To approach this proposal, I think we need two new features:

- Register and use UDFs with the catalog and schema. I think we can do something at:
https://github.com/apache/datafusion/blob/d460abb63f6ff3abce6de022834f6f17c77db411/datafusion/core/src/execution/session_state.rs#L1773-L1778
- Implement `search_path` for searching function candidates.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the function-registration area in datafusion/core/src/execution/session_state.rs referenced by the issue, then investigate how function candidates are resolved. The proposed work covers catalog/schema-qualified UDF registration and a search_path. Done means a schema-qualified custom self.avg can handle Duration while the default avg still handles integer input.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.