Support Custom Function Registration with Catalog and Schema
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 362
Description
### Is your feature request related to a problem or challenge?
In my case, I would like to extend a default function's supporting signature. For example, the `avg` function in DataFusion doesn't support calculating the Duration average.
However, suppose I want to register a custom `avg` for Duration. In that case, it means that I need to re-implement the whole `avg` function because the new implementation will override the previous one when registered. I want to propose an approach to register the custom implementation and keep the original implementation at the same time.
### Describe the solution you'd like
In Postgres or DuckDB, their function definitions have catalog and schema. We can create a macro or UDF with the specific catalog or schema prefix. Then, we can set up the search_path to decide where to find the functions. If no matching candidates are found in the search path, we can find the candidate in the default path.
If we have similar features, I imagine I can follow these steps to extend an existing function easily:
1. Create a schema `self`.
2. Implement an `avg` function for Duration.
3. Register the `self.avg` function.
4. Set `search_path` to `self`.
5. Query the SQL `select avg(time_col1 - time_col2) from table1` (Invoke self.avg internally).
6. Query the SQL `select avg(int_col) from table1` (Invoke default avg internally).
To approach this proposal, I think we need two new features:
- Register and use UDFs with the catalog and schema. I think we can do something at:
https://github.com/apache/datafusion/blob/d460abb63f6ff3abce6de022834f6f17c77db411/datafusion/core/src/execution/session_state.rs#L1773-L1778
- Implement `search_path` for searching function candidates.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start with the function-registration area in datafusion/core/src/execution/session_state.rs referenced by the issue, then investigate how function candidates are resolved. The proposed work covers catalog/schema-qualified UDF registration and a search_path. Done means a schema-qualified custom self.avg can handle Duration while the default avg still handles integer input.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100