unitycatalog / unitycatalog/unitycatalog
Spark connector does not support invoking Unity Catalog functions
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 3.5k
- Forks
- 672
- Avg merge
- 6d 16h
- Merged PRs (30d)
- 30
Description
Summary
Functions can be created and stored in Unity Catalog via the CLI (bin/uc function create), but they cannot be invoked from Spark SQL. When referencing a function in a query (e.g. catalog.schema.function_name(column)), Spark throws AnalysisException: Catalog development does not support functions.
Environment
- Unity Catalog server: 0.4.0
- Spark: 4.0.1
- Delta Lake: 4.0.1
- PostgreSQL: 17.7
Steps to Reproduce
1. Create a function via CLI:
bin/uc function create
--full_name development.sales.format_order_status
--input_params "status STRING"
--data_type STRING
--def "return 'In behandeling' if status == 'processing' else 'Verzonden' if status == 'shipped' else 'Afgeleverd' if status == 'delivered' else status"
--comment "Formats order status for display in Dutch."
2. Use the function in Spark SQL:
spark.sql("""SELECT *, development.sales.format_order_status(status) AS status_nl FROM development.sales.orders_884 LIMIT 100""")
Errors Observed
Client (Spark / PySpark)
AnalysisException: Catalog development does not support functions.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failure with the bin/uc function create command and the Spark SQL query shown in the issue. Trace how Spark resolves development.sales.format_order_status and compare that path with the Unity Catalog function created through the CLI. Done means the same SQL invocation succeeds instead of reporting that the catalog does not support functions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, postgresql, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100