unitycatalog / unitycatalog/unitycatalog

Spark connector does not support invoking Unity Catalog functions

Open
#1,386 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
3.5k
Forks
672
Avg merge
6d 16h
Merged PRs (30d)
30

Description

Summary

Functions can be created and stored in Unity Catalog via the CLI (bin/uc function create), but they cannot be invoked from Spark SQL. When referencing a function in a query (e.g. catalog.schema.function_name(column)), Spark throws AnalysisException: Catalog development does not support functions.

Environment

  • Unity Catalog server: 0.4.0
  • Spark: 4.0.1
  • Delta Lake: 4.0.1
  • PostgreSQL: 17.7

Steps to Reproduce

1. Create a function via CLI:

bin/uc function create
--full_name development.sales.format_order_status
--input_params "status STRING"
--data_type STRING
--def "return 'In behandeling' if status == 'processing' else 'Verzonden' if status == 'shipped' else 'Afgeleverd' if status == 'delivered' else status"
--comment "Formats order status for display in Dutch."

2. Use the function in Spark SQL:

spark.sql("""SELECT *, development.sales.format_order_status(status) AS status_nl FROM development.sales.orders_884 LIMIT 100""")

Errors Observed

Client (Spark / PySpark)
AnalysisException: Catalog development does not support functions.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with the bin/uc function create command and the Spark SQL query shown in the issue. Trace how Spark resolves development.sales.format_order_status and compare that path with the Unity Catalog function created through the CLI. Done means the same SQL invocation succeeds instead of reporting that the catalog does not support functions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, postgresql, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.