unitycatalog / unitycatalog/unitycatalog

Question about the delta catalog managed commit

Open
#1,332 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
3.5k
Forks
672
Avg merge
6d 16h
Merged PRs (30d)
30

Description

When I try to create a table following the demo code.  https://github.com/unitycatalog/unitycatalog/blob/2e4376a7ed6b506315fa8c54de273e48[…]park/src/main/scala/io/unitycatalog/spark/UCSingleCatalog.scala

Step 1. Create a staging table
Step 2. Get the tmp credential via staging table, and commit the 0000.json to the staging table location.
Step3. Create the normal table(External) using the same table name, staging table location.

But at the Step3, it throw an exception.

com.databricks.sdk.core.error.platform.InvalidParameterValue: Input path url 's3://xxxxxxx/unity-catalog/xxxxxx/__unitystorage/catalogs/00216c35-b906-4425-ba63-5ccbeab49acd/tables/fb1bb278-34e3-4cc2-9f4c-736c46d70efa' overlaps with managed storage within 'CreateTable' call. 

Because I try to create a External  table with the managed storage, the External  table location is the staging table location.

But if I try to change the table type to Managed at Step3, it will throw another exception.
com.databricks.sdk.core.error.platform.InvalidParameterValue: Request to create table

'streamnative_workspace.public_default.test__unity__catalog__yan__7' from outside of Databricks Unity Catalog enabled compute environment is denied. Table type must be EXTERNAL to create table externally, instead it is 'MANAGED'.

Anyone can help to check this case? Thanks  (edited)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with connectors/spark/src/main/scala/io/unitycatalog/spark/UCSingleCatalog.scala around line 92 and reproduce the three-step staging-table flow from the issue. Compare the EXTERNAL and MANAGED CreateTable failures, then document the expected supported flow and verify it no longer produces the reported exceptions.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.