unitycatalog / unitycatalog/unitycatalog
Question about the delta catalog managed commit
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 3.5k
- Forks
- 672
- Avg merge
- 6d 16h
- Merged PRs (30d)
- 30
Description
When I try to create a table following the demo code. https://github.com/unitycatalog/unitycatalog/blob/2e4376a7ed6b506315fa8c54de273e48[…]park/src/main/scala/io/unitycatalog/spark/UCSingleCatalog.scala
Step 1. Create a staging table
Step 2. Get the tmp credential via staging table, and commit the 0000.json to the staging table location.
Step3. Create the normal table(External) using the same table name, staging table location.
But at the Step3, it throw an exception.
com.databricks.sdk.core.error.platform.InvalidParameterValue: Input path url 's3://xxxxxxx/unity-catalog/xxxxxx/__unitystorage/catalogs/00216c35-b906-4425-ba63-5ccbeab49acd/tables/fb1bb278-34e3-4cc2-9f4c-736c46d70efa' overlaps with managed storage within 'CreateTable' call.
Because I try to create a External table with the managed storage, the External table location is the staging table location.
But if I try to change the table type to Managed at Step3, it will throw another exception.
com.databricks.sdk.core.error.platform.InvalidParameterValue: Request to create table
'streamnative_workspace.public_default.test__unity__catalog__yan__7' from outside of Databricks Unity Catalog enabled compute environment is denied. Table type must be EXTERNAL to create table externally, instead it is 'MANAGED'.
Anyone can help to check this case? Thanks (edited)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with connectors/spark/src/main/scala/io/unitycatalog/spark/UCSingleCatalog.scala around line 92 and reproduce the three-step staging-table flow from the issue. Compare the EXTERNAL and MANAGED CreateTable failures, then document the expected supported flow and verify it no longer produces the reported exceptions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100