duckdb / duckdb/unity_catalog

Reading external delta table registered in unity catalog doesn't work

Open
#109 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
110
Forks
44
Avg merge
2d 13h
Merged PRs (30d)
4

Description

Thanks for building this extension. I've found some scenarios where DuckDB's performance seems pretty magical, so I'm keen to use DuckDB if I can, but I'm currently having issues with unity catalog.

Created a simple external table in our Azure Databricks unity catalog
Image
Querying this with other delta table clients works fine.

Attempt to read this using DuckDB CLI:
Install extensions, setup auth and attach a unity catalog
```
INSTALL delta;
LOAD delta;
INSTALL azure;
LOAD azure;
INSTALL unity_catalog;
LOAD unity_catalog;
CREATE SECRET (
TYPE azure,
PROVIDER credential_chain,
CHAIN cli
);
CREATE SECRET (
TYPE unity_catalog,
TOKEN ,
ENDPOINT
);
ATTACH 'dev_user' as dev_user (TYPE UC_CATALOG);
```
NOTE: we shouldn't need to configure the `azure` secret. It should be using unity catalog token federation, but that is a [separate issue](https://github.com/duckdb/unity_catalog/issues/36).

Test that unity catalog is working to some extent
```
SHOW TABLES FROM dev_user;
```
Returns a correct list of tables

Attempt to read an external table
```
SELECT * FROM dev_user.tomnewton.external_unity_catalog;
```
```
Binder Error:
SELECT list is empty after resolving * expressions!
```

I'm testing with duckdb 1.5.5 using the python client and the CLI. Attempting to connect to unity catalog in an Azure managed Databrticks using unity_catalog extension version fd85147

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the DuckDB CLI reproduction using the delta, azure, and unity_catalog extensions, then verify that SHOW TABLES FROM dev_user works before querying the external table. Trace the unity_catalog external-table path and compare it with the working table listing; done means SELECT * FROM dev_user.tomnewton.external_unity_catalog returns the table columns and rows instead of an empty SELECT list error.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, cpp
Domain
cloud, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.