duckdb / duckdb/unity_catalog

Credential vending is AWS-only — GCS and Azure tables can't be read

Open
#116 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
110
Forks
44
Avg merge
2d 13h
Merged PRs (30d)
4

Description

## Summary

The extension only handles AWS credentials, so Unity Catalog tables backed by GCS or Azure can't be read. `ATTACH … (TYPE unity_catalog)` succeeds and `SHOW ALL TABLES` works — both are metadata-only — but `SELECT` fails once credentials are vended, because the vended credential is dropped and an S3 secret is created regardless of where the table actually lives.

## Detail

Unity Catalog vends a different credential shape per cloud (per the UC API spec, `TemporaryCredentials`):

| cloud | field | kind |
| --- | --- | --- |
| AWS | `aws_temp_credentials` | key + secret + session token |
| Azure | `azure_user_delegation_sas` | SAS token |
| GCP | `gcp_oauth_token` | OAuth bearer token |

The delta/v1 credentials endpoint uses a parallel shape: `s3.access-key-id` … for AWS, `gcs.oauth-token` for GCP.

Four touchpoints assume AWS:

| file:line | what |
| --- | --- |
| `src/uc_api.cpp:409` | reads only `aws_temp_credentials` |
| `src/uc_api.cpp:441` | reads only `s3.access-key-id` and siblings |
| `src/storage/uc_table_set.cpp:214` | `input.type = "s3"`, unconditionally |
| `src/storage/uc_table_entry.cpp:206` | `sec.type = "s3"`, unconditionally (scan-plan path) |

The `ATTACH` surface is AWS-shaped too — `AWS_REGION` is the only cloud parameter the `unity_catalog` secret accepts.

## Suggested change

Branch on the table's `storage_location` scheme and create a matching secret:

- `gs://` → `gcs` secret carrying `bearer_token` from `gcp_oauth_token.oauth_token` (or `gcs.oauth-token` on the managed path)
- `abfss://`/`az://` → `azure` secret from `azure_user_delegation_sas.sasToken`
- `s3://` → unchanged

Everything downstream is already cloud-agnostic: httpfs handles GCS and duckdb-azure handles Azure, and duckdb-delta has working branches for both.

## Dependency

For GCS the Delta *log* read additionally needs duckdb/duckdb-delta#356 — the bearer token can't currently reach delta-kernel-rs's object store. The data-file half already works through httpfs. Azure has no such blocker: object_store 0.13.2 already exposes a SAS config key.

I have the GCS half working end-to-end against a live UC workspace and am happy to contribute this change too, once the duckdb-delta side has a direction.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.