duckdb / duckdb/unity_catalog

INTERNAL Error on time-travel SELECT against Databricks UC managed Delta table

Open
#79 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
110
Forks
44
Avg merge
2d 13h
Merged PRs (30d)
4

Description

# `INTERNAL Error: Attempting to dereference an optional pointer that is not set` on time-travel `SELECT … AT (VERSION => 0)` against a Databricks Unity Catalog managed Delta table

## Summary

Issuing a time-travel SELECT against a managed Delta table that lives in a Databricks Unity Catalog metastore crashes the DuckDB process with an `INTERNAL Error` (assertion failure) inside `UCTableSet::GetEntry`. The current-state SELECT against the same table works fine; only the versioned lookup crashes.

Per the extension's [Features doc](https://duckdb.org/docs/current/core_extensions/unity_catalog#features), time travel (`SELECT * FROM .. AT (VERSION => ..);`) is listed as a supported feature.

## Environment

- **DuckDB:** v1.5.2 (Variegata) `8a5851971f` (CLI, installed via Homebrew)
- **Python module:** `duckdb==1.5.2` (separate, also reproduces)
- **Extensions:**
- `unity_catalog` 1.5.2 build, installed from `extensions.duckdb.org`
- `delta` 1.5.2 build, installed from `extensions.duckdb.org`
- **Platform:** macOS arm64 (`osx_arm64`)
- **UC backend:** Databricks Unity Catalog — workspace enrolled in the **External Access to Unity Catalog Managed Delta Table** preview, metastore has *External Data Access* enabled.

## Reproduction

1. Inside Databricks, create a catalog + schema and seed a managed Delta table from the built-in TPCH sample (this is one of several `CREATE OR REPLACE TABLE` statements in our setup script):

```sql
CREATE OR REPLACE TABLE ..orders
AS SELECT * FROM samples.tpch.orders;
```

This produces commit version 0 with `engineInfo = Databricks-Runtime/...` and *without* the `delta.feature.catalogManaged` property.

2. From outside Databricks, append a few rows via external Apache Spark using `delta-spark 4.2.0` + `unitycatalog-spark 0.4.1`. The append succeeds and produces commit versions 1+ where `engineInfo = Apache-Spark/4.1.1 Delta-Lake/4.2.0`. (Adding the `delta.feature.catalogManaged='supported'` property happens during the external write path.)

3. From DuckDB CLI:

```sql
INSTALL unity_catalog; LOAD unity_catalog;
INSTALL delta; LOAD delta;

CREATE OR REPLACE SECRET (
TYPE UNITY_CATALOG,
TOKEN '',
ENDPOINT 'https://.cloud.databricks.com/',
AWS_REGION ''
);

ATTACH '' AS (TYPE UNITY_CATALOG);
USE .;

-- This works:
SELECT * FROM orders LIMIT 3;

-- This crashes:
SELECT * FROM orders AT (VERSION => 0) LIMIT 3;
```

## Expected behavior

The time-travel SELECT returns 3 rows from commit version 0 of the table (the original snapshot before any external commits), per the documented feature.

## Actual behavior

```
INTERNAL Error:
Attempting to dereference an optional pointer that is not set
```

DuckDB exits with an assertion failure (the "internal error" page at https://duckdb.org/docs/stable/dev/internal_errors says these are always bugs).

The current-state read (`SELECT * FROM orders LIMIT 3`) returns rows correctly using the same SECRET / ATTACH. Only the `AT (VERSION => N)` clause triggers the crash.

## Stack trace

```
0 _ZN6duckdb9ExceptionC2ENS_13ExceptionTypeERKNSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEE + 52
1 _ZN6duckdb17InternalExceptionC1ERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEE + 20
2 _ZNK6duckdb12optional_ptrINS_12CatalogEntryELb1EE10CheckValidEv + 92
3 _ZN6duckdb7Catalog9GetSchemaERNS_13ClientContextERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEE + 164
4 _ZN6duckdb16TableInformation10GetVersionERNS_13ClientContextERKNS_15EntryLookupInfoE + 588
5 _ZN6duckdb10UCTableSet8GetEntryERNS_13ClientContextERKNS_15EntryLookupInfoE + 128
6 _ZN6duckdb7Catalog22TryLookupEntryInternalENS_18CatalogTransactionERKNSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEERKNS_15EntryLookupInfoE + 192
7 duckdb::Catalog::TryLookupEntry(duckdb::CatalogEntryRetriever&, duckdb::vector> const&, duckdb::EntryLookupInfo const&, duckdb::OnEntryNotFound, bool) + 208
8 duckdb::Catalog::TryLookupEntry(duckdb::CatalogEntryRetriever&, std::__1::basic_string, std::__1::allocator> const&, std::__1::basic_string, std::__1::allocator> const&, duckdb::EntryLookupInfo const&, duckdb::OnEntryNotFound) + 620
9 duckdb::Catalog::GetEntry(...)
10 duckdb::CatalogEntryRetriever::GetEntry(...)
11 duckdb::Binder::Bind(duckdb::BaseTableRef&) + 432
12 duckdb::Binder::Bind(duckdb::TableRef&) + 144
13 duckdb::Binder::BindNode(duckdb::SelectNode&) + 84
14 duckdb::Binder::BindNode(duckdb::QueryNode&) + 412
15 duckdb::Planner::CreatePlan(duckdb::SQLStatement&) + 156
... (full trace truncated)
```

The relevant frames are 2–5: an `optional_ptr::CheckValid` assert reached from `Catalog::GetSchema`, called by `TableInformation::GetVersion`, called by `UCTableSet::GetEntry`. Looks like the unity_catalog extension's versioned-lookup path expected to find a schema entry that was not present.

## Additional context

We have also seen the same `INTERNAL Error: Attempting to dereference an optional pointer that is not set` when querying the *current* state of tables that were created entirely by external engines with `delta.feature.catalogManaged='supported'` from version 0 onward (e.g. an `orders_summary` produced by external `delta-spark 4.2.0` CTAS, or an `orders_stream` produced by external Spark Structured Streaming). Direct UC REST calls (`/api/2.1/unity-catalog/tables`) with the same SP OAuth token return 200 with the table metadata, so the catalog responses themselves look reachable and well-formed; the failure is in how the unity_catalog extension consumes them.

That suggests two related bugs (or one with two manifestations) in the unity_catalog `osx_arm64` 1.5.2 build's lookup paths against Databricks UC:

1. Time-travel `AT (VERSION => N)` against any managed Delta table.
2. Current-state SELECT against tables that have `delta.feature.catalogManaged='supported'` from version 0.

I don't have a UC-OSS repro at hand — the failures may be specific to the Databricks UC REST surface.

## Workspace info

The two Databricks workspaces I reproduced this against are not public; happy to provide additional response captures if helpful. Both metastores are on AWS `us-west-2`.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.