INTERNAL Error on time-travel SELECT against Databricks UC managed Delta table
- Dominant language
- C++
- Stars
- 110
- Forks
- 44
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 4
Description
# `INTERNAL Error: Attempting to dereference an optional pointer that is not set` on time-travel `SELECT … AT (VERSION => 0)` against a Databricks Unity Catalog managed Delta table
## Summary
Issuing a time-travel SELECT against a managed Delta table that lives in a Databricks Unity Catalog metastore crashes the DuckDB process with an `INTERNAL Error` (assertion failure) inside `UCTableSet::GetEntry`. The current-state SELECT against the same table works fine; only the versioned lookup crashes.
Per the extension's [Features doc](https://duckdb.org/docs/current/core_extensions/unity_catalog#features), time travel (`SELECT * FROM .. AT (VERSION => ..);`) is listed as a supported feature.
## Environment
- **DuckDB:** v1.5.2 (Variegata) `8a5851971f` (CLI, installed via Homebrew)
- **Python module:** `duckdb==1.5.2` (separate, also reproduces)
- **Extensions:**
- `unity_catalog` 1.5.2 build, installed from `extensions.duckdb.org`
- `delta` 1.5.2 build, installed from `extensions.duckdb.org`
- **Platform:** macOS arm64 (`osx_arm64`)
- **UC backend:** Databricks Unity Catalog — workspace enrolled in the **External Access to Unity Catalog Managed Delta Table** preview, metastore has *External Data Access* enabled.
## Reproduction
1. Inside Databricks, create a catalog + schema and seed a managed Delta table from the built-in TPCH sample (this is one of several `CREATE OR REPLACE TABLE` statements in our setup script):
```sql
CREATE OR REPLACE TABLE ..orders
AS SELECT * FROM samples.tpch.orders;
```
This produces commit version 0 with `engineInfo = Databricks-Runtime/...` and *without* the `delta.feature.catalogManaged` property.
2. From outside Databricks, append a few rows via external Apache Spark using `delta-spark 4.2.0` + `unitycatalog-spark 0.4.1`. The append succeeds and produces commit versions 1+ where `engineInfo = Apache-Spark/4.1.1 Delta-Lake/4.2.0`. (Adding the `delta.feature.catalogManaged='supported'` property happens during the external write path.)
3. From DuckDB CLI:
```sql
INSTALL unity_catalog; LOAD unity_catalog;
INSTALL delta; LOAD delta;
CREATE OR REPLACE SECRET (
TYPE UNITY_CATALOG,
TOKEN '',
ENDPOINT 'https://.cloud.databricks.com/',
AWS_REGION ''
);
ATTACH '' AS (TYPE UNITY_CATALOG);
USE .;
-- This works:
SELECT * FROM orders LIMIT 3;
-- This crashes:
SELECT * FROM orders AT (VERSION => 0) LIMIT 3;
```
## Expected behavior
The time-travel SELECT returns 3 rows from commit version 0 of the table (the original snapshot before any external commits), per the documented feature.
## Actual behavior
```
INTERNAL Error:
Attempting to dereference an optional pointer that is not set
```
DuckDB exits with an assertion failure (the "internal error" page at https://duckdb.org/docs/stable/dev/internal_errors says these are always bugs).
The current-state read (`SELECT * FROM orders LIMIT 3`) returns rows correctly using the same SECRET / ATTACH. Only the `AT (VERSION => N)` clause triggers the crash.
## Stack trace
```
0 _ZN6duckdb9ExceptionC2ENS_13ExceptionTypeERKNSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEE + 52
1 _ZN6duckdb17InternalExceptionC1ERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEE + 20
2 _ZNK6duckdb12optional_ptrINS_12CatalogEntryELb1EE10CheckValidEv + 92
3 _ZN6duckdb7Catalog9GetSchemaERNS_13ClientContextERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEE + 164
4 _ZN6duckdb16TableInformation10GetVersionERNS_13ClientContextERKNS_15EntryLookupInfoE + 588
5 _ZN6duckdb10UCTableSet8GetEntryERNS_13ClientContextERKNS_15EntryLookupInfoE + 128
6 _ZN6duckdb7Catalog22TryLookupEntryInternalENS_18CatalogTransactionERKNSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEERKNS_15EntryLookupInfoE + 192
7 duckdb::Catalog::TryLookupEntry(duckdb::CatalogEntryRetriever&, duckdb::vector> const&, duckdb::EntryLookupInfo const&, duckdb::OnEntryNotFound, bool) + 208
8 duckdb::Catalog::TryLookupEntry(duckdb::CatalogEntryRetriever&, std::__1::basic_string, std::__1::allocator> const&, std::__1::basic_string, std::__1::allocator> const&, duckdb::EntryLookupInfo const&, duckdb::OnEntryNotFound) + 620
9 duckdb::Catalog::GetEntry(...)
10 duckdb::CatalogEntryRetriever::GetEntry(...)
11 duckdb::Binder::Bind(duckdb::BaseTableRef&) + 432
12 duckdb::Binder::Bind(duckdb::TableRef&) + 144
13 duckdb::Binder::BindNode(duckdb::SelectNode&) + 84
14 duckdb::Binder::BindNode(duckdb::QueryNode&) + 412
15 duckdb::Planner::CreatePlan(duckdb::SQLStatement&) + 156
... (full trace truncated)
```
The relevant frames are 2–5: an `optional_ptr::CheckValid` assert reached from `Catalog::GetSchema`, called by `TableInformation::GetVersion`, called by `UCTableSet::GetEntry`. Looks like the unity_catalog extension's versioned-lookup path expected to find a schema entry that was not present.
## Additional context
We have also seen the same `INTERNAL Error: Attempting to dereference an optional pointer that is not set` when querying the *current* state of tables that were created entirely by external engines with `delta.feature.catalogManaged='supported'` from version 0 onward (e.g. an `orders_summary` produced by external `delta-spark 4.2.0` CTAS, or an `orders_stream` produced by external Spark Structured Streaming). Direct UC REST calls (`/api/2.1/unity-catalog/tables`) with the same SP OAuth token return 200 with the table metadata, so the catalog responses themselves look reachable and well-formed; the failure is in how the unity_catalog extension consumes them.
That suggests two related bugs (or one with two manifestations) in the unity_catalog `osx_arm64` 1.5.2 build's lookup paths against Databricks UC:
1. Time-travel `AT (VERSION => N)` against any managed Delta table.
2. Current-state SELECT against tables that have `delta.feature.catalogManaged='supported'` from version 0.
I don't have a UC-OSS repro at hand — the failures may be specific to the Databricks UC REST surface.
## Workspace info
The two Databricks workspaces I reproduced this against are not public; happy to provide additional response captures if helpful. Both metastores are on AWS `us-west-2`.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.