Query QuestDB data from Spark, Trino and Snowflake via Iceberg and Delta Lake

Start with the cold-storage manager and the Cold Storage v2 work in #130, then trace the partition state exposed by table_cold_partitions(). Define the publish, catalog, schema-evolution, retention, and Delta Lake boundaries from the scope; done means the checklist is implemented and external readers can observe consistent snapshots.

Summary

QuestDB Enterprise cold storage already writes each partition to the object store as standard Parquet. This item makes that data readable outside QuestDB by keeping lakehouse table metadata in sync with the partition state QuestDB already tracks for cold storage. Every upload, compaction and drop becomes an atomic snapshot of an Apache Iceberg table, and in a second phase a Delta Lake table, so Spark, Trino, DuckDB, Snowflake, Databricks and PyIceberg query QuestDB's data in place with no copy and no scheduled job.

QuestDB remains the only writer. The lakehouse metadata is a projection of QuestDB's partition transaction log, not a second write path.

Current Limitations

• Registering cold partitions as Iceberg tables is manual: users run PyIceberg or the Iceberg Java API and schedule it to pick up new partitions
• Manual registration cannot follow QuestDB rewriting a partition's Parquet: today a late write re-uploads a live partition in place, and Cold Storage v2 (#130) compaction publishes a new base under a new object key and retires the one an external table still references
• Remote garbage collection knows nothing about external snapshots, so a retired file can disappear under a reader
• Registering with PyIceberg writes Iceberg format v2, so nanosecond timestamps are downcast to microseconds and UUIDs are exposed as fixed[16] rather than the Iceberg uuid type

Features

How it works

1. The cold-storage manager, one per cluster, uploads or compacts a partition as it does today.
2. In the same step it commits an Iceberg snapshot through the Iceberg catalog: data files added and removed, column statistics from the Parquet footers, and schema or partition spec updates if the table changed.
3. External engines query the Iceberg table straight from the object store. QuestDB is not involved in the read.
4. Remote garbage collection deletes a retired file only after the last snapshot referencing it has expired.
5. External readers see data as of the last publish. Late rows held in local Delta Runs (#130) become visible once they are compacted into the remote base.

Example

Illustrative syntax:

-- QuestDB: tier partitions to the object store and publish them as an Iceberg table
CREATE TABLE trades (...)
    STORAGE POLICY (TO PARQUET 1d, TO REMOTE 3d, DROP LOCAL 30d)
    ICEBERG CATALOG 'lakehouse' TABLE 'market.trades';

-- Trino, Spark or DuckDB: query the same files with no copy
SELECT symbol, count(*)
FROM lakehouse.market.trades
WHERE ts >= DATE '2026-01-01'
GROUP BY symbol;

Benefits

• Readable outside QuestDB — any Iceberg engine reads cold data in place, with partition pruning and column statistics; Delta engines follow in the second phase
• No pipelines — no export job, no cron, no second copy of the data
• Always consistent — atomic snapshots mean no torn reads across uploads or compaction
• Full fidelity — nanosecond timestamps, UUIDs and decimals up to precision 38 are exposed with their native Iceberg types
• Time travel — Iceberg snapshots expose earlier versions of cold data within the retention window

Scope

• Iceberg metadata writer in the cold-storage manager: snapshots for upload, compaction and partition removal
• REST and AWS Glue catalog support
• Type mappings, field ID alignment, name mapping for existing partitions, format v3 for nanosecond tables
• Snapshot-aware remote garbage collection with a retention setting
• Schema evolution mapping
• Published snapshot per partition in table_cold_partitions() and publish lag per table
• Second phase: Delta Lake _delta_log publishing

Enterprise feature: requires cold storage. Catalog order, the format version policy and the snapshot retention default remain work in progress.

Builds on #130 (Cold Storage v2). Supersedes #91.