apache / apache/incubator-xtable

Add support for OneLake Catalog in RunCatalogSync

Open
#810 5 comments 0 reactions 1 assignee Claimed by @vaibhavk1992 View on GitHub
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

## Summary

Add Microsoft Fabric's OneLake Catalog as a new catalog sync target in XTable's `RunCatalogSync`, alongside the existing Glue, HMS, and (documented) Unity Catalog integrations. This would allow XTable users to automatically register and refresh tables in Microsoft Fabric lakehouses after format conversions, enabling discovery through the OneLake Catalog UI (embedded in Teams, Excel, and Copilot Studio).

## Motivation

Microsoft Fabric is a widely adopted analytics platform, and many organizations store their lakehouse data in OneLake. Currently, XTable users targeting Fabric must manually register converted tables. A native `CatalogSyncClient` for OneLake would close this gap and bring XTable's catalog sync story to parity across the major cloud data platforms:

| Cloud | Catalog | XTable Support |
|-------|---------|----------------|
| AWS | Glue Data Catalog | Implemented |
| On-prem / multi-cloud | Hive Metastore (HMS) | Implemented |
| Databricks | Unity Catalog | Documented (manual SQL DDL) |
| **Microsoft Fabric** | **OneLake Catalog** | **Not yet supported** |

## Available APIs

Microsoft Fabric exposes several REST APIs that could be used to implement this:

1. **Fabric Lakehouse REST API** (`https://api.fabric.microsoft.com/v1/workspaces/{id}/lakehouses/{id}/tables`) — supports listing tables, loading files into Delta tables, and table maintenance. This is the most promising path for write operations (creating/refreshing table registrations).

2. **OneLake Table APIs** (`https://onelake.table.fabric.microsoft.com`) — provides read-only metadata operations compatible with the Iceberg REST Catalog (IRC) and Unity Catalog API standards. Write operations are not yet supported but are on the roadmap.

3. **OneLake filesystem APIs** (ADLS Gen2-compatible) — for direct file operations on OneLake storage paths (`abfss://`).

Authentication uses Microsoft Entra ID (Azure AD) OAuth tokens with the `https://storage.azure.com/` audience.

## Proposed Implementation

Following the existing Glue/HMS patterns:

1. **New module**: `xtable-azure` (or add to an existing module)
2. **`OneLakeCatalogSyncClient`** implementing `CatalogSyncClient` with `getCatalogType()` returning `"ONELAKE"`
3. **`OneLakeCatalogConfig`** for Fabric-specific config (tenant ID, workspace ID, lakehouse ID, auth settings)
4. **Per-format `CatalogTableBuilder` implementations** (Delta, Iceberg, Hudi) using the Fabric Lakehouse REST API
5. **`CatalogType.ONELAKE`** constant
6. **ServiceLoader registration** in `META-INF/services/`
7. **Optional**: `CatalogConversionSource` implementation using the read-only OneLake Table APIs

## References

- [OneLake Catalog Overview](https://learn.microsoft.com/en-us/fabric/governance/onelake-catalog-overview)
- [OneLake Table APIs Overview](https://learn.microsoft.com/en-us/fabric/onelake/table-apis/table-apis-overview)
- [Fabric Lakehouse REST API](https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-api)
- [OneLake Table APIs for Delta (Unity Catalog compatible)](https://learn.microsoft.com/en-us/fabric/onelake/table-apis/delta-table-apis-overview)
- [OneLake Table APIs for Iceberg (IRC compatible)](https://learn.microsoft.com/en-us/fabric/onelake/table-apis/iceberg-table-apis-overview)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.