Azure / Azure/azure-sdk-for-cpp

Add OneLake workspace endpoint factory to Data Lake client

Open
#7,274 1 comment 0 reactions 0 assignees View on GitHub
customer-reported needs-triage question
Dominant language
C++
Stars
205
Forks
172
Avg merge
1d 3h
Merged PRs (30d)
37

Description

# OneLake Workspace Endpoint Support for Azure SDK for C++

## Tracking

- Private Link implementation: [ADO 5112511](https://msdata.visualstudio.com/Database%20Systems/_workitems/edit/5112511)

## Motivation

Microsoft Fabric workspace-level private links require OneLake data-plane requests for a GUID workspace to use a workspace-specific DFS host. Consumers of `azure-storage-files-datalake` currently have to duplicate cloud/ring detection, GUID normalization, endpoint validation, and hostname construction before creating a Data Lake client.

The merged `microsoft/pg_azure_cdc` implementation demonstrates the scenario across PostgreSQL 14-19, but the endpoint rule is a OneLake service contract rather than CDC business logic.

Microsoft documents OneLake as compatible with Azure Data Lake Storage and Blob APIs and publishes both workspace-scoped DFS and Blob FQDNs. Azure Storage for Java already contains OneLake-specific compatibility behavior. The Data Lake package is therefore the narrowest existing C++ package that owns both endpoint forms and the workspace-as-filesystem hierarchy.

## Proposed Public API

Add this static factory to `Azure::Storage::Files::DataLake::DataLakeFileSystemClient`:

```cpp
static DataLakeFileSystemClient CreateForOneLakeWorkspace(
const std::string& oneLakeServiceUrl,
const std::string& workspaceId,
std::shared_ptr credential,
const DataLakeClientOptions& options = DataLakeClientOptions());
```

Example:

```cpp
auto credential = std::make_shared();
auto workspaceClient = DataLakeFileSystemClient::CreateForOneLakeWorkspace(
"https://onelake.dfs.fabric.microsoft.com", workspaceId, credential);
auto directoryClient = workspaceClient.GetDirectoryClient(
artifactId + "/Files/landing");
```

The factory is explicit rather than an option on generic Data Lake clients. Callers opt into OneLake endpoint semantics by choosing the factory.

`DataLakeFileSystemClient` is the correct hierarchy level because OneLake maps a workspace to an ADLS filesystem/container. A directory factory would mix endpoint selection with Fabric item paths. A public URL resolver would expose endpoint mechanics without constructing a consistently configured DFS/Blob client pair.

## Behavior

1. Inspect the raw input and reject user information (`@`), query strings, and fragments before parsing it with `Azure::Core::Url`.
2. Require HTTPS, no explicit port, and an empty path or `/`.
3. Validate `workspaceId` as a 32-character compact GUID or 36-character hyphenated GUID.
4. Normalize the GUID to lowercase without hyphens for DNS.
5. Accept the endpoint grammar below for an approved Fabric cloud. Endpoint matching is case-insensitive; generated hosts are lowercase.
6. For supported shared OneLake endpoints, construct and validate the companion pair explicitly:

- DFS: `.z.dfs.`
- Blob: `.z.blob.`

The factory does not use the generic `.dfs.` to `.blob.` mapper because ring hosts use labels such as `daily-dfs` and `daily-blob`. It creates the DFS pipeline and Blob container client from their respective validated URLs, then returns the file-system client through its existing private constructor.
7. Append the caller-provided workspace ID as the encoded file-system path segment to both URLs.
8. Accept shared DFS, shared Blob, general API, workspace DFS, and workspace Blob inputs. Normalize every accepted input to the same workspace DFS/Blob pair. For a workspace input, require its compact ID and `zXY` label to agree with `workspaceId`; preserve the canonical ring and approved cloud from the input host.
9. Reject a workspace host for a different workspace, an incorrect `zXY` label, malformed DNS labels, trailing-dot hosts, IP literals, unsupported domains, and dedicated endpoints. CDC continues to use the existing constructor for dedicated endpoints, where workspace routing does not apply.
10. Reject unsafe or unsupported client options before constructing either client:
- nonempty `SecondaryHostForRetryReads`;
- a populated `CustomerProvidedKey`;
- an `Audience` other than `DataLakeAudience::DefaultAudience`.
11. Reject a null token credential.
12. Construction makes no token request and no network request.

Supported cloud domains:

- `fabric.microsoft.com`
- `fabric-df.microsoft.com`
- `fabric.microsoft.us`
- `fabric.sovcloud-api.fr`

The proposed support matrix is normative for the initial implementation and matches the existing OneLakeClient/CDC behavior. OneLake service-owner approval of the matrix is a release gate; the SDK and CDC integration will not ship with a partial matrix.

Accepted service URL authorities for each approved cloud:

- `https://onelake.dfs.`
- `https://onelake.blob.`
- `https://-onelake.dfs.`
- `https://-onelake.blob.`
- `https://api.onelake.`
- `https://-api.onelake.`
- `https://.z.dfs.`
- `https://.z.blob.`

`` is a nonempty DNS prefix made of ASCII lowercase letters, digits, and hyphens, with no leading, trailing, or adjacent hyphen. Region prefixes are not copied into workspace FQDNs.

Known ring aliases take precedence over regional-prefix parsing:

- base
- `daily-` and `i-daily-` -> `daily-`
- `dxt-` and `i-dxt-` -> `dxt-`
- `msit-` and `i-msit-` -> `msit-`

An unrecognized valid prefix on a shared endpoint is treated as a region and removed. A ring prefix is valid on either the shared DFS/Blob endpoint or the general API endpoint. Workspace hosts accept only the canonical output ring (`daily-`, `dxt-`, or `msit-`), not `i-` aliases. Explicit ports, trailing-dot hosts, empty labels, IP literals, user information, query strings, fragments, and nonempty paths are rejected. Every authority not described above throws `std::invalid_argument`.

## Security

- The factory accepts only token credentials. OneLake does not use Storage shared keys for this scenario.
- Endpoint matching is case-insensitive and occurs at DNS-label boundaries.
- Every destination produced by the factory and the SDK's built-in DFS/Blob and retry behavior is validated before a credential-bearing client is returned.
- Caller-provided pipeline policies and transports are trusted extension points. They can rewrite requests and are outside the factory's destination guarantee.
- Workspace IDs are validated before inclusion in a host.
- No network request occurs during construction.
- Invalid client inputs and unsafe options throw `std::invalid_argument`, documented in Doxygen.
- `EnableTenantDiscovery` is supported; its challenge request targets the same validated OneLake endpoint.

## Compatibility

The API is additive. The algorithm can be implemented without new dependencies on both current `main` and CDC's pinned SDK commit `1c32f92de06445c467715b55860083202194aee4`, but the unchanged pinned package does not contain the API. CDC must vendor the first released package containing it. Raw URL checks beyond `Azure::Core::Url` are required for the documented grammar.

Adding a public method requires a minor Data Lake package version increment and changelog/API-view review. CDC must bump its SDK submodule after the SDK change is available.

## OneLake Capability Boundary

`DataLakeFileSystemClient` combines DFS-backed and Blob-backed methods. The factory explicitly constructs the documented workspace DFS and Blob companions because the generic SDK mapper does not understand ring-specific labels. `GetUrl()` returns the explicitly constructed workspace Blob URL, consistent with existing Data Lake clients.

OneLake supports both API families but differs from an Azure Storage account. The API documentation and sample promise only OneLake-supported data operations. Workspace creation/deletion, access-policy mutation, ACL mutation, encryption-scope behavior, access tiers, and other Fabric-managed operations remain subject to the documented OneLake API parity restrictions.

## Testing

Offline unit tests cover:

- all four cloud domains;
- base, daily, dxt, and msit rings;
- regional endpoint prefixes;
- compact and hyphenated GUIDs;
- case normalization;
- already workspace-scoped endpoints;
- dedicated endpoints;
- explicit ports;
- malformed workspace IDs;
- userinfo, query, fragment, non-HTTPS, and unsupported domains;
- DNS label-boundary attacks;
- mismatched workspace FQDN IDs and `zXY` labels;
- nonempty secondary retry hosts;
- custom audiences and customer-provided keys;
- null credentials;
- `EnableTenantDiscovery` on and off;
- nonempty service URL paths;
- generated child directory URL encoding;
- exact workspace DFS and Blob hosts.

The table-driven URL matrix includes:

- base and regional global DFS/Blob inputs;
- base and regional general API inputs;
- every canonical ring on global DFS, Blob, and general API inputs;
- every `i-` ring alias canonicalized on shared inputs;
- rejection of `i-` aliases on workspace hosts;
- ring-versus-region precedence;
- unknown valid regional prefixes and malformed prefixes;
- shared/workspace DFS and Blob inputs;
- trailing-dot, empty-label, IP-literal, userinfo, query, fragment, path, and port rejection;
- exact filesystem `GetUrl()` and child `GetUrl()` values;
- tenant-discovery challenge and retry requests remaining on the validated host.

All generated URLs are canonical and omit an explicit port.

Offline tests use a counting credential to prove construction does not request a token. Mock-transport tests exercise one DFS-backed and one Blob-backed operation and assert exact request hosts. Service-owner confirmation and live DFS-backed and Blob-backed contract results for every public cloud-domain/ring combination are release gates; endpoint-construction tests themselves require no live Fabric resource.

## CDC Integration

CDC retains:

- `azure_cdc.onelake_use_workspace_dns` and its default-off rollout control;
- parsing of its destination artifact path;
- choice of `Files` versus `REPIntegration`;
- validation of destination-path structure, artifact roots, dedicated endpoints, and CDC-specific identifiers;
- GUC logging and error reporting.

CDC removes:

- the workspace FQDN formula and validation;
- GUID compaction for hostname generation;
- workspace-host generation and idempotence logic.

When the GUC is on for a shared OneLake endpoint, CDC constructs the workspace file-system client through the SDK factory and derives artifact/directory/file clients through the SDK hierarchy. Recursive upload and rename helpers carry clients plus structured filesystem paths instead of rebuilding child URLs. Rename cleanup uses `DataLakeFileSystemClient::GetFileClient(destinationPath)`. This removes CDC's need to retrieve or reconstruct an effective DFS URL. Dedicated endpoints bypass the factory and retain the existing constructor path. When the GUC is off, CDC continues using its existing URL construction path unchanged.

CDC release-gate tests cover:

- GUC-on shared endpoint routing through the SDK factory;
- GUC-off URL and request behavior unchanged;
- dedicated endpoint bypass;
- `Files` and `REPIntegration` artifact roots;
- recursive upload using child clients;
- rename destination construction and cleanup through structured filesystem paths;
- the existing cloud/ring/GUID matrix delegated to SDK tests rather than duplicated in CDC.

## Upstream Process

This is a new permanent public API and requires Azure Storage and OneLake service-owner review before a non-draft PR. The issue explicitly requests confirmation for API placement, sovereign cloud suffixes, test-ring behavior, and the live contract matrix. The implementation can proceed locally behind offline tests while that review is pending, but the SDK release and CDC integration wait for approval of the complete matrix.

Contributor guide

Open the contributing guide

Research direction

Start with the existing DataLakeFileSystemClient constructors, DFS/Blob URL mapping, and private constructor, then compare the proposed factory behavior with the endpoint and security rules in the issue. Add the described offline table-driven URL and option-validation tests, counting-credential test, and mock-transport checks; done means canonical validated DFS/Blob clients with no construction-time network request.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, cpp
Domain
backend-api-design, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.