airbytehq / airbytehq/airbyte

[destination-gcs-data-lake][destination-s3-data-lake] Add Polaris-Realm header support for multi-realm deployments

Open
#71,777 2 comments 0 reactions 0 assignees View on GitHub
autoteam connectors/destination/gcs-data-lake connectors/destination/s3-data-lake team/destinations
Dominant language
Python
Stars
22.1k
Forks
5.3k
PR merge metrics
PR metrics pending

Description

## Feature Request

### Connector Names
- destination-gcs-data-lake
- destination-s3-data-lake

### Problem Description

Customers using Apache Polaris with **multi-realm deployments** cannot connect to their Polaris catalog because the `Polaris-Realm` header is not configurable. The connector currently assumes a single-realm deployment (the default "POLARIS" realm).

When attempting to connect, customers receive the error:
```
Could not connect with provided configuration. Error: Unable to process: {"error": {"message": "Missing or invalid realm","type": "MissingOrInvalidRealm","code": 404}}
```

### Background

Apache Polaris supports multi-realm deployments where different realms can isolate catalogs, principals, and access controls. According to the [Polaris documentation](https://polaris.apache.org/releases/1.1.0/configuration/), the `Polaris-Realm` header must be included in API requests to specify which realm to use.

### Current State

The connector test code already uses the realm header internally (see `PolarisEnvironment.kt`), but this is hardcoded to "POLARIS" and not exposed as a user-configurable option.

Current Polaris configuration options:
- `serverUri`
- `catalogName`
- `clientId`
- `clientSecret`
- `scope` (S3 Data Lake only)
- `oauth2ServerUri` (S3 Data Lake only)

### Proposed Solution

Add an optional `realm` field to the Polaris catalog configuration that allows users to specify the `Polaris-Realm` header value. This would involve:

1. **CDK changes** (`airbyte-cdk/bulk/toolkits/load-iceberg-parquet/`):
- Add `realm` field to `PolarisCatalogSpecification`
- Add `realm` field to `PolarisCatalogConfiguration`

2. **Connector changes**:
- Pass the realm as a `header.Polaris-Realm` property in the catalog configuration (similar to how BigLake uses `header.Authorization`)

The Iceberg REST catalog client supports custom headers via the `header.{name}` property pattern, so this should be straightforward to implement.

### Implementation Difficulty

**Low-Medium effort** - The infrastructure for custom headers already exists in the Iceberg REST catalog client.

### Related Issues

- https://github.com/airbytehq/airbyte/issues/50397 - OAuth2 scope parameter support (similar Polaris configuration enhancement)

### Additional Context

This feature request was created based on Zendesk ticket #15902.

Requested by: Vai Ignatavicius (TSE)

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/10929

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.