[destination-gcs-data-lake][destination-s3-data-lake] Add Polaris-Realm header support for multi-realm deployments
- Dominant language
- Python
- Stars
- 22.1k
- Forks
- 5.3k
- PR merge metrics
- PR metrics pending
Description
## Feature Request
### Connector Names
- destination-gcs-data-lake
- destination-s3-data-lake
### Problem Description
Customers using Apache Polaris with **multi-realm deployments** cannot connect to their Polaris catalog because the `Polaris-Realm` header is not configurable. The connector currently assumes a single-realm deployment (the default "POLARIS" realm).
When attempting to connect, customers receive the error:
```
Could not connect with provided configuration. Error: Unable to process: {"error": {"message": "Missing or invalid realm","type": "MissingOrInvalidRealm","code": 404}}
```
### Background
Apache Polaris supports multi-realm deployments where different realms can isolate catalogs, principals, and access controls. According to the [Polaris documentation](https://polaris.apache.org/releases/1.1.0/configuration/), the `Polaris-Realm` header must be included in API requests to specify which realm to use.
### Current State
The connector test code already uses the realm header internally (see `PolarisEnvironment.kt`), but this is hardcoded to "POLARIS" and not exposed as a user-configurable option.
Current Polaris configuration options:
- `serverUri`
- `catalogName`
- `clientId`
- `clientSecret`
- `scope` (S3 Data Lake only)
- `oauth2ServerUri` (S3 Data Lake only)
### Proposed Solution
Add an optional `realm` field to the Polaris catalog configuration that allows users to specify the `Polaris-Realm` header value. This would involve:
1. **CDK changes** (`airbyte-cdk/bulk/toolkits/load-iceberg-parquet/`):
- Add `realm` field to `PolarisCatalogSpecification`
- Add `realm` field to `PolarisCatalogConfiguration`
2. **Connector changes**:
- Pass the realm as a `header.Polaris-Realm` property in the catalog configuration (similar to how BigLake uses `header.Authorization`)
The Iceberg REST catalog client supports custom headers via the `header.{name}` property pattern, so this should be straightforward to implement.
### Implementation Difficulty
**Low-Medium effort** - The infrastructure for custom headers already exists in the Iceberg REST catalog client.
### Related Issues
- https://github.com/airbytehq/airbyte/issues/50397 - OAuth2 scope parameter support (similar Polaris configuration enhancement)
### Additional Context
This feature request was created based on Zendesk ticket #15902.
Requested by: Vai Ignatavicius (TSE)
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/10929
Contributor guide
Assessment
This issue has not been assessed yet.