airbytehq / airbytehq/airbyte

GCS Source and Destination connectors use inconsistent authentication methods

Ouverte
#71,828 4 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
autoteam connectors/destination/gcs connectors/source/gcs team/db-dw-sources
Langage dominant
Python
Étoiles
22.1k
Forks
5.3k
Métriques de merge des PR
Métriques de PR en attente

Description

## Summary

The GCS source and destination connectors use different authentication approaches, creating an inconsistent user experience. Users must create different types of credentials depending on whether they're reading from or writing to GCS.

**Requested by:** @tgonzalezc5 (teo@airbyte.io)

## Current State

### GCS Source Connector (Python-based)
Supports **two** authentication methods:
- **OAuth** (Authenticate via Google) - uses client_id, client_secret, access_token, refresh_token
- **Service Account JSON key** - uses the full JSON key file content

The source uses the native Google Cloud Storage Python client library (`google-cloud-storage`).

**Relevant files:**
- `airbyte-integrations/connectors/source-gcs/source_gcs/config.py` - defines `OAuthCredentials` and `ServiceAccountCredentials` classes
- `airbyte-integrations/connectors/source-gcs/source_gcs/stream_reader.py` - implements `_get_credentials()` method handling both auth types

### GCS Destination Connector (Java/Kotlin-based)
Supports **only one** authentication method:
- **HMAC keys** - uses hmac_key_access_id + hmac_key_secret

The destination uses the AWS S3 SDK with GCS's S3-compatible endpoint (`https://storage.googleapis.com`).

**Relevant files:**
- `airbyte-integrations/connectors/destination-gcs/src/main/resources/spec.json` - defines only HMAC_KEY credential type (lines 72-109)
- `airbyte-cdk/java/airbyte-cdk/gcs-destinations/src/main/kotlin/io/airbyte/cdk/integrations/destination/gcs/GcsDestinationConfig.kt` - creates S3 client with HMAC credentials
- `airbyte-cdk/java/airbyte-cdk/gcs-destinations/src/main/kotlin/io/airbyte/cdk/integrations/destination/gcs/credential/` - credential configuration classes

## Problem

1. **Inconsistent UX**: Users need to create different credential types for source vs destination
2. **Non-standard approach**: HMAC keys are primarily intended for S3 migration scenarios, not as the primary GCP auth method
3. **Documentation mismatch**: [Source docs](https://docs.airbyte.com/integrations/sources/gcs) describe Service Account setup, while [Destination docs](https://docs.airbyte.com/integrations/destinations/gcs) describe HMAC setup

## GCP Best Practices

According to [Google Cloud documentation](https://cloud.google.com/storage/docs/authentication/hmackeys):
- **Service Account keys (JSON)** are the standard for server-to-server authentication
- **HMAC keys** are specifically designed for S3-compatible API access and migration scenarios from other cloud providers
- **OAuth** is recommended for user-facing applications

Google recommends using native GCP authentication (Service Account or OAuth) over HMAC when possible.

## Recommended Solution

Add Service Account JSON key authentication to the GCS destination connector to match the source connector's capabilities:

### Option A: Add Service Account support alongside HMAC (Recommended)
1. Add Service Account JSON key as a new authentication option in the destination spec
2. Keep HMAC keys for backward compatibility and S3 migration scenarios
3. This would require either:
- Switching to the native Google Cloud Storage Java client library, OR
- Using the GCS JSON API with service account credentials

### Option B: Full parity with source
Add both Service Account and OAuth support to the destination, providing complete feature parity with the source connector.

## Affected Components

### Files requiring changes:
1. `airbyte-integrations/connectors/destination-gcs/src/main/resources/spec.json`
- Add new credential type options (Service Account, potentially OAuth)

2. `airbyte-cdk/java/airbyte-cdk/gcs-destinations/src/main/kotlin/io/airbyte/cdk/integrations/destination/gcs/GcsDestinationConfig.kt`
- Handle new credential types in `createS3Client()` method
- May need to switch from AWS S3 SDK to native GCS client

3. `airbyte-cdk/java/airbyte-cdk/gcs-destinations/src/main/kotlin/io/airbyte/cdk/integrations/destination/gcs/credential/`
- Add new credential config classes for Service Account (and OAuth if implementing Option B)

4. `airbyte-cdk/java/airbyte-cdk/gcs-destinations/build.gradle` (or equivalent)
- Add Google Cloud Storage Java client library dependency if switching from S3 SDK

### Documentation:
- `docs/integrations/destinations/gcs.md` - Update setup guide to include Service Account instructions

## Additional Context

The destination's current approach of using the AWS S3 SDK with GCS's S3-compatible endpoint was likely chosen for code reuse with the S3 destination. However, this architectural decision has led to the authentication inconsistency.

## Related Issues
- [#27664](https://github.com/airbytehq/airbyte/issues/27664) - Destination Google Cloud Storage (GCS): testing against AWS S3

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/10951

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.