airbytehq / airbytehq/airbyte-api-python-sdk

SourceConfiguration Union type deserializes to wrong type (e.g., SourceAirtable for S3 sources)

未關閉
#135 1 則留言 1 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
61
分支
27
平均合併
4 分鐘
30 天內合併 PR
2

描述

## Summary

When calling `get_source()` or similar methods that return a `SourceResponse`, the `configuration` field is incorrectly deserialized to `SourceAirtable` regardless of the actual source type. This prevents users from accessing source-specific configuration fields like `bucket`, `streams`, or `globs` for S3 sources.

## Root Cause

The `SourceConfiguration` type is a Python `Union` of 500+ source configuration types. When deserializing JSON responses:

1. `dataclasses_json` tries each type in the Union in order until one succeeds
2. `SourceAirtable` appears early in the Union (position 8) and has **all optional fields**
3. The `@dataclass_json(undefined=Undefined.EXCLUDE)` decorator ignores unknown fields
4. Since `SourceAirtable` has no required fields, it successfully deserializes ANY JSON payload

The OpenAPI spec uses `oneOf` without a `discriminator` field, so there is no way for the SDK to determine the correct type to deserialize to.

## Reproduction

```python
from airbyte_api import AirbyteAPI
from airbyte_api.models import Security

client = AirbyteAPI(security=Security(bearer_auth="..."))

# Get an S3 source
response = client.sources.get_source(source_id="1f5ca207-5c40-48d6-b9d1-6667de9fe427")
source = response.source_response

print(f"Source type: {source.source_type}") # "s3"
print(f"Config type: {type(source.configuration).__name__}") # "SourceAirtable" (WRONG!)
print(f"Has bucket? {hasattr(source.configuration, 'bucket')}") # False (WRONG!)
```

## Expected Behavior

The `configuration` field should be deserialized to `SourceS3` when `source_type` is `"s3"`.

## Workaround

Users can access the raw configuration dict or manually deserialize to the correct type:

```python
import json
from airbyte_api import utils
from airbyte_api.models import SourceS3

# Option 1: Access raw config dict
raw_config = response.raw_response.json()["configuration"]
print(raw_config["bucket"]) # Works
print(raw_config["streams"][0]["globs"]) # Works

# Option 2: Manually deserialize to correct type
config_json = json.dumps(response.raw_response.json()["configuration"])
s3_config = utils.unmarshal_json(config_json, SourceS3)
print(s3_config.bucket) # Works
print(s3_config.streams[0].globs) # Works
```

## Potential Fix

A proper fix would require either:
1. Adding a `discriminator` field to the OpenAPI spec for `SourceConfiguration` using the `sourceType` property
2. Modifying Speakeasy's generation to handle discriminated unions based on a sibling field

Since this is a generated SDK, any direct code changes would be overwritten on regeneration.

## Context

This issue was reported by a customer trying to modify `streams.globs` at runtime from Airflow. Investigation requested by @iherdt-airbyte.

Related: Commit `87f7e7ba` removed some discriminators from the OpenAPI spec in September 2024.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。