airbytehq / airbytehq/airbyte

[source-redshift] DATE and TIMESTAMP columns discovered as plain String (missing JSON Schema format hints)

オープン
#83,809 コメント 2 件 リアクション 0 件 担当者 0 名 GitHub で見る
autoteam community connectors/destination/postgres connectors/source/redshift team/use
主要言語
Python
スター
22.1k
フォーク
5.4k
平均マージ
5時間
マージ済み PR(30日)
671

説明

## Environment
- Connector: `source-redshift`
- Stack: Legacy JDBC CDK (`AbstractJdbcSource` / `JdbcSourceOperations`)

## Current Behavior
Given a Redshift table:

```sql
CREATE TABLE invoices (
invoice_date DATE,
due_date DATE,
created_at TIMESTAMP,
amount NUMERIC,
invoice_id INTEGER,
....
);
```

Schema discovery produces:

| Column | Discovered Airbyte type |
|--------|-------------------------|
| `invoice_date` | String |
| `due_date` | String |
| `created_at` | String |
| `amount` | Number |
| `invoice_id` | Integer |

Image

Catalog JSON Schema for the date/timestamp columns is effectively `{"type":"string"}` with **no** `format` (`date` / `date-time`).

Downstream, destinations that honor semantic types (e.g. `destination-postgres`) therefore create `VARCHAR` columns instead of `DATE` / `TIMESTAMP`.

Redshift metadata itself is correct (`DATE` / `TIMESTAMP`). The destination mapping is also correct when given format-aware schema types. The loss happens during **source discovery**.

## Expected Behavior
Discovery should emit semantic Airbyte temporal types, e.g.:

| Column | Expected type |
|--------|----------------|
| `invoice_date` | Date (`JsonSchemaType.STRING_DATE`, `format=date`) |
| `due_date` | Date |
| `created_at` | Timestamp without timezone (`STRING_TIMESTAMP_WITHOUT_TIMEZONE`, `format=date-time`) |

So destinations can create native temporal columns.

## Root Cause
`RedshiftSourceOperations` extends `JdbcSourceOperations` but does **not** override `getAirbyteType`.

The legacy JDBC CDK default maps temporal JDBC types to plain string:

```kotlin
// airbyte-cdk/.../JdbcSourceOperations.kt
JDBCType.DATE -> JsonSchemaType.STRING
JDBCType.TIME -> JsonSchemaType.STRING
JDBCType.TIMESTAMP -> JsonSchemaType.STRING
```

Discovery path:

```
RedshiftSource → AbstractJdbcSource / AbstractDbSource.discover
→ discoverInternal (JDBC metadata → JDBCType on CommonField) // still correct
→ DbSourceDiscoverUtil.convertTableInfosToAirbyteCatalog
→ sourceOperations.getAirbyteType(JDBCType) // ★ semantic info lost here
→ AirbyteCatalog
```

`RedshiftSourceOperations` only customizes **value** serialization (`putTimestamp`, `copyToJsonField`, etc.), not catalog type mapping. Git history under `source-redshift` has no prior `getAirbyteType` / `STRING_DATE` mapping — this appears to be an omission.

## Comparison with other connectors
| Connector | Temporal discover mapping |
|-----------|---------------------------|
| **ClickHouse** | Overrides `getAirbyteType` → `STRING_DATE` / `STRING_TIMESTAMP_*` (fixed in #72484 / #72483) |
| **DB2 / SingleStore** | Same override pattern |
| **MySQL / Postgres** | Bulk CDK FieldType mappers → `DATE` / `TIMESTAMP_*` |
| **Redshift** | Inherits CDK default → plain `STRING` |

Note: changing the CDK default was briefly tried historically and reverted (#7859 / #7969), so a **connector-local** override is the established fix pattern.

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Read RedshiftSourceOperations alongside airbyte-cdk/.../JdbcSourceOperations.kt, starting at getAirbyteType and the existing temporal mappings. Verify discovery through DbSourceDiscoverUtil.convertTableInfosToAirbyteCatalog and confirm that DATE and TIMESTAMP columns produce format-aware catalog schemas rather than plain strings.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
kotlin
領域
databases
issue の種類
バグ
難易度
3/5
見積もり時間
1〜2日
活発さ
静か
明瞭さ
明確に書かれている
初心者へのやさしさ
82/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。