[source-redshift] DATE and TIMESTAMP columns discovered as plain String (missing JSON Schema format hints)
- 主要言語
- Python
- スター
- 22.1k
- フォーク
- 5.4k
- 平均マージ
- 5時間
- マージ済み PR(30日)
- 671
説明
## Environment
- Connector: `source-redshift`
- Stack: Legacy JDBC CDK (`AbstractJdbcSource` / `JdbcSourceOperations`)
## Current Behavior
Given a Redshift table:
```sql
CREATE TABLE invoices (
invoice_date DATE,
due_date DATE,
created_at TIMESTAMP,
amount NUMERIC,
invoice_id INTEGER,
....
);
```
Schema discovery produces:
| Column | Discovered Airbyte type |
|--------|-------------------------|
| `invoice_date` | String |
| `due_date` | String |
| `created_at` | String |
| `amount` | Number |
| `invoice_id` | Integer |
Catalog JSON Schema for the date/timestamp columns is effectively `{"type":"string"}` with **no** `format` (`date` / `date-time`).
Downstream, destinations that honor semantic types (e.g. `destination-postgres`) therefore create `VARCHAR` columns instead of `DATE` / `TIMESTAMP`.
Redshift metadata itself is correct (`DATE` / `TIMESTAMP`). The destination mapping is also correct when given format-aware schema types. The loss happens during **source discovery**.
## Expected Behavior
Discovery should emit semantic Airbyte temporal types, e.g.:
| Column | Expected type |
|--------|----------------|
| `invoice_date` | Date (`JsonSchemaType.STRING_DATE`, `format=date`) |
| `due_date` | Date |
| `created_at` | Timestamp without timezone (`STRING_TIMESTAMP_WITHOUT_TIMEZONE`, `format=date-time`) |
So destinations can create native temporal columns.
## Root Cause
`RedshiftSourceOperations` extends `JdbcSourceOperations` but does **not** override `getAirbyteType`.
The legacy JDBC CDK default maps temporal JDBC types to plain string:
```kotlin
// airbyte-cdk/.../JdbcSourceOperations.kt
JDBCType.DATE -> JsonSchemaType.STRING
JDBCType.TIME -> JsonSchemaType.STRING
JDBCType.TIMESTAMP -> JsonSchemaType.STRING
```
Discovery path:
```
RedshiftSource → AbstractJdbcSource / AbstractDbSource.discover
→ discoverInternal (JDBC metadata → JDBCType on CommonField) // still correct
→ DbSourceDiscoverUtil.convertTableInfosToAirbyteCatalog
→ sourceOperations.getAirbyteType(JDBCType) // ★ semantic info lost here
→ AirbyteCatalog
```
`RedshiftSourceOperations` only customizes **value** serialization (`putTimestamp`, `copyToJsonField`, etc.), not catalog type mapping. Git history under `source-redshift` has no prior `getAirbyteType` / `STRING_DATE` mapping — this appears to be an omission.
## Comparison with other connectors
| Connector | Temporal discover mapping |
|-----------|---------------------------|
| **ClickHouse** | Overrides `getAirbyteType` → `STRING_DATE` / `STRING_TIMESTAMP_*` (fixed in #72484 / #72483) |
| **DB2 / SingleStore** | Same override pattern |
| **MySQL / Postgres** | Bulk CDK FieldType mappers → `DATE` / `TIMESTAMP_*` |
| **Redshift** | Inherits CDK default → plain `STRING` |
Note: changing the CDK default was briefly tried historically and reverted (#7859 / #7969), so a **connector-local** override is the established fix pattern.
コントリビューションガイド
調査の方向性
Read RedshiftSourceOperations alongside airbyte-cdk/.../JdbcSourceOperations.kt, starting at getAirbyteType and the existing temporal mappings. Verify discovery through DbSourceDiscoverUtil.convertTableInfosToAirbyteCatalog and confirm that DATE and TIMESTAMP columns produce format-aware catalog schemas rather than plain strings.
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- kotlin
- 領域
- databases
- issue の種類
- バグ
- 難易度
- 3/5
- 見積もり時間
- 1〜2日
- 活発さ
- 静か
- 明瞭さ
- 明確に書かれている
- 初心者へのやさしさ
- 82/100