airbytehq / airbytehq/PyAirbyte

feat: PyAirbyte-powered Smoke Test Source for Destination Regression Testing

未关闭
#981 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
344
派生
77
平均合并
1 天 11 小时
30 天内合并 PR
35

描述

## Summary

Create a **PyAirbyte-powered Smoke Test Source** connector designed specifically for destination regression testing. This should be implemented as a stacked PR on top of #969 (universal source/destination).

## Context

We want to test destinations the same way we test sources today -- in a regression test framework that has **no side effects and no customer impact**. The Smoke Test Source is the first building block of this framework.

Full Destination Regression Test Plan

### Architecture Overview

#### 1. Smoke Test Source (This Issue)

A purpose-built source connector that generates test data for destinations:

- **Predefined scenarios:** Ships with a curated set of data patterns known to commonly cause destination failures (edge-case types, schema variations, etc.)
- **Dynamic injection:** Supports dynamically adding new failure conditions discovered from production issues. When AI Triage identifies a customer-facing failure, an **anonymized failure condition** is generated from the error log and added to the smoke test source's scenario set.

#### 2. Test Harness

Orchestrates the end-to-end regression test:

1. **Record Injection** -- Feeds test scenarios from the Smoke Test Source into the destination under test.
2. **Destination Write** -- The destination connector processes the injected records.
3. **Destination Describe** -- Reads back the destination's state/schema. *(Today, destinations cannot describe their own data. The code for this exists in integration tests but must be promoted into the connector's main dependencies so it ships with the connector.)*
4. **Data Dumper** -- Extracts the written data from destination storage for comparison. *(Same as Describe -- must be moved from integration test code into the shipping connector.)*

#### 3. Control Test (Two-Version Comparison)

Every regression test runs against **two versions** of the destination connector (current release vs. candidate):

- Catches regressions and unexpected behavioral differences even when no known failure exists.
- Produces a **Diff Report** comparing the two runs.

#### 4. Diff Report (Per Table)

| Check | Description |
|---|---|
| **Dest Schema / Types** | Schema and column type correctness |
| **Record Count** | Expected vs. actual row counts |
| **Record Data / Null Handling** | Data integrity, especially null value handling |
| **Table Name** | Correct table naming |
| **Column Name + Casing** | Column names and case-sensitivity preservation |
| **Spec Output Diff** | Differences in connector spec output between versions |

#### 5. TDD Workflow (AI-Assisted)

1. **AI Triage** identifies a destination failure from production error logs.
2. An **anonymized failure condition** is generated and injected into the Smoke Test Source.
3. The regression test now has a **failing test case** for the specific destination.
4. A PR is opened, and **TDD** proceeds -- iterating until the test passes.
5. (`ai-prove-fix`) The fix is validated by the regression framework before merge.

#### Out of Scope

- Speed Mode (performance optimization of the test harness).

## Scope of This Issue

Build the Smoke Test Source as a PyAirbyte-powered connector, stacking on the universal source/destination work in #969. Specifically:

### Requirements

1. **Predefined test scenarios** -- A curated set of data patterns that commonly break destinations:
- Various column types (including edge cases like timestamps, large decimals, nested JSON, etc.)
- Null handling edge cases
- Table/column naming edge cases (special characters, casing, long names, reserved words)
- Schema variations and evolution scenarios
- Large and small record batches

2. **Dynamic scenario injection** -- Ability to add new failure scenarios at runtime (e.g., from a config file or catalog entry), so that newly discovered production failures can be reproduced without code changes.

3. **Standard Airbyte Source interface** -- Implements `spec()`, `check()`, `discover()`, `read()` per the CDK, so it can be used with any destination via the standard Airbyte protocol.

4. **Anonymized failure reproduction** -- Support for defining scenarios based on anonymized production error patterns (no real customer data).

### Relationship to #969

This should be built on top of the universal connector infrastructure from #969, reusing its patterns for CLI entrypoints, Dockerfile, and PyAirbyte integration.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。