airbytehq / airbytehq/PyAirbyte
feat: PyAirbyte-powered Smoke Test Source for Destination Regression Testing
- 主要语言
- Python
- 星标
- 344
- 派生
- 77
- 平均合并
- 1 天 11 小时
- 30 天内合并 PR
- 35
描述
## Summary
Create a **PyAirbyte-powered Smoke Test Source** connector designed specifically for destination regression testing. This should be implemented as a stacked PR on top of #969 (universal source/destination).
## Context
We want to test destinations the same way we test sources today -- in a regression test framework that has **no side effects and no customer impact**. The Smoke Test Source is the first building block of this framework.
Full Destination Regression Test Plan
### Architecture Overview
#### 1. Smoke Test Source (This Issue)
A purpose-built source connector that generates test data for destinations:
- **Predefined scenarios:** Ships with a curated set of data patterns known to commonly cause destination failures (edge-case types, schema variations, etc.)
- **Dynamic injection:** Supports dynamically adding new failure conditions discovered from production issues. When AI Triage identifies a customer-facing failure, an **anonymized failure condition** is generated from the error log and added to the smoke test source's scenario set.
#### 2. Test Harness
Orchestrates the end-to-end regression test:
1. **Record Injection** -- Feeds test scenarios from the Smoke Test Source into the destination under test.
2. **Destination Write** -- The destination connector processes the injected records.
3. **Destination Describe** -- Reads back the destination's state/schema. *(Today, destinations cannot describe their own data. The code for this exists in integration tests but must be promoted into the connector's main dependencies so it ships with the connector.)*
4. **Data Dumper** -- Extracts the written data from destination storage for comparison. *(Same as Describe -- must be moved from integration test code into the shipping connector.)*
#### 3. Control Test (Two-Version Comparison)
Every regression test runs against **two versions** of the destination connector (current release vs. candidate):
- Catches regressions and unexpected behavioral differences even when no known failure exists.
- Produces a **Diff Report** comparing the two runs.
#### 4. Diff Report (Per Table)
| Check | Description |
|---|---|
| **Dest Schema / Types** | Schema and column type correctness |
| **Record Count** | Expected vs. actual row counts |
| **Record Data / Null Handling** | Data integrity, especially null value handling |
| **Table Name** | Correct table naming |
| **Column Name + Casing** | Column names and case-sensitivity preservation |
| **Spec Output Diff** | Differences in connector spec output between versions |
#### 5. TDD Workflow (AI-Assisted)
1. **AI Triage** identifies a destination failure from production error logs.
2. An **anonymized failure condition** is generated and injected into the Smoke Test Source.
3. The regression test now has a **failing test case** for the specific destination.
4. A PR is opened, and **TDD** proceeds -- iterating until the test passes.
5. (`ai-prove-fix`) The fix is validated by the regression framework before merge.
#### Out of Scope
- Speed Mode (performance optimization of the test harness).
## Scope of This Issue
Build the Smoke Test Source as a PyAirbyte-powered connector, stacking on the universal source/destination work in #969. Specifically:
### Requirements
1. **Predefined test scenarios** -- A curated set of data patterns that commonly break destinations:
- Various column types (including edge cases like timestamps, large decimals, nested JSON, etc.)
- Null handling edge cases
- Table/column naming edge cases (special characters, casing, long names, reserved words)
- Schema variations and evolution scenarios
- Large and small record batches
2. **Dynamic scenario injection** -- Ability to add new failure scenarios at runtime (e.g., from a config file or catalog entry), so that newly discovered production failures can be reproduced without code changes.
3. **Standard Airbyte Source interface** -- Implements `spec()`, `check()`, `discover()`, `read()` per the CDK, so it can be used with any destination via the standard Airbyte protocol.
4. **Anonymized failure reproduction** -- Support for defining scenarios based on anonymized production error patterns (no real customer data).
### Relationship to #969
This should be built on top of the universal connector infrastructure from #969, reusing its patterns for CLI entrypoints, Dockerfile, and PyAirbyte integration.
贡献指南
评估
这个 Issue 还没有评估数据。