airbytehq / airbytehq/airbyte-python-cdk
Declarative: Pattern-based partition routing for dynamic repository discovery
- 主要語言
- Python
- 星號
- 26
- 分支
- 53
- 平均合併
- 2 天 6 小時
- 30 天內合併 PR
- 10
描述
## Problem
Some connectors need to dynamically discover resources based on wildcard patterns (e.g., `airbytehq/*` to sync all repositories in an organization). Currently, this requires custom Python code to expand patterns into actual resource lists, preventing connectors from being fully declarative (manifest-only).
## Current Workaround
Connectors like `source-github` implement custom logic to expand wildcard patterns. See [`source-github/source.py:215-313`](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github/source_github/source.py#L215-L313) which implements `_get_org_repositories()` to:
- Parse patterns like `airbytehq/*` or `airbytehq/a*`
- Call GitHub API to list all repositories in the organization
- Filter repositories by pattern matching
- Return expanded list of repositories
## Proposed Solution
Add a declarative `PatternPartitionRouter` component that supports:
1. **Pattern expansion**: Accept wildcard patterns and expand them via API calls
2. **API-based discovery**: Call an API endpoint to list available resources
3. **Pattern matching**: Filter resources by glob/regex patterns
4. **Caching**: Cache expanded patterns to avoid repeated API calls
## Example Configuration
```yaml
partition_router:
type: PatternPartitionRouter
pattern_field: "repositories" # Config field containing patterns
discovery_requester:
type: HttpRequester
url_base: "https://api.github.com"
path: "/orgs/{{ pattern.org }}/repos"
authenticator:
type: BearerAuthenticator
api_token: "{{ config.access_token }}"
pattern_parser:
type: GlobPattern
separator: "/"
wildcard: "*"
record_selector:
extractor:
field_path: []
partition_fields:
- owner: "{{ item.owner.login }}"
- name: "{{ item.name }}"
```
## Example Use Cases
**Input patterns:**
- `airbytehq/*` → Expands to all repos in airbytehq org
- `airbytehq/airbyte` → Single specific repo
- `airbytehq/a*` → All repos starting with 'a' in airbytehq org
- `org1/repo1 org2/*` → Multiple patterns
**Expanded partitions:**
```
[
{"owner": "airbytehq", "name": "airbyte"},
{"owner": "airbytehq", "name": "airbyte-platform"},
{"owner": "airbytehq", "name": "airbyte-python-cdk"},
...
]
```
## Impact
This would enable connectors like `source-github`, `source-gitlab`, and others to support wildcard patterns declaratively without custom Python code.
## Related
- airbytehq/airbyte-python-cdk#714 - Survey showing 74.3% of certified manifest-only connectors require custom components
- Requested by @aaronsteers in https://airbytehq-team.slack.com/archives/C08BHPUMEPJ/p1762665961303909
## Code References
- [`source-github/spec.json:80-96`](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github/source_github/spec.json#L80-L96) - Repository pattern configuration
- [`source-github/source.py:215-313`](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github/source_github/source.py#L215-L313) - `_get_org_repositories()` pattern expansion logic
貢獻指南
評估
這個 Issue 還沒有評估資料。