airbytehq / airbytehq/airbyte-python-cdk

Declarative: Pattern-based partition routing for dynamic repository discovery

Aperta
#838 0 commenti 0 reazioni 1 assegnatario Rivendicata da @aaronsteers Vedi su GitHub
manifest-only feature gaps
Lingua principale
Python
Stelle
26
Fork
53
Merge medio
2g 6h
PR unite (30g)
10

Descrizione

## Problem

Some connectors need to dynamically discover resources based on wildcard patterns (e.g., `airbytehq/*` to sync all repositories in an organization). Currently, this requires custom Python code to expand patterns into actual resource lists, preventing connectors from being fully declarative (manifest-only).

## Current Workaround

Connectors like `source-github` implement custom logic to expand wildcard patterns. See [`source-github/source.py:215-313`](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github/source_github/source.py#L215-L313) which implements `_get_org_repositories()` to:
- Parse patterns like `airbytehq/*` or `airbytehq/a*`
- Call GitHub API to list all repositories in the organization
- Filter repositories by pattern matching
- Return expanded list of repositories

## Proposed Solution

Add a declarative `PatternPartitionRouter` component that supports:
1. **Pattern expansion**: Accept wildcard patterns and expand them via API calls
2. **API-based discovery**: Call an API endpoint to list available resources
3. **Pattern matching**: Filter resources by glob/regex patterns
4. **Caching**: Cache expanded patterns to avoid repeated API calls

## Example Configuration

```yaml
partition_router:
type: PatternPartitionRouter
pattern_field: "repositories" # Config field containing patterns
discovery_requester:
type: HttpRequester
url_base: "https://api.github.com"
path: "/orgs/{{ pattern.org }}/repos"
authenticator:
type: BearerAuthenticator
api_token: "{{ config.access_token }}"
pattern_parser:
type: GlobPattern
separator: "/"
wildcard: "*"
record_selector:
extractor:
field_path: []
partition_fields:
- owner: "{{ item.owner.login }}"
- name: "{{ item.name }}"
```

## Example Use Cases

**Input patterns:**
- `airbytehq/*` → Expands to all repos in airbytehq org
- `airbytehq/airbyte` → Single specific repo
- `airbytehq/a*` → All repos starting with 'a' in airbytehq org
- `org1/repo1 org2/*` → Multiple patterns

**Expanded partitions:**
```
[
{"owner": "airbytehq", "name": "airbyte"},
{"owner": "airbytehq", "name": "airbyte-platform"},
{"owner": "airbytehq", "name": "airbyte-python-cdk"},
...
]
```

## Impact

This would enable connectors like `source-github`, `source-gitlab`, and others to support wildcard patterns declaratively without custom Python code.

## Related

- airbytehq/airbyte-python-cdk#714 - Survey showing 74.3% of certified manifest-only connectors require custom components
- Requested by @aaronsteers in https://airbytehq-team.slack.com/archives/C08BHPUMEPJ/p1762665961303909

## Code References

- [`source-github/spec.json:80-96`](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github/source_github/spec.json#L80-L96) - Repository pattern configuration
- [`source-github/source.py:215-313`](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github/source_github/source.py#L215-L313) - `_get_org_repositories()` pattern expansion logic

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.