airbytehq / airbytehq/airbyte-python-cdk

Declarative: Pattern-based partition routing for dynamic repository discovery

Abierto
#838 0 comentarios 0 reacciones 1 asignado Reclamado por @aaronsteers Ver en GitHub
manifest-only feature gaps
Lenguaje dominante
Python
Estrellas
26
Forks
53
Merge medio
2 d 6 h
PR fusionados (30 d)
10

Descripción

## Problem

Some connectors need to dynamically discover resources based on wildcard patterns (e.g., `airbytehq/*` to sync all repositories in an organization). Currently, this requires custom Python code to expand patterns into actual resource lists, preventing connectors from being fully declarative (manifest-only).

## Current Workaround

Connectors like `source-github` implement custom logic to expand wildcard patterns. See [`source-github/source.py:215-313`](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github/source_github/source.py#L215-L313) which implements `_get_org_repositories()` to:
- Parse patterns like `airbytehq/*` or `airbytehq/a*`
- Call GitHub API to list all repositories in the organization
- Filter repositories by pattern matching
- Return expanded list of repositories

## Proposed Solution

Add a declarative `PatternPartitionRouter` component that supports:
1. **Pattern expansion**: Accept wildcard patterns and expand them via API calls
2. **API-based discovery**: Call an API endpoint to list available resources
3. **Pattern matching**: Filter resources by glob/regex patterns
4. **Caching**: Cache expanded patterns to avoid repeated API calls

## Example Configuration

```yaml
partition_router:
type: PatternPartitionRouter
pattern_field: "repositories" # Config field containing patterns
discovery_requester:
type: HttpRequester
url_base: "https://api.github.com"
path: "/orgs/{{ pattern.org }}/repos"
authenticator:
type: BearerAuthenticator
api_token: "{{ config.access_token }}"
pattern_parser:
type: GlobPattern
separator: "/"
wildcard: "*"
record_selector:
extractor:
field_path: []
partition_fields:
- owner: "{{ item.owner.login }}"
- name: "{{ item.name }}"
```

## Example Use Cases

**Input patterns:**
- `airbytehq/*` → Expands to all repos in airbytehq org
- `airbytehq/airbyte` → Single specific repo
- `airbytehq/a*` → All repos starting with 'a' in airbytehq org
- `org1/repo1 org2/*` → Multiple patterns

**Expanded partitions:**
```
[
{"owner": "airbytehq", "name": "airbyte"},
{"owner": "airbytehq", "name": "airbyte-platform"},
{"owner": "airbytehq", "name": "airbyte-python-cdk"},
...
]
```

## Impact

This would enable connectors like `source-github`, `source-gitlab`, and others to support wildcard patterns declaratively without custom Python code.

## Related

- airbytehq/airbyte-python-cdk#714 - Survey showing 74.3% of certified manifest-only connectors require custom components
- Requested by @aaronsteers in https://airbytehq-team.slack.com/archives/C08BHPUMEPJ/p1762665961303909

## Code References

- [`source-github/spec.json:80-96`](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github/source_github/spec.json#L80-L96) - Repository pattern configuration
- [`source-github/source.py:215-313`](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github/source_github/source.py#L215-L313) - `_get_org_repositories()` pattern expansion logic

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.