airbytehq / airbytehq/airbyte-python-cdk

Declarative: Multi-token authenticator with rate-limit-aware rotation

Đang mở
#835 0 bình luận 0 reaction 1 người được giao Được @aaronsteers nhận Xem trên GitHub
manifest-only feature gaps
Ngôn ngữ chính
Python
Star
26
Fork
53
Merge trung bình
2 ngày 6 giờ
Pull request đã merge (30 ngày)
10

Mô tả

## Problem

Many API connectors need to rotate through multiple authentication tokens to increase effective rate limits and avoid hitting API quotas. Currently, this requires custom Python code and cannot be implemented in declarative manifest-only connectors.

## Current Workaround

Connectors like `source-github` implement custom authenticators in Python. See [`source-github/utils.py:53-186`](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github/source_github/utils.py#L53-L186) which implements `MultipleTokenAuthenticatorWithRateLimiter` that:
- Cycles through multiple tokens automatically
- Tracks separate rate limits per token per API type (REST vs GraphQL)
- Automatically switches tokens when limits are reached
- Sleeps until reset if all tokens are exhausted

## Proposed Solution

Add a declarative `MultiTokenAuthenticator` component that supports:
1. **Token rotation**: Accept multiple tokens and cycle through them
2. **Rate limit tracking**: Monitor rate limits per token via response headers or dedicated endpoints
3. **Separate counters**: Support different rate limit buckets (e.g., REST vs GraphQL)
4. **Automatic switching**: Switch to next token when current token is rate-limited
5. **Sleep/retry logic**: Wait until rate limit reset when all tokens are exhausted

## Example Configuration

```yaml
authenticator:
type: MultiTokenAuthenticator
tokens:
- "{{ config.tokens[0] }}"
- "{{ config.tokens[1] }}"
- "{{ config.tokens[2] }}"
rate_limit_tracking:
- resource: "core" # REST API
header_prefix: "x-ratelimit-"
- resource: "graphql" # GraphQL API
header_prefix: "x-ratelimit-"
max_wait_time: "{{ config.max_waiting_time }}"
```

## Impact

This would enable connectors like `source-github`, `source-gitlab`, and others to be fully declarative while maintaining their multi-token rate limiting capabilities.

## Related

- airbytehq/airbyte-python-cdk#714 - Survey showing 74.3% of certified manifest-only connectors require custom components
- Requested by @aaronsteers in https://airbytehq-team.slack.com/archives/C08BHPUMEPJ/p1762665961303909

## Code References

- [`source-github/utils.py:53-186`](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github/source_github/utils.py#L53-L186) - `MultipleTokenAuthenticatorWithRateLimiter` implementation
- [`source-github/spec.json:57-62`](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github/source_github/spec.json#L57-L62) - Multi-token input configuration

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.