source-github: Implement early termination for date-filtered syncs on multiple streams
- Vorherrschende Sprache
- Python
- Sterne
- 22.1k
- Forks
- 5.3k
- PR-Merge-Kennzahlen
- PR-Kennzahlen ausstehend
Beschreibung
## Summary
Several streams in source-github can be extremely slow when syncing with a `start_date` filter, particularly for repositories with large histories. This is because the underlying GitHub APIs do not support server-side date filtering, requiring the connector to paginate through the entire history and filter client-side.
## Problem
The following streams use the `SemiIncrementalMixin` pattern which fetches all records and filters client-side:
| Stream | API Type | Volume Potential | Notes |
|--------|----------|------------------|-------|
| `pull_requests` | REST | High | No `since` parameter available |
| `pull_request_stats` | GraphQL | High | Same data as pull_requests, GraphQL version |
| `issue_events` | REST | Very High | Can be massive for active repositories |
| `stargazers` | REST | Very High | Popular repos can have 100k+ stars |
| `commit_comments` | REST | Moderate | No date filter support |
| `events` | REST | High | No date filter support |
| `deployments` | REST | Moderate | No date filter support |
| `projects` | REST | Low | No date filter support |
| `releases` | REST | Low | Usually small volume |
For example, the `pull_requests` stream on a repository with 15,000+ PRs requires 150+ API calls (100 per page) before any records are output, even when the user only wants data from the last 30 days.
**Streams NOT affected** (already have API-level date filtering via `since` parameter):
- `issues`, `comments`, `commits`, `review_comments` - These use `IncrementalMixin` which passes `since` to the API
**Already optimized:**
- `issue_milestones` - Uses `is_sorted = "desc"` with early termination
- `workflow_runs` - Has custom early termination logic based on 32-day re-run window
## Proposed Solution
Implement early termination for affected streams when `start_date` is configured:
1. Use descending order (`direction: desc`) to fetch most recently updated records first
2. Stop pagination when encountering records with cursor field older than `start_date`
3. This would reduce API calls from hundreds to potentially just a few pages for date-filtered syncs
The existing `SemiIncrementalMixin` already has early termination logic for descending order (line 345-346 in streams.py):
```python
elif self.is_sorted == "desc" and cursor_value < start_point:
break
```
The enhancement would extend this behavior to initial syncs when `start_date` is configured, similar to how `issue_milestones` already works.
## Alternative Considered: GraphQL Search API
For `pull_requests` specifically, GitHub's GraphQL Search API supports date filtering via search qualifiers:
```graphql
query {
search(query: "is:pr created:>=2024-01-01 repo:owner/repo", type: ISSUE, first: 100) {
nodes { ... on PullRequest { ... } }
}
}
```
However, this approach has limitations:
- **Result cap**: Search API is limited to 1000 results per query
- **Rate limits**: Different rate limiting (30 requests/minute vs GraphQL's 5000 points/hour)
- **Complexity**: Would require fallback logic when hitting the 1000 result limit, adding code complexity
- **Maintainability**: Relies on search syntax (`is:pr`) remaining stable
For most customers with large repositories, the 1000 result limit would trigger fallback to the current pagination approach anyway, negating the optimization while adding complexity.
## Expected Impact
- Significantly faster initial syncs when `start_date` is set
- Reduced API calls and rate limit consumption
- Better user experience for customers syncing recent data from large repositories
## Affected Streams Priority
Based on typical data volumes, recommended implementation order:
1. `pull_requests` / `pull_request_stats` - High volume, commonly used
2. `issue_events` - Very high volume potential
3. `stargazers` - Very high volume for popular repos
4. `events` - High volume
5. `commit_comments`, `deployments`, `projects`, `releases` - Lower priority
---
**Requested by:** @lleadbet
---
**Internal Tracking:** airbytehq/oncall#11097
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.