airbytehq / airbytehq/airbyte

source-youtube-data: Multiple channel_ids causes videos/video/comments streams to return no data

Ouverte
#72,638 4 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
autoteam community connectors/source/youtube-data team/use
Langage dominant
Python
Étoiles
22.1k
Forks
5.3k
Métriques de merge des PR
Métriques de PR en attente

Description

## Bug Description

When configuring the YouTube Data source connector with multiple `channel_ids`, the `videos`, `video`, and `comments` streams return no data. The streams only work correctly when a single channel ID is provided.

## Environment

- Connector: `source-youtube-data`
- Connector Version: 0.0.45
- Airbyte Platform: Self-hosted (Airbyte OSS)

## Steps to Reproduce

1. Create a YouTube Data source with multiple channel IDs:
```json
{
"credentials": {
"auth_method": "api_key",
"api_key": "YOUR_API_KEY"
},
"channel_ids": ["UCJr72fY4cTaNZv7WPbvjaSw", "UC8lxnUR_CzruT2KA6cb7p0Q"]
}
```
2. Run a sync
3. Observe that `videos` and `video` streams return 0 records

## Expected Behavior

All configured channels should have their videos discovered and synced.

## Actual Behavior

- With 1 channel: `videos` and `video` streams work correctly
- With 2+ channels: `videos` and `video` streams return no data

## Root Cause Analysis

Looking at the [manifest.yaml](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-youtube-data/manifest.yaml), the `videos` stream passes the entire `channel_ids` array to the YouTube Search API's `channelId` parameter:

```yaml
videos:
type: DeclarativeStream
name: videos
retriever:
requester:
path: search
request_parameters:
channelId: "{{ config.channel_ids }}" # BUG: passes array
```

However, the [YouTube Data API v3 Search endpoint](https://developers.google.com/youtube/v3/docs/search/list) only accepts a **single** `channelId` value, not an array. When an array is passed, the API likely rejects or ignores the invalid input.

This is in contrast to the `channels` stream which correctly uses a `ListPartitionRouter` to iterate over each channel ID:

```yaml
channels:
retriever:
partition_router:
type: ListPartitionRouter
values: "{{ config.channel_ids }}"
request_option:
field_name: id
```

## Suggested Fix

Add a `ListPartitionRouter` to the `videos` stream (and `channel_comments` stream which has the same issue):

```yaml
videos:
type: DeclarativeStream
name: videos
retriever:
type: SimpleRetriever
requester:
$ref: "#/definitions/base_requester"
path: search
http_method: GET
request_parameters:
type: video
record_selector:
type: RecordSelector
extractor:
type: DpathExtractor
field_path:
- items
- "*"
- id
paginator:
type: DefaultPaginator
page_token_option:
type: RequestOption
inject_into: request_parameter
field_name: pageToken
page_size_option:
type: RequestOption
inject_into: request_parameter
field_name: maxResults
pagination_strategy:
type: CursorPagination
page_size: 50
cursor_value: "{{ response.nextPageToken }}"
stop_condition: "{{ not response.get('nextPageToken') }}"
partition_router:
type: ListPartitionRouter
values: "{{ config.channel_ids }}"
cursor_field: channel_id
request_option:
type: RequestOption
inject_into: request_parameter
field_name: channelId
```

## Affected Streams

- `videos` - passes array to `channelId` parameter
- `video` - depends on `videos` as parent stream, so also broken
- `comments` - depends on `videos` as parent stream, so also broken
- `channel_comments` - same bug pattern with `allThreadsRelatedToChannelId`

## Workaround

Create separate Airbyte sources for each YouTube channel instead of using a single source with multiple channel IDs.

## Additional Context

- [YouTube Data API Search documentation](https://developers.google.com/youtube/v3/docs/search/list)
- [StackOverflow discussion confirming channelId only accepts single value](https://stackoverflow.com/questions/71256178/is-it-possible-to-search-multiple-channels-with-one-query)

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11141

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.