airbytehq / airbytehq/airbyte-python-cdk

docs: Document pattern for using non-exposed stream definitions as parent streams

Đang mở
#866 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
26
Fork
53
Merge trung bình
2 ngày 6 giờ
Pull request đã merge (30 ngày)
10

Mô tả

## Summary

When building complex connectors with multi-level substream hierarchies, it's useful to define stream definitions that are only used internally as parent streams for other streams, without exposing them as top-level streams. This pattern is currently undocumented but is actively used in production connectors.

## Problem

The current YAML Reference documentation explains that only entries in the top-level `streams:` array are exposed as runnable streams, but it doesn't explicitly document the pattern of:
1. Defining a full stream definition in `definitions` that is NOT listed in `streams:`
2. Using that definition solely as a `parent_stream_config` for another stream
3. The naming convention some connectors use (e.g., `__` prefix) to signal "internal helper"

This pattern is particularly useful for 3-level nested substream hierarchies where an intermediate stream is needed to provide partition keys but shouldn't be exposed to users.

## Example Implementation: Jira Connector

The Jira connector uses this pattern extensively. Here are code permalinks:

### Internal/Private Stream Definitions (in `definitions`, NOT in `streams:`)

- [`__issue_property_keys_substream`](https://github.com/airbytehq/airbyte/blob/b4a7fe87016dd6e12688d82545809696b0292055/airbyte-integrations/connectors/source-jira/manifest.yaml#L10055) - Used as parent for `issue_properties_stream`
- [`__custom_issue_fields_substream`](https://github.com/airbytehq/airbyte/blob/b4a7fe87016dd6e12688d82545809696b0292055/airbyte-integrations/connectors/source-jira/manifest.yaml#L9882) - Used as parent for `issue_custom_field_contexts`
- [`__issue_custom_field_contexts_substream`](https://github.com/airbytehq/airbyte/blob/b4a7fe87016dd6e12688d82545809696b0292055/airbyte-integrations/connectors/source-jira/manifest.yaml#L9971) - Used as parent for `issue_custom_field_options`
- [`__boards_substream`](https://github.com/airbytehq/airbyte/blob/b4a7fe87016dd6e12688d82545809696b0292055/airbyte-integrations/connectors/source-jira/manifest.yaml#L11744) - Used as parent for board-related streams
- [`__story_points_issue_fields_substream`](https://github.com/airbytehq/airbyte/blob/b4a7fe87016dd6e12688d82545809696b0292055/airbyte-integrations/connectors/source-jira/manifest.yaml#L13990) - Used for story points configuration

### How It's Used (3-level hierarchy example)

The [`issue_properties_stream`](https://github.com/airbytehq/airbyte/blob/b4a7fe87016dd6e12688d82545809696b0292055/airbyte-integrations/connectors/source-jira/manifest.yaml#L10080-L10090) references the internal `__issue_property_keys_substream` as its parent:

```yaml
issue_properties_stream:
# ...
retriever:
# ...
partition_router:
type: SubstreamPartitionRouter
parent_stream_configs:
- type: ParentStreamConfig
stream: "#/definitions/__issue_property_keys_substream" # <-- Internal stream reference
```

This creates a 3-level hierarchy:
1. `issues_stream` (grandparent - exposed)
2. `__issue_property_keys_substream` (parent - internal, NOT exposed)
3. `issue_properties_stream` (child - exposed)

### Top-level `streams:` Section

The [streams section](https://github.com/airbytehq/airbyte/blob/b4a7fe87016dd6e12688d82545809696b0292055/airbyte-integrations/connectors/source-jira/manifest.yaml#L14555) only lists the streams that should be exposed to users - the `__`-prefixed definitions are intentionally omitted.

## Suggested Documentation

Add a section to the YAML Reference or a new "Advanced Patterns" page that documents:

1. **Pattern**: Using stream definitions as internal parent streams
2. **Use case**: Multi-level substream hierarchies where intermediate streams shouldn't be exposed
3. **Naming convention**: The `__` prefix convention (optional but recommended for clarity)
4. **Behavior**: Streams not listed in `streams:` will not be exposed by `source.streams(config)` - attempting to sync them will silently no-op
5. **Testing implications**: When writing mock server tests, always verify stream names against the `streams:` section to avoid testing non-existent streams

## Context

This issue was discovered while creating comprehensive mock server tests for the Jira connector (airbytehq/airbyte#70884). The pattern caused confusion when attempting to test `issue_property_keys` as a stream, only to discover it's an internal-only definition.

---
**Requested by:** AJ Steers (@aaronsteers)
**Related PR:** https://github.com/airbytehq/airbyte/pull/70884
**Devin session:** https://app.devin.ai/sessions/f152f435f9d146688e476611ff864c30

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.