airbytehq / airbytehq/airbyte-python-cdk

docs: Document pattern for using non-exposed stream definitions as parent streams

Abierto
#866 1 comentario 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
26
Forks
53
Merge medio
2 d 6 h
PR fusionados (30 d)
10

Descripción

## Summary

When building complex connectors with multi-level substream hierarchies, it's useful to define stream definitions that are only used internally as parent streams for other streams, without exposing them as top-level streams. This pattern is currently undocumented but is actively used in production connectors.

## Problem

The current YAML Reference documentation explains that only entries in the top-level `streams:` array are exposed as runnable streams, but it doesn't explicitly document the pattern of:
1. Defining a full stream definition in `definitions` that is NOT listed in `streams:`
2. Using that definition solely as a `parent_stream_config` for another stream
3. The naming convention some connectors use (e.g., `__` prefix) to signal "internal helper"

This pattern is particularly useful for 3-level nested substream hierarchies where an intermediate stream is needed to provide partition keys but shouldn't be exposed to users.

## Example Implementation: Jira Connector

The Jira connector uses this pattern extensively. Here are code permalinks:

### Internal/Private Stream Definitions (in `definitions`, NOT in `streams:`)

- [`__issue_property_keys_substream`](https://github.com/airbytehq/airbyte/blob/b4a7fe87016dd6e12688d82545809696b0292055/airbyte-integrations/connectors/source-jira/manifest.yaml#L10055) - Used as parent for `issue_properties_stream`
- [`__custom_issue_fields_substream`](https://github.com/airbytehq/airbyte/blob/b4a7fe87016dd6e12688d82545809696b0292055/airbyte-integrations/connectors/source-jira/manifest.yaml#L9882) - Used as parent for `issue_custom_field_contexts`
- [`__issue_custom_field_contexts_substream`](https://github.com/airbytehq/airbyte/blob/b4a7fe87016dd6e12688d82545809696b0292055/airbyte-integrations/connectors/source-jira/manifest.yaml#L9971) - Used as parent for `issue_custom_field_options`
- [`__boards_substream`](https://github.com/airbytehq/airbyte/blob/b4a7fe87016dd6e12688d82545809696b0292055/airbyte-integrations/connectors/source-jira/manifest.yaml#L11744) - Used as parent for board-related streams
- [`__story_points_issue_fields_substream`](https://github.com/airbytehq/airbyte/blob/b4a7fe87016dd6e12688d82545809696b0292055/airbyte-integrations/connectors/source-jira/manifest.yaml#L13990) - Used for story points configuration

### How It's Used (3-level hierarchy example)

The [`issue_properties_stream`](https://github.com/airbytehq/airbyte/blob/b4a7fe87016dd6e12688d82545809696b0292055/airbyte-integrations/connectors/source-jira/manifest.yaml#L10080-L10090) references the internal `__issue_property_keys_substream` as its parent:

```yaml
issue_properties_stream:
# ...
retriever:
# ...
partition_router:
type: SubstreamPartitionRouter
parent_stream_configs:
- type: ParentStreamConfig
stream: "#/definitions/__issue_property_keys_substream" # <-- Internal stream reference
```

This creates a 3-level hierarchy:
1. `issues_stream` (grandparent - exposed)
2. `__issue_property_keys_substream` (parent - internal, NOT exposed)
3. `issue_properties_stream` (child - exposed)

### Top-level `streams:` Section

The [streams section](https://github.com/airbytehq/airbyte/blob/b4a7fe87016dd6e12688d82545809696b0292055/airbyte-integrations/connectors/source-jira/manifest.yaml#L14555) only lists the streams that should be exposed to users - the `__`-prefixed definitions are intentionally omitted.

## Suggested Documentation

Add a section to the YAML Reference or a new "Advanced Patterns" page that documents:

1. **Pattern**: Using stream definitions as internal parent streams
2. **Use case**: Multi-level substream hierarchies where intermediate streams shouldn't be exposed
3. **Naming convention**: The `__` prefix convention (optional but recommended for clarity)
4. **Behavior**: Streams not listed in `streams:` will not be exposed by `source.streams(config)` - attempting to sync them will silently no-op
5. **Testing implications**: When writing mock server tests, always verify stream names against the `streams:` section to avoid testing non-existent streams

## Context

This issue was discovered while creating comprehensive mock server tests for the Jira connector (airbytehq/airbyte#70884). The pattern caused confusion when attempting to test `issue_property_keys` as a stream, only to discover it's an internal-only definition.

---
**Requested by:** AJ Steers (@aaronsteers)
**Related PR:** https://github.com/airbytehq/airbyte/pull/70884
**Devin session:** https://app.devin.ai/sessions/f152f435f9d146688e476611ff864c30

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.