airbytehq / airbytehq/airbyte-python-cdk

SubstreamPartitionRouter emits partitions for explicitly null parent_key values, producing literal 'None' in request paths

Open Beginner friendly
#1,127 1 comment 0 reactions 0 assignees View on GitHub
community
Dominant language
Python
Stars
26
Forks
53
Avg merge
2d 6h
Merged PRs (30d)
10

Description

## Symptom

`SubstreamPartitionRouter` handles a missing `parent_key` path and an explicitly null `parent_key` value inconsistently:

- parent record LACKS the `parent_key` path >> `dpath.get` raises `KeyError` >> the record is silently skipped (`emit_slice = False`), no partition is produced;
- parent record HAS the path but its value is explicitly `null` >> `dpath.get` returns `None` >> a partition IS produced with a `None` partition value.

A downstream requester that interpolates the partition value into its path then issues a garbage request with the literal string `None` in the URL, e.g. `GET https://api.linkedin.com/rest/posts/None`. Most APIs answer 400, which maps to FAIL via `DEFAULT_ERROR_MAPPING` and kills the whole sync.

## Root cause

At tag v7.23.8, `airbyte_cdk/sources/declarative/partition_routers/substream_partition_router.py`:

- lines 232-239: only the `KeyError` branch sets `emit_slice = False` (with the in-code FIXME noting the missing log);
- line 215-216: the guard only checks `parent_record is not None`, not the extracted `partition_value`.

So `partition_value = None` flows through to the yielded `StreamSlice`, and Jinja stringifies it during request interpolation (`'None'`).

## Minimal reproduction

Parent stream returns `{"id": "c1", "content": {"reference": null}}`; child stream config:

```yaml
partition_router:
type: SubstreamPartitionRouter
parent_stream_configs:
- type: ParentStreamConfig
parent_key: content/reference
partition_field: post_urn
stream: { $ref: "#/definitions/streams/parent" }
```

with requester `path: posts/{{ stream_slice.get('post_urn') }}`. Observed request: `GET /posts/None`. A parent record with no `content` key at all is correctly skipped.

## Expected

A null partition value should be treated like a missing one (skip the record, ideally with a log line), or at minimum be skippable via configuration. Emitting a partition whose value is `None` is never useful: it cannot address a real resource.

## Impact

Any substream whose parent API emits explicit `null` for the parent key field fails the sync with a confusing 400 on a `.../None` URL. Connectors have to work around it with response-filter IGNOREs on the child requester, which also masks genuinely malformed requests.

## Precedent

Found while reviewing https://github.com/airbytehq/airbyte/pull/81509 (source-linkedin-ads `videos` stream: `creatives.content.reference` >> `GET /rest/posts/{urn}`). The connector-side mitigation there is an IGNORE response filter matching URN-related 400s.

Contributor guide

Open the contributing guide

Research direction

Start in airbyte_cdk/sources/declarative/partition_routers/substream_partition_router.py around lines 215-239, then use the minimal reproduction with an explicitly null parent_key to trace the emitted StreamSlice. Done means null partition values are skipped like missing paths, preventing a request path containing literal None; preserve the existing behavior for valid values.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.