airbytehq / airbytehq/airbyte-python-cdk

SubstreamPartitionRouter emits partitions for explicitly null parent_key values, producing literal 'None' in request paths

Đang mở Phù hợp với người mới
#1,127 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
community
Ngôn ngữ chính
Python
Star
26
Fork
53
Merge trung bình
2 ngày 6 giờ
Pull request đã merge (30 ngày)
10

Mô tả

## Symptom

`SubstreamPartitionRouter` handles a missing `parent_key` path and an explicitly null `parent_key` value inconsistently:

- parent record LACKS the `parent_key` path >> `dpath.get` raises `KeyError` >> the record is silently skipped (`emit_slice = False`), no partition is produced;
- parent record HAS the path but its value is explicitly `null` >> `dpath.get` returns `None` >> a partition IS produced with a `None` partition value.

A downstream requester that interpolates the partition value into its path then issues a garbage request with the literal string `None` in the URL, e.g. `GET https://api.linkedin.com/rest/posts/None`. Most APIs answer 400, which maps to FAIL via `DEFAULT_ERROR_MAPPING` and kills the whole sync.

## Root cause

At tag v7.23.8, `airbyte_cdk/sources/declarative/partition_routers/substream_partition_router.py`:

- lines 232-239: only the `KeyError` branch sets `emit_slice = False` (with the in-code FIXME noting the missing log);
- line 215-216: the guard only checks `parent_record is not None`, not the extracted `partition_value`.

So `partition_value = None` flows through to the yielded `StreamSlice`, and Jinja stringifies it during request interpolation (`'None'`).

## Minimal reproduction

Parent stream returns `{"id": "c1", "content": {"reference": null}}`; child stream config:

```yaml
partition_router:
type: SubstreamPartitionRouter
parent_stream_configs:
- type: ParentStreamConfig
parent_key: content/reference
partition_field: post_urn
stream: { $ref: "#/definitions/streams/parent" }
```

with requester `path: posts/{{ stream_slice.get('post_urn') }}`. Observed request: `GET /posts/None`. A parent record with no `content` key at all is correctly skipped.

## Expected

A null partition value should be treated like a missing one (skip the record, ideally with a log line), or at minimum be skippable via configuration. Emitting a partition whose value is `None` is never useful: it cannot address a real resource.

## Impact

Any substream whose parent API emits explicit `null` for the parent key field fails the sync with a confusing 400 on a `.../None` URL. Connectors have to work around it with response-filter IGNOREs on the child requester, which also masks genuinely malformed requests.

## Precedent

Found while reviewing https://github.com/airbytehq/airbyte/pull/81509 (source-linkedin-ads `videos` stream: `creatives.content.reference` >> `GET /rest/posts/{urn}`). The connector-side mitigation there is an IGNORE response filter matching URN-related 400s.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Start in airbyte_cdk/sources/declarative/partition_routers/substream_partition_router.py around lines 215-239, then use the minimal reproduction with an explicitly null parent_key to trace the emitted StreamSlice. Done means null partition values are skipped like missing paths, preventing a request path containing literal None; preserve the existing behavior for valid values.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python
Lĩnh vực
api, backend
Loại issue
Lỗi
Độ khó
2/5
Thời gian dự kiến
1-3 giờ
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Đặc tả rõ ràng
Mức phù hợp với người mới
82/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.