airbytehq / airbytehq/airbyte-python-cdk

SubstreamPartitionRouter emits partitions for explicitly null parent_key values, producing literal 'None' in request paths

Abierto Apto para principiantes
#1,127 1 comentario 0 reacciones 0 asignados Ver en GitHub
community
Lenguaje dominante
Python
Estrellas
26
Forks
53
Merge medio
2 d 6 h
PR fusionados (30 d)
10

Descripción

## Symptom

`SubstreamPartitionRouter` handles a missing `parent_key` path and an explicitly null `parent_key` value inconsistently:

- parent record LACKS the `parent_key` path >> `dpath.get` raises `KeyError` >> the record is silently skipped (`emit_slice = False`), no partition is produced;
- parent record HAS the path but its value is explicitly `null` >> `dpath.get` returns `None` >> a partition IS produced with a `None` partition value.

A downstream requester that interpolates the partition value into its path then issues a garbage request with the literal string `None` in the URL, e.g. `GET https://api.linkedin.com/rest/posts/None`. Most APIs answer 400, which maps to FAIL via `DEFAULT_ERROR_MAPPING` and kills the whole sync.

## Root cause

At tag v7.23.8, `airbyte_cdk/sources/declarative/partition_routers/substream_partition_router.py`:

- lines 232-239: only the `KeyError` branch sets `emit_slice = False` (with the in-code FIXME noting the missing log);
- line 215-216: the guard only checks `parent_record is not None`, not the extracted `partition_value`.

So `partition_value = None` flows through to the yielded `StreamSlice`, and Jinja stringifies it during request interpolation (`'None'`).

## Minimal reproduction

Parent stream returns `{"id": "c1", "content": {"reference": null}}`; child stream config:

```yaml
partition_router:
type: SubstreamPartitionRouter
parent_stream_configs:
- type: ParentStreamConfig
parent_key: content/reference
partition_field: post_urn
stream: { $ref: "#/definitions/streams/parent" }
```

with requester `path: posts/{{ stream_slice.get('post_urn') }}`. Observed request: `GET /posts/None`. A parent record with no `content` key at all is correctly skipped.

## Expected

A null partition value should be treated like a missing one (skip the record, ideally with a log line), or at minimum be skippable via configuration. Emitting a partition whose value is `None` is never useful: it cannot address a real resource.

## Impact

Any substream whose parent API emits explicit `null` for the parent key field fails the sync with a confusing 400 on a `.../None` URL. Connectors have to work around it with response-filter IGNOREs on the child requester, which also masks genuinely malformed requests.

## Precedent

Found while reviewing https://github.com/airbytehq/airbyte/pull/81509 (source-linkedin-ads `videos` stream: `creatives.content.reference` >> `GET /rest/posts/{urn}`). The connector-side mitigation there is an IGNORE response filter matching URN-related 400s.

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Start in airbyte_cdk/sources/declarative/partition_routers/substream_partition_router.py around lines 215-239, then use the minimal reproduction with an explicitly null parent_key to trace the emitted StreamSlice. Done means null partition values are skipped like missing paths, preventing a request path containing literal None; preserve the existing behavior for valid values.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python
Área
api, backend
Tipo de issue
Error
Dificultad
2/5
Tiempo estimado
1-3 horas
Estado de actividad
Activo
Claridad
Bien especificado
Aptitud para principiantes
82/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.