airbytehq / airbytehq/airbyte

Source Stripe: Expandable fields missing in Events API may lead to incomplete data

Offen
#38,039 5 Kommentare 2 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
autoteam connectors/source/stripe feedback-requested Icebox missing-data team/connectors-python team/critical-connectors team/extensibility
Vorherrschende Sprache
Python
Sterne
22.1k
Forks
5.3k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

## Current situation

The Stripe API doesn't support retrieving `expandable` fields through the Events API. This makes it impossible to construct the full data in incremental sync without retrieving the full objects. This may imply that existing data is updated with `null` when writing in the destination. One example of this expandable fields in the API is the [Price.tiers object.](https://docs.stripe.com/api/prices/object#price_object-tiers)

## Reproducing the issue

Create a new pricing object with `billing_scheme = tiered`, and fill the required fields in the Tiers object. Then query for events related to the newly created Price. Results don't include tiers information.

## Desired situation

Connector logic to retrieve updated `expandable` fields in incremental sync is implemented. We update multiple objects from a single API call if possible for performance. We make sure `expandable` fields don't override values with null in destinations when not present.

## Proposed solution

We keep track of which fields are `expandable` throughout the API. Objects with expandable data that also have update events are queried to populate all fields when doing incremental sync.

## Expected outcome

Data is complete and up-to-date even when expandable data is present in API objects. Test end to end to check no wrong nulls are written. As we will perform one HTTP query per record, we expect this to slow the syncs but don't have a good measure of how much.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.