airbytehq / airbytehq/airbyte

[source-iterable] users stream ships a static schema that drops all custom user fields (and misdeclares itblInternal + date formats)

Open
#79,634 3 comments 0 reactions 0 assignees View on GitHub
area/connectors autoteam community connectors/source/iterable hyd-review needs-triage team/extensibility team/use type/bug
Dominant language
Python
Stars
22.1k
Forks
5.3k
PR merge metrics
PR metrics pending

Description

### Connector Name

source-iterable

### Connector Version

0.7.2

### What step the error happened?

During the sync

### Relevant information

### Summary

The `users` stream declares a **static `users.json` schema** (and `metadata.autoImportSchema.users: false`) that likely reflects the original developer's account and not the account being synced. As a result, account-defined custom user `dataFields` are never declared, and with the BigQuery direct-load destination (which materializes only declared fields, with no `_airbyte_data` JSON fallback) they are **silently dropped** — no error, empty `_airbyte_meta.changes`.

Three distinct problems, all in this one stream, plus one observation that helps mitigate a known issue:

### 1. The schema hard-codes a different tenant's custom fields, and captures none of the account's own

[`users.json`](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-iterable/source_iterable/schemas/users.json) enumerates ~70 properties that are plainly one specific customer's Shopify-style custom dataFields (`aov`, `ltr`, `boughtSas`, `totalOrders`, `shopify_created_at`, `shopify_updated_at`, `admin_graphql_api_id`, `mostRecentOrderCards`, `firstPurchaseDate`, …). Because custom dataFields are per-account, these are useless to every other account — they appear as permanently-`NULL` columns — while the connector captures **none** of the syncing account's actual custom fields.

Measured on one real account's export (`export/data.json?dataTypeName=user`, `Content-Type: application/x-json-stream`): **207 unique top-level keys across 6,830 records**, vs ~53 declared in `users.json`. The bulk of the account's data is dropped.

The same hard-coded Shopify-style remnant also appears in `email_send.json` / `email_send_skip.json` under `transactionalData` (fixed sub-schema: `sku`, `vendor`, `product_type`, `compare_at_price`, `handle`, `imageUrl`, …), so custom transactional-data fields are dropped there too.

**Proposed fix (mirrors how this connector already handles event custom data):**

The `events` stream already solves exactly this: [`EventsRecordExtractor`](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-iterable/source_iterable/components.py#L14) pops the known common fields and places everything else into a generic `data` object (`events.json` declares [`data`](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-iterable/source_iterable/schemas/events.json#L36-L39) as a bare `{type: [null, object]}`, which the destination maps to a JSON column). The `users` stream should do the same: keep the [genuinely-standard top-level fields](https://support.iterable.com/hc/en-us/articles/217744303-User-Profile-Fields-Used-by-Iterable#fields-managed-by-iterable), and emit the remaining account-defined dataFields into a `data` object (same field name and approach as `events`, for consistency).

Alternatively/additionally, enable dynamic schema discovery via `GET /api/users/getFields` and set `autoImportSchema.users: true`.

Either way, the static list of one tenant's fields should be removed.

### 2. `itblInternal` is declared as a nested object, but the export returns flattened dotted keys

`users.json` declares [`itblInternal`](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-iterable/source_iterable/schemas/users.json#L322-L341) as a nested object with sub-properties `emailDomain`, `documentCreatedAt`, `documentUpdatedAt`. But the [`export/data.json`](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-iterable/source_iterable/manifest.yaml#L299) endpoint (not to be confused with a file called `data.json`) returns these as **flat top-level keys with dots in the name**, e.g.,`itblInternal.emailDomain`. So the nested `itblInternal` column **never populates** (it's `NULL` in the destination despite the data being present under the dotted keys, which are themselves dropped as undeclared).

Further, the declared [`documentUpdatedAt` and `documentCreatedAt`](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-iterable/source_iterable/schemas/users.json#L330-L339) aren't even present in the export. An undeclared property: `itblInternal.isUnknownUser` is present (which is missing from the schema).

Confirmed across 6,830 records: **zero** top-level values are nested objects — the export is fully flattened. The declared `itblInternal.documentCreatedAt`/`documentUpdatedAt` aren't in the export at all, and the real `isUnknownUser` isn't in the declared schema — so the nested declaration is wrong on both shape and field list.

Note: if a customer is using Iterable's brand affinity features, the `itblDS` field would have the same issue; testing shows it outputs a `itblDS.brandAffinityLabel` field.

### 3. `signupDate` / `profileUpdatedAt` declared `format: date-time` but values aren't RFC 3339

[Both](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-iterable/source_iterable/schemas/users.json#L384-L393) are declared `{type: ["null","string"], format: "date-time"}`, but the export returns space-separated, non-RFC3339 strings, e.g. `2026-04-13 15:02:15 +00:00` (space instead of `T`, space before the offset). The Connector Builder flags this as *"Detected schema and declared schema are different / String does not match format date-time."* It doesn't break syncs (the destination's parser and the stream's `cursor_datetime_formats` both tolerate the space form), but the declared schema is invalid for the data. Fix: add `airbyte_type: timestamp_with_timezone` (or correct the declared format).

### 4. Primary key ambiguity on the `users` stream

This is a bug only to the extent that the Airbyte internals do not handle missing primary key values well (at least not in the case of Iterable, reported in Airbyte Zendesk Ticket ID: 17638). If a primary key column has a null value, it presents to the user as:

> Warning from destination: Input was fully read, but some streams did not receive a terminal stream status message. If the destination did not encounter other errors, this likely indicates an error in the source or platform. Streams without a status message: [campaigns_metrics, email_click, list_users, users, message_types, campaigns, channels, lists, email_open]

In Iterable "hybrid" projects, the configuration [may use](https://support.iterable.com/hc/en-us/articles/9216719179796-Project-Types-and-Unique-Identifiers#unique-identifiers) `userId` OR `email`. Further, if using `email`, this is not stable across user's email changes.

Suggestion is to add the [`itblUserId`](https://support.iterable.com/hc/en-us/articles/217744303-User-Profile-Fields-Used-by-Iterable#itbluserid) field to the schema. While it's officially not recommended as a lookup, it does make sense to use in the Airbyte replication context. It's stable across email changes and present even if a user does not have a userId.

### Impact

For any account with custom user fields (the common case for Iterable), the `users` stream is largely non-functional into BigQuery (and presumably other destinations): the warehouse table ends up with another tenant's NULL columns and none of the real data.

### Relevant log output

```shell

```

### Contribute

- [x] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/12838

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.