feat(source-mongodb): Handle polymorphic fields where the same field has different BSON types across documents
- Linguagem predominante
- Python
- Estrelas
- 22.1k
- Forks
- 5.3k
- Métricas de merge de PRs
- Métricas de PR pendentes
Descrição
## Problem
When using MongoDB source with Schema Enforced mode, schema discovery samples a subset of documents to infer field types. If a field stores values of different BSON types across documents — for example, some documents store it as a JSON object while others store it as a plain string — the discovered schema reflects only the type found in the sample.
When the actual data contains values of a different type than what was discovered, downstream destinations (such as BigQuery) fail to deserialize the value into the expected column type. This results in a `DESTINATION_SERIALIZATION_ERROR`, and the affected field is silently nullified — the sync succeeds, but data is lost for those records.
## Context and Related Work
- A similar issue was addressed for array-type mismatches in https://github.com/airbytehq/airbyte/pull/70980, where non-array values are wrapped in arrays when the schema expects an array. However, this handling does not extend to other type mismatches (e.g., object vs. string).
- https://github.com/airbytehq/airbyte/discussions/45130 proposed support for user-defined schemas as an alternative approach.
- The schema-less mode avoids the problem by treating all data as blobs, but this sacrifices the benefits of typed schema enforcement.
## Proposed Enhancement
Extend the MongoDB source connector's Schema Enforced mode to gracefully handle polymorphic fields — fields where the BSON type varies across documents. Possible approaches include:
1. **Union types in schema discovery** — When sampling detects multiple types for the same field, represent the field as a union type (e.g., `["object", "string"]`) rather than picking a single type.
2. **Runtime type coercion** — Similar to the array-wrapping fix in https://github.com/airbytehq/airbyte/pull/70980, detect type mismatches at read time and coerce values to a compatible representation so they are not silently nullified.
3. **User-defined schema overrides** — Allow users to explicitly define the expected schema for specific fields or collections, as proposed in https://github.com/airbytehq/airbyte/discussions/45130.
## Expected Behavior
Fields with varying BSON types across documents should not result in silent data loss. The connector should either discover the correct union type or coerce values at read time so that all records are faithfully replicated to the destination.
## Current Behavior
- Schema discovery picks a single type based on the sample.
- Documents where the field has a different type cause destination serialization errors.
- The affected field values are silently nullified, leading to data loss without clear user-facing warnings.
---
_Requested by Vikram Srigada via Zendesk Ticket [#16309](https://airbyte1416.zendesk.com/agent/tickets/16309)._
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/12121
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.