Source MongoDB Incremental Sync

Bug Description

Bug Description:

MongoDB timestamp fields (createdAt, updatedAt) created via Mongoose's timestamps: true option are not being recognized as valid incremental sync field options in PostHog's Data Warehouse MongoDB connector. Only the _id field appears as available for incremental replication, despite timestamp fields being at the root level of documents.

Steps to Reproduce:

1. Connect MongoDB Atlas database as a source in PostHog Data Warehouse
2. Use MongoDB collections with schemas that include timestamps: true (creating createdAt and updatedAt fields)
3. Navigate to sync settings for a table
4. Attempt to select an incremental sync method
5. Observe available fields for incremental sync cursor

Expected Behavior:

MongoDB's auto-generated createdAt and updatedAt timestamp fields (ISODate format) should be recognized as valid incremental sync cursor fields, as they are:

• At the root level of documents
• Native MongoDB Date type (ISODate)
• Valid timestamp/datetime fields that meet PostHog's supported field types for incremental replication

Actual Behavior:

Only _id field appears as an option for incremental sync. The createdAt and updatedAt timestamp fields are not detected as valid replication keys.

Inconsistency Observed:

In one table, a date field WAS successfully recognized and appeared as an incremental sync option, suggesting the schema discovery can detect date fields but is doing so inconsistently across collections.

Environment:

• PostHog Cloud
• MongoDB Atlas
• All collections use timestamps: true for automatic timestamp management

Additional Context:

This prevents efficient incremental syncing based on update timestamps, forcing use of either _id (which only captures new documents, not updates) or full table sync (inefficient for large collections).

Debug info

Start by tracing MongoDB connector schema discovery and the sync-settings logic that filters available incremental cursor fields. Reproduce the issue with Atlas collections using Mongoose timestamps, compare the inconsistent date-field detection, and verify that root-level createdAt and updatedAt appear as valid options without losing _id.