[source-mongodb-v2] _id field schema discovery is extremely expensive
- Lingua principale
- Python
- Stelle
- 22.1k
- Fork
- 5.3k
- Metriche di merge delle PR
- Metriche PR in attesa
Descrizione
### Connector Name
source-mongodb-v2
### Connector Version
2.0.7
### What step the error happened?
During the sync
### Relevant information
We are syncing a collection with tens of millions of documents. We noticed in our MongoDB logs that we had some queries taking **30 minutes**. It seems that it's this query:
https://github.com/airbytehq/airbyte/blob/a8e626e457d88014d2236624751e787f6040cb5d/airbyte-integrations/connectors/source-mongodb-v2/src/main/java/io/airbyte/integrations/source/mongodb/InitialSnapshotHandler.java#L125
Here's a screenshot from our MongoDB Atlas
It seems that the purpose of this query is to discover if any `_id` fields have a type other that `ObjectId`. In our case we are willing to guarantee that they don't. Could you introduce an option to skip this discovery? Doing a full collection scan is very expensive for this operation, it's unindexable and so it will always be slow.
### Relevant log output
```shell
N/A
```
### Contribute
- [x] Yes, I want to contribute
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/12891
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.