airbytehq / airbytehq/airbyte

[source-mongodb-v2] _id field schema discovery is extremely expensive

Aperta
#80,160 4 commenti 0 reazioni 0 assegnatari Vedi su GitHub
area/connectors autoteam community connectors/source/mongodb-v2 hyd-fix hyd-review needs-triage team/extensibility type/bug
Lingua principale
Python
Stelle
22.1k
Fork
5.3k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

### Connector Name

source-mongodb-v2

### Connector Version

2.0.7

### What step the error happened?

During the sync

### Relevant information

We are syncing a collection with tens of millions of documents. We noticed in our MongoDB logs that we had some queries taking **30 minutes**. It seems that it's this query:
https://github.com/airbytehq/airbyte/blob/a8e626e457d88014d2236624751e787f6040cb5d/airbyte-integrations/connectors/source-mongodb-v2/src/main/java/io/airbyte/integrations/source/mongodb/InitialSnapshotHandler.java#L125

Here's a screenshot from our MongoDB Atlas
Image

It seems that the purpose of this query is to discover if any `_id` fields have a type other that `ObjectId`. In our case we are willing to guarantee that they don't. Could you introduce an option to skip this discovery? Doing a full collection scan is very expensive for this operation, it's unindexable and so it will always be slow.

### Relevant log output

```shell
N/A
```

### Contribute

- [x] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/12891

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.