MongoDB Connector: noTimeout cursors disallowed on Atlas free tier [DBZ-4849]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-4849](https://issues.redhat.com/browse/DBZ-4849)
When i try to use DBZ with MongoDB Atlas (managed db service) M0 tier (free tier), i encounter this error
{code:java}
2022-03-11 15:18:27,553 ERROR Error while attempting to read from oplog on 'atlas-immt1l-shard-0/cluster0-shard-00-01.ii90w.mongodb.net:27017': Query failed with error code 8000 and error message 'noTimeout cursors are disallowed in this atlas tier' on server cluster0-shard-00-01.ii90w.mongodb.net:27017 (io.debezium.connector.mongodb.MongoDbStreamingChangeEventSource) [debezium-mongodbconnector-post-change-event-source-coordinator]
com.mongodb.MongoQueryException: Query failed with error code 8000 and error message 'noTimeout cursors are disallowed in this atlas tier' on server cluster0-shard-00-01.ii90w.mongodb.net:27017
at com.mongodb.internal.operation.FindOperation$1.call(FindOperation.java:662) {code}
Soon after this error, DBZ connector stops
{code:java}
2022-03-11 15:18:27,587 INFO Stopping down connector (io.debezium.connector.common.BaseSourceTask) [task-thread-post-mongodb-connector-0]
2022-03-11 15:18:27,745 INFO Closed connection [connectionId{localValue:15, serverValue:93730}] to cluster0-shard-00-01.ii90w.mongodb.net:27017 because the pool has been closed. (org.mongodb.driver.connection) [debezium-mongodbconnector-post-change-event-source-coordinator]
2022-03-11 15:18:28,132 INFO Closed connection [connectionId{localValue:12, serverValue:93731}] to cluster0-shard-00-01.ii90w.mongodb.net:27017 because the pool has been closed. (org.mongodb.driver.connection) [debezium-mongodbconnector-post-change-event-source-coordinator]
2022-03-11 15:18:28,337 INFO Finished streaming (io.debezium.pipeline.ChangeEventSourceCoordinator) [debezium-mongodbconnector-post-change-event-source-coordinator] {code}
I know that workaround is to launch a higher tier mongodb cluster, but my point is to enable DBZ to work with even the lowest MongoDB Atlas tier, for which we need to add some feature in DBZ mongodb connector such that it can periodically refresh the session using refreshsession call explicitly in the background. May be a {{while}} loop includes a block that uses [{{refreshSessions}}|https://docs.mongodb.com/manual/reference/command/refreshSessions/#mongodb-dbcommand-dbcmd.refreshSessions] to refresh the session every 5 minutes. Since the session will never exceed the 30 minute idle timeout, the cursor can remain open indefinitely.
This solution is describe here https://docs.mongodb.com/manual/reference/method/cursor.noCursorTimeout/
Don't forget to see the attached connector logs
Contributor guide
Research direction
Start with the MongoDbStreamingChangeEventSource entry point named in the error and review how the oplog cursor and session are managed. Reproduce the failure against a MongoDB Atlas M0 tier, then verify that the connector keeps streaming without stopping when noTimeout cursors are disallowed, using the refreshSessions and noCursorTimeout references in the issue as context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mongodb
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100