debezium / debezium/dbz

MongoDB Connector: noTimeout cursors disallowed on Atlas free tier [DBZ-4849]

Open
#593 0 comments 0 reactions 0 assignees View on GitHub
component/mongodb-connector migrated-from-jira type/bug
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-4849](https://issues.redhat.com/browse/DBZ-4849)

When i try to use DBZ with MongoDB Atlas (managed db service) M0 tier (free tier), i encounter this error
{code:java}
2022-03-11 15:18:27,553 ERROR Error while attempting to read from oplog on 'atlas-immt1l-shard-0/cluster0-shard-00-01.ii90w.mongodb.net:27017': Query failed with error code 8000 and error message 'noTimeout cursors are disallowed in this atlas tier' on server cluster0-shard-00-01.ii90w.mongodb.net:27017 (io.debezium.connector.mongodb.MongoDbStreamingChangeEventSource) [debezium-mongodbconnector-post-change-event-source-coordinator]
com.mongodb.MongoQueryException: Query failed with error code 8000 and error message 'noTimeout cursors are disallowed in this atlas tier' on server cluster0-shard-00-01.ii90w.mongodb.net:27017
at com.mongodb.internal.operation.FindOperation$1.call(FindOperation.java:662) {code}
Soon after  this error, DBZ connector stops
{code:java}
2022-03-11 15:18:27,587 INFO Stopping down connector (io.debezium.connector.common.BaseSourceTask) [task-thread-post-mongodb-connector-0]
2022-03-11 15:18:27,745 INFO Closed connection [connectionId{localValue:15, serverValue:93730}] to cluster0-shard-00-01.ii90w.mongodb.net:27017 because the pool has been closed. (org.mongodb.driver.connection) [debezium-mongodbconnector-post-change-event-source-coordinator]
2022-03-11 15:18:28,132 INFO Closed connection [connectionId{localValue:12, serverValue:93731}] to cluster0-shard-00-01.ii90w.mongodb.net:27017 because the pool has been closed. (org.mongodb.driver.connection) [debezium-mongodbconnector-post-change-event-source-coordinator]
2022-03-11 15:18:28,337 INFO Finished streaming (io.debezium.pipeline.ChangeEventSourceCoordinator) [debezium-mongodbconnector-post-change-event-source-coordinator] {code}
I know that workaround is to launch a higher tier mongodb cluster, but my point is to enable DBZ to work with even the lowest MongoDB Atlas tier, for which we need to add some feature in DBZ mongodb connector such that it can periodically refresh the session using refreshsession call explicitly in the background. May be a {{while}} loop includes a block that uses [{{refreshSessions}}|https://docs.mongodb.com/manual/reference/command/refreshSessions/#mongodb-dbcommand-dbcmd.refreshSessions] to refresh the session every 5 minutes. Since the session will never exceed the 30 minute idle timeout, the cursor can remain open indefinitely.

 

This solution is describe here https://docs.mongodb.com/manual/reference/method/cursor.noCursorTimeout/

 

Don't forget to see the attached connector logs

 

 

 

 

 

Contributor guide

Open the contributing guide

Research direction

Start with the MongoDbStreamingChangeEventSource entry point named in the error and review how the oplog cursor and session are managed. Reproduce the failure against a MongoDB Atlas M0 tier, then verify that the connector keeps streaming without stopping when noTimeout cursors are disallowed, using the refreshSessions and noCursorTimeout references in the issue as context.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.