Source CockroachDB: OOM Issues with CockroachDB Source

### Connector Name

source-cockroachdb

### Connector Version

0.1.22
Airbyte 0.50.34

### What step the error happened?

During the sync

### Relevant information

We are trying to replicate a large table (80M rows) and no matter how much RAM I give the machine Airbyte always runs out of memory. Latest attempt was 80GB RAM, it ran for 3.5 hours with no progress shown and then gave the error.

[Full Log File](https://github.com/airbytehq/airbyte/files/13353971/d4a20d9f_dfbe_4c34_a67f_76b26d7e3025_job_10611_attempt_1_txt.txt)

### Relevant log output

```shell
1. The job is initiated with 80g memory limit

2023-11-14 13:06:06 INFO i.a.w.p.DockerProcessFactory(create):140 - Creating docker container = source-cockroachdb-check-10611-0-rkoqa with resources io.airbyte.config.ResourceRequirements@774bbe8c[cpuRequest=,cpuLimit=0.75,memoryRequest=,memoryLimit=80g,additionalProperties={}] and allowedHosts io.airbyte.config.AllowedHosts@3136884[hosts=[motion-prod-gnp.gcp-us-central1.cockroachlabs.cloud, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}]

2. For three hours I only see the following log and no other info. Why does it say 29GB memory? Setting a Memory Request does not affect this. Any way to see what else is going on?

2023-11-14 16:44:13 destination > 2023-11-14 16:44:13 INFO i.a.c.i.d.FlushWorkers(printWorkerInfo):146 - WORKER INFO
2023-11-14 16:44:13 destination > Pool queue size: 0, Active threads: 0
2023-11-14 16:44:23 destination > 2023-11-14 16:44:23 INFO i.a.c.i.d.b.BufferManager(printQueueInfo):120 - START OF QUEUE INFO
2023-11-14 16:44:23 destination > This represents an estimation of the size of the elements contain in the in memory buffer.
2023-11-14 16:44:23 destination > Global Mem Manager -- max: 29 GB, allocated: 10 MB (10.0 MB), % used: 3.3674568965517244E-4
2023-11-14 16:44:23 destination > State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.000000
2023-11-14 16:44:23 destination > END OF QUEUE INFO

3. After 3 hours I get heap space error. No visible progress has been made
2023-11-14 16:44:32 source > Terminating due to java.lang.OutOfMemoryError: Java heap space
2023-11-14 16:44:33 INFO i.a.w.g.BufferedReplicationWorker(readFromSource):372 - readFromSource: source exception
io.airbyte.workers.internal.exception.SourceException: Source process exited with non-zero exit code 3
at io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:369) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:243) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-11-14 16:44:33 INFO i.a.w.g.BufferedReplicationWorker(readFromSource):382 - readFromSource: done. (source.isFinished:true, fromSource.isClosed:false)
2023-11-14 16:44:33 INFO i.a.w.g.BufferedReplicationWorker(processMessage):422 - processMessage: done. (fromSource.isDone:true, forDest.isClosed:false)

"streamStats" : [ ],
"performanceMetrics" : {
"processFromSource" : {
"elapsedTimeInNanos" : 0,
"executionCount" : 0,
"avgExecTimeInNanos" : "NaN"
},
"readFromSource" : {
"elapsedTimeInNanos" : 13086346532795,
"executionCount" : 284,
"avgExecTimeInNanos" : 4.607868497463028E10
},
"processFromDest" : {
"elapsedTimeInNanos" : 0,
"executionCount" : 0,
"avgExecTimeInNanos" : "NaN"
},
"writeToDest" : {
"elapsedTimeInNanos" : 0,
"executionCount" : 0,
"avgExecTimeInNanos" : "NaN"
},
"readFromDest" : {
"elapsedTimeInNanos" : 13089923207107,
"executionCount" : 403,
"avgExecTimeInNanos" : 3.2481199025079403E10
}
}
}
2023-11-14 16:44:36 INFO i.a.w.g.ReplicationWorkerHelper(getReplicationOutput):245 - failures: [ {
"failureOrigin" : "source",
"internalMessage" : "Source process exited with non-zero exit code 3",
"externalMessage" : "Something went wrong within the source connector",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 10611,
"connector_command" : "read"
},
"stacktrace" : "io.airbyte.workers.internal.exception.SourceException: Source process exited with non-zero exit code 3\n\tat io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:369)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:243)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\n",
"timestamp" : 1699980273203
} ]
2023-11-14 16:44:36 INFO i.a.c.i.LineGobbler(voidCall):149 -
2023-11-14 16:44:36 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- END REPLICATION -----
```