checkOneFutureAndWaitForOther(Helpers.scala:33) while running catboost_spark on google dataproc

**Problem**: can't run the demo code of catboost_spark on google dataproc.
**catboost** tried both version:1.2.7 and 1.1.1
**Operating System**: Dataproc 2.0-debian10, spark 3.1, scala 2.12 more details: https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-release-2.0
**CPU**: Yes
GPU: No

Hi team, I can't run the spark demo in Dataproc cluster,I tried the demo data and my own dataset, both fails for same reason. One thing to mention that the code works well when running locally on my laptop, but fails every time when submitting to yarn cluster.

I also find the similar issues raised before: https://github.com/catboost/catboost/issues/1765

## spark properties:

"spark.sql.shuffle.partitions":10,
"spark.default.parallelism":10,
"spark.task.cpus":8,
"spark.executor.cores":8,
"spark.scheduler.minRegisteredResourcesRatio": 1,
"spark.scheduler.maxRegisteredResourcesWaitingTime": "12m",
"spark.submit.deployMode": "client",
"spark.executor.instances":2,
"spark.executor.memory":"40g",
"spark.driver.memory":"20g",
"spark.driver.cores":4,
"spark.task.maxFailures":1

## Code
from pyspark.sql import Row,SparkSession
from pyspark.ml.linalg import Vectors, VectorUDT
from pyspark.sql.types import *
import catboost_spark
srcDataSchema = [
StructField("features", VectorUDT()),
StructField("label", StringType())
]

trainData = [
Row(Vectors.dense(0.1, 0.2, 0.11), "0"),
Row(Vectors.dense(0.97, 0.82, 0.33), "1"),
Row(Vectors.dense(0.13, 0.22, 0.23), "1"),
Row(Vectors.dense(0.8, 0.62, 0.0), "0")
]

trainDf = spark.createDataFrame(spark.sparkContext.parallelize(trainData), StructType(srcDataSchema))
trainPool = catboost_spark.Pool(trainDf)

evalData = [
Row(Vectors.dense(0.22, 0.33, 0.9), "1"),
Row(Vectors.dense(0.11, 0.1, 0.21), "0"),
Row(Vectors.dense(0.77, 0.0, 0.0), "1")
]

evalDf = spark.createDataFrame(spark.sparkContext.parallelize(evalData), StructType(srcDataSchema))
evalPool = catboost_spark.Pool(evalDf)

classifier = catboost_spark.CatBoostClassifier()

# train a model
model = classifier.fit(trainPool, evalDatasets=[evalPool])

# apply the model
predictions = model.transform(evalPool.data)
predictions.show()

## Logs
### dirver
```
24/09/30 03:50:21 INFO ai.catboost.spark.CatBoostClassifier: fit. partitionCount=2
24/09/30 03:50:21 INFO ai.catboost.spark.CatBoostClassifier: fit. train.prepareDatasetForTraining: start
24/09/30 03:50:21 INFO ai.catboost.spark.CatBoostClassifier: fit. train.prepareDatasetForTraining: finish
24/09/30 03:50:21 INFO ai.catboost.spark.CatBoostClassifier: fit. eval #0.prepareDatasetForTraining: start
24/09/30 03:50:21 INFO ai.catboost.spark.CatBoostClassifier: fit. eval #0.prepareDatasetForTraining: finish
24/09/30 03:50:21 INFO ai.catboost.spark.impl.CatBoostMasterWrapper: downloadQuantizedPoolToTempFiles for Learn Dataset: start
24/09/30 03:50:21 INFO ai.catboost.spark.impl.CatBoostMasterWrapper: loadQuantizedDatasets for Learn Dataset: start
24/09/30 03:50:23 INFO ai.catboost.spark.impl.CatBoostMasterWrapper: loadQuantizedDatasets for Learn Dataset: finish
24/09/30 03:50:23 INFO ai.catboost.spark.impl.CatBoostMasterWrapper: Learn Dataset: save loaded data to files: start
24/09/30 03:50:23 INFO ai.catboost.spark.impl.CatBoostMasterWrapper: Learn Dataset: save loaded data to files: finish
24/09/30 03:50:23 INFO ai.catboost.spark.impl.CatBoostMasterWrapper: downloadQuantizedPoolToTempFiles for Learn Dataset: finish
24/09/30 03:50:23 INFO ai.catboost.spark.impl.CatBoostMasterWrapper: downloadQuantizedPoolToTempFiles for Eval Dataset #0: start
24/09/30 03:50:23 INFO ai.catboost.spark.impl.CatBoostMasterWrapper: loadQuantizedDatasets for Eval Dataset #0: start
24/09/30 03:50:23 INFO ai.catboost.spark.impl.CatBoostMasterWrapper: loadQuantizedDatasets for Eval Dataset #0: finish
24/09/30 03:50:23 INFO ai.catboost.spark.impl.CatBoostMasterWrapper: Eval Dataset #0: save loaded data to files: start
24/09/30 03:50:23 INFO ai.catboost.spark.impl.CatBoostMasterWrapper: Eval Dataset #0: save loaded data to files: finish
24/09/30 03:50:23 INFO ai.catboost.spark.impl.CatBoostMasterWrapper: downloadQuantizedPoolToTempFiles for Eval Dataset #0: finish
24/09/30 03:50:23 INFO ai.catboost.spark.CatBoostClassifier: fit. TrainingDriver listening port = 46811
24/09/30 03:50:23 INFO ai.catboost.spark.CatBoostClassifier: fit. Training started
24/09/30 03:50:23 INFO ai.catboost.spark.TrainingDriver: started
24/09/30 03:50:23 INFO ai.catboost.spark.TrainingDriver: wait for workers info
24/09/30 03:50:23 INFO ai.catboost.spark.UpdatableWorkersInfo: received workerInfo=partitionId=1, partitionSize=0, host=10.174.160.34, port=40057
24/09/30 03:50:23 WARN org.apache.spark.storage.BlockManagerMasterEndpoint: No more replicas available for rdd_111_0 !
24/09/30 03:50:23 WARN org.apache.spark.storage.BlockManagerMasterEndpoint: No more replicas available for rdd_88_0 !
24/09/30 03:50:23 WARN org.apache.spark.storage.BlockManagerMasterEndpoint: No more replicas available for rdd_104_0 !
24/09/30 03:50:24 WARN org.apache.spark.deploy.yarn.YarnAllocator: Container from a bad node: container_e01_1727658604123_0008_01_000002 on host: lgb-xinghan-w-1. Exit status: 134. Diagnostics: [2024-09-30 03:50:23.991]Exception from container-launch.
Container id: container_e01_1727658604123_0008_01_000002
Exit code: 134

[2024-09-30 03:50:23.993]Container exited with a non-zero exit code 134. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
/bin/bash: line 1: 25934 Aborted /usr/lib/jvm/temurin-8-jdk-amd64/bin/java -server -Xmx40960m '-verbose:gc' '-XX:+PrintGCDetails' '-XX:+PrintGCDateStamps' -Djava.io.tmpdir=/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/tmp '-Dspark.driver.port=34647' '-Dspark.ui.port=0' '-Dspark.rpc.message.maxSize=512' -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/userlogs/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002 -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.YarnCoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@lgb-xinghan-w-0:34647 --executor-id 1 --hostname lgb-xinghan-w-1 --cores 8 --app-id application_1727658604123_0008 --resourceProfileId 0 --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/__app__.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/livy-rsc-0.8.0-incubating-SNAPSHOT.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/minlog-1.3.0.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/netty-all-4.1.47.Final.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/livy-api-0.8.0-incubating-SNAPSHOT.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/kryo-shaded-4.0.2.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/objenesis-2.5.1.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/spark-bigquery-with-dependencies_2.12-0.31.1.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/catboost-spark_3.1_2.12_1.2.7_with_dependencies-0.1.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/commons-codec-1.9.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/livy-core_2.12-0.8.0-incubating-SNAPSHOT.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/livy-repl_2.12-0.8.0-incubating-SNAPSHOT.jar > /var/log/hadoop-yarn/userlogs/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/stdout 2> /var/log/hadoop-yarn/userlogs/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/stderr
Last 4096 bytes of stderr :
24/09/30 03:49:57 INFO com.google.cloud.dataproc.DataprocSparkPlugin: Registered 110 executor metrics
24/09/30 03:50:23 INFO CatBoostWorker[partitionId=0]: processPartition: start
24/09/30 03:50:23 INFO CatBoostWorker[partitionId=0]: processPartition: get data providers: start
24/09/30 03:50:23 INFO CatBoostWorker[partitionId=0]: processPartition: get data providers: finish
24/09/30 03:50:23 INFO CatBoostWorker[partitionId=0]: processPartition: CreateTrainingDataForWorker: start
24/09/30 03:50:23 INFO CatBoostWorker[partitionId=0]: processPartition: CreateTrainingDataForWorker: finish
24/09/30 03:50:23 INFO ai.catboost.spark.TrainingDriver: Reserved port 33129 for CatBoost worker
24/09/30 03:50:23 INFO ai.catboost.spark.TrainingDriver: wait for CatBoost worker to start listening at port 33129
24/09/30 03:50:23 INFO CatBoostWorker[partitionId=0]: processPartition: start RunWorker
uncaught exception:
address -> 0x40218920410
what() -> "(Error 9: Bad file descriptor) util/network/socket.h:76: setsockopt() failed for reuse addr"
type -> TSystemError

[2024-09-30 03:50:23.994]Container exited with a non-zero exit code 134. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
/bin/bash: line 1: 25934 Aborted /usr/lib/jvm/temurin-8-jdk-amd64/bin/java -server -Xmx40960m '-verbose:gc' '-XX:+PrintGCDetails' '-XX:+PrintGCDateStamps' -Djava.io.tmpdir=/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/tmp '-Dspark.driver.port=34647' '-Dspark.ui.port=0' '-Dspark.rpc.message.maxSize=512' -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/userlogs/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002 -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.YarnCoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@lgb-xinghan-w-0:34647 --executor-id 1 --hostname lgb-xinghan-w-1 --cores 8 --app-id application_1727658604123_0008 --resourceProfileId 0 --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/__app__.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/livy-rsc-0.8.0-incubating-SNAPSHOT.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/minlog-1.3.0.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/netty-all-4.1.47.Final.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/livy-api-0.8.0-incubating-SNAPSHOT.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/kryo-shaded-4.0.2.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/objenesis-2.5.1.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/spark-bigquery-with-dependencies_2.12-0.31.1.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/catboost-spark_3.1_2.12_1.2.7_with_dependencies-0.1.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/commons-codec-1.9.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/livy-core_2.12-0.8.0-incubating-SNAPSHOT.jar --user-class-path file:/mnt/1/hadoop/yarn/nm-local-dir/usercache/livy/appcache/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/livy-repl_2.12-0.8.0-incubating-SNAPSHOT.jar > /var/log/hadoop-yarn/userlogs/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/stdout 2> /var/log/hadoop-yarn/userlogs/application_1727658604123_0008/container_e01_1727658604123_0008_01_000002/stderr
Last 4096 bytes of stderr :
24/09/30 03:49:57 INFO com.google.cloud.dataproc.DataprocSparkPlugin: Registered 110 executor metrics
24/09/30 03:50:23 INFO CatBoostWorker[partitionId=0]: processPartition: start
24/09/30 03:50:23 INFO CatBoostWorker[partitionId=0]: processPartition: get data providers: start
24/09/30 03:50:23 INFO CatBoostWorker[partitionId=0]: processPartition: get data providers: finish
24/09/30 03:50:23 INFO CatBoostWorker[partitionId=0]: processPartition: CreateTrainingDataForWorker: start
24/09/30 03:50:23 INFO CatBoostWorker[partitionId=0]: processPartition: CreateTrainingDataForWorker: finish
24/09/30 03:50:23 INFO ai.catboost.spark.TrainingDriver: Reserved port 33129 for CatBoost worker
24/09/30 03:50:23 INFO ai.catboost.spark.TrainingDriver: wait for CatBoost worker to start listening at port 33129
24/09/30 03:50:23 INFO CatBoostWorker[partitionId=0]: processPartition: start RunWorker
uncaught exception:
address -> 0x40218920410
what() -> "(Error 9: Bad file descriptor) util/network/socket.h:76: setsockopt() failed for reuse addr"
type -> TSystemError

.....
.....

.
24/09/30 03:50:24 ERROR org.apache.spark.scheduler.TaskSetManager: Task 0 in stage 60.0 failed 1 times; aborting job
24/09/30 03:50:24 INFO ai.catboost.spark.TrainingDriver: close updatableWorkersInfo
24/09/30 03:50:24 INFO ai.catboost.spark.UpdatableWorkersInfo: Shutdown remaining workers: start
24/09/30 03:50:24 INFO ai.catboost.spark.UpdatableWorkersInfo: Shutdown remaining workers: no remaining workers
24/09/30 03:50:24 INFO ai.catboost.spark.UpdatableWorkersInfo: Shutdown remaining workers: finish
24/09/30 03:50:24 INFO ai.catboost.spark.TrainingDriver: closed
24/09/30 03:50:24 INFO ai.catboost.spark.TrainingDriver: finished

```
### worker container
```
================= Logs for container_e01_1727658604123_0008_01_000002
24/09/30 03:49:57 INFO com.google.cloud.dataproc.DataprocSparkPlugin: Registered 110 executor metrics
24/09/30 03:50:23 INFO CatBoostWorker[partitionId=0]: processPartition: start
24/09/30 03:50:23 INFO CatBoostWorker[partitionId=0]: processPartition: get data providers: start
24/09/30 03:50:23 INFO CatBoostWorker[partitionId=0]: processPartition: get data providers: finish
24/09/30 03:50:23 INFO CatBoostWorker[partitionId=0]: processPartition: CreateTrainingDataForWorker: start
24/09/30 03:50:23 INFO CatBoostWorker[partitionId=0]: processPartition: CreateTrainingDataForWorker: finish
24/09/30 03:50:23 INFO ai.catboost.spark.TrainingDriver: Reserved port 33129 for CatBoost worker
24/09/30 03:50:23 INFO ai.catboost.spark.TrainingDriver: wait for CatBoost worker to start listening at port 33129
24/09/30 03:50:23 INFO CatBoostWorker[partitionId=0]: processPartition: start RunWorker
uncaught exception:
address -> 0x40218920410
what() -> "(Error 9: Bad file descriptor) util/network/socket.h:76: setsockopt() failed for reuse addr"
type -> TSystemError
```