Error while using mmlspark Lightgbm : Py4JJavaError('An error occurred while calling o527.fit.\n', JavaObject id=o680)

I am trying to use hyperopt to tune the hyperparameters for the LightGBM model and sometime I get this error `Py4JJavaError('An error occurred while calling o527.fit.\n', JavaObject id=o680)`.
In my previous run, out of 50 hyperopt trials, 34 failed due to above error while 14 trials were successful. Failed trials = 3, 4, 17-50. I am using lightbm from mmlspark and running an airflow DAG to execute it. I can't seem to understand why I am getting this error an dhow can I avoid this. The error block looks like this :

```
[2021-12-08 02:18:47,355] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 21/12/08 02:18:47 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_10_160 !
[2021-12-08 02:18:47,355] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 21/12/08 02:18:47 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_10_165 !
[2021-12-08 02:18:47,355] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 21/12/08 02:18:47 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_10_163 !
[2021-12-08 02:18:47,355] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 21/12/08 02:18:47 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_10_3 !
[2021-12-08 02:18:47,355] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 21/12/08 02:18:47 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_10_34 !
[2021-12-08 02:18:47,355] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 21/12/08 02:18:47 WARN storage.BlockManagerMasterEndpoint: No more replicas available for rdd_10_99 !
[2021-12-08 02:20:19,951] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 21/12/08 02:20:19 WARN scheduler.TaskSetManager: Lost task 30.1 in stage 59.0 (TID 2749, 100.66.217.123, executor 5): java.net.ConnectException: Connection refused (Connection refused)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.net.PlainSocketImpl.socketConnect(Native Method)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.net.Socket.connect(Socket.java:589)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.net.Socket.connect(Socket.java:538)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.net.Socket.(Socket.java:434)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.net.Socket.(Socket.java:211)
[2021-12-08 02:20:19,952] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at com.microsoft.ml.spark.lightgbm.TrainUtils$.getNetworkInitNodes(TrainUtils.scala:456)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at com.microsoft.ml.spark.lightgbm.TrainUtils$$anonfun$16.apply(TrainUtils.scala:549)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at com.microsoft.ml.spark.lightgbm.TrainUtils$$anonfun$16.apply(TrainUtils.scala:544)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at com.microsoft.ml.spark.core.env.StreamUtilities$.using(StreamUtilities.scala:29)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at com.microsoft.ml.spark.lightgbm.TrainUtils$.trainLightGBM(TrainUtils.scala:543)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at com.microsoft.ml.spark.lightgbm.LightGBMBase$$anonfun$7.apply(LightGBMBase.scala:225)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at com.microsoft.ml.spark.lightgbm.LightGBMBase$$anonfun$7.apply(LightGBMBase.scala:225)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:188)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:185)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
[2021-12-08 02:20:19,953] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:55)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.scheduler.Task.run(Task.scala:123)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
[2021-12-08 02:20:19,954] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
[2021-12-08 02:20:19,955] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
[2021-12-08 02:20:19,955] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[2021-12-08 02:20:19,955] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[2021-12-08 02:20:19,955] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.lang.Thread.run(Thread.java:748)
[2021-12-08 02:20:19,955] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold
[2021-12-08 02:20:31,380] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 21/12/08 02:20:31 WARN scheduler.TaskSetManager: Lost task 6.1 in stage 59.0 (TID 2748, 100.66.217.123, executor 5): java.net.ConnectException: Connection refused (Connection refused)
[2021-12-08 02:20:31,380] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.net.PlainSocketImpl.socketConnect(Native Method)
[2021-12-08 02:20:31,380] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
[2021-12-08 02:20:31,380] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
[2021-12-08 02:20:31,380] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.net.Socket.connect(Socket.java:589)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.net.Socket.connect(Socket.java:538)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.net.Socket.(Socket.java:434)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.net.Socket.(Socket.java:211)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at com.microsoft.ml.spark.lightgbm.TrainUtils$.getNetworkInitNodes(TrainUtils.scala:456)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at com.microsoft.ml.spark.lightgbm.TrainUtils$$anonfun$16.apply(TrainUtils.scala:549)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at com.microsoft.ml.spark.lightgbm.TrainUtils$$anonfun$16.apply(TrainUtils.scala:544)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at com.microsoft.ml.spark.core.env.StreamUtilities$.using(StreamUtilities.scala:29)
[2021-12-08 02:20:31,381] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at com.microsoft.ml.spark.lightgbm.TrainUtils$.trainLightGBM(TrainUtils.scala:543)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at com.microsoft.ml.spark.lightgbm.LightGBMBase$$anonfun$7.apply(LightGBMBase.scala:225)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at com.microsoft.ml.spark.lightgbm.LightGBMBase$$anonfun$7.apply(LightGBMBase.scala:225)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:188)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$5.apply(objects.scala:185)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
[2021-12-08 02:20:31,382] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:55)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.scheduler.Task.run(Task.scala:123)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[2021-12-08 02:20:31,383] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[2021-12-08 02:20:31,384] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold at java.lang.Thread.run(Thread.java:748)
[2021-12-08 02:20:31,384] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold
[2021-12-08 02:20:40,794] {base_task_runner.py:101} {PID:76 TID:140123783399168} INFO - Job 351492: Subtask run_hyperparam_search_with_fold 21/12/08 02:20:40 ERROR scheduler.TaskSetManager: Task 6 in stage 59.0 failed 8 times; aborting job
[2021-12-08 02:20:40,812] {logging_mixin.py:103} {PID:118 TID:140374674740992} INFO -

[2021-12-08 02:20:40,812] {logging_mixin.py:103} {PID:118 TID:140374674740992} INFO -
[2021-12-08 02:20:40,812] {logging_mixin.py:95} {PID:118 TID:140374674740992} INFO - [2021-12-08 02:20:40,812] {agg_modeling.py:562} {PID:118 TID:140374674740992} INFO - Exception while executing train_gbm (agg_modeling.py) : Py4JJavaError('An error occurred while calling o527.fit.\n', JavaObject id=o680)
[2021-12-08 02:20:40,812] {logging_mixin.py:95} {PID:118 TID:140374674740992} INFO -
[2021-12-08 02:20:40,813] {logging_mixin.py:103} {PID:118 TID:140374674740992} INFO -
0%| | 0/50 [24:31

Begin at TrainUtils.scala:456 and LightGBMBase.scala:225, then inspect the surrounding Spark logs for BlockManager replica loss and Connection refused. Reproduce the failing versus successful hyperopt trials in the Airflow DAG and document a reproducible cause and avoidance or handling; completion requires a confirmed explanation, not merely the Py4J wrapper.

Written by the indexing model from the issue text.