Errors Training on Large Dataset : LightGBM

Im trying to fit a classifier with LightGBM on a large dataset. There are about 900,000,000 rows and 40 columns, 7 of which are integers being treated as categorical.

- SynapseML Version: com.microsoft.azure:synapseml_2.12:0.9.5
- Spark Version 3,2
- Spark Platform: AWS EMR

The current cluster is

6 workers with 16 VCPUS and 64GB RAM each.

The call to Lightgbm is as follows:

```
lgb_estimator = LightGBMClassifier(objective ="binary", learningRate = 0.1, numIterations = 222,
categoricalSlotNames = ["cat1",
"cat2",
"cat3",
"cat4",
"cat5",
"cat6",
"cat7"],
numLeaves= 31,
probabilityCol='probs',
featuresCol='features',
labelCol='target',
useBarrierExecutionMode=True
)

lgbmModel = lgb_estimator.fit(df_train)
```

I have had various errors running on a 50% sample but it completes with a much smaller sample. I switched to useBarrierExecutionMode = True which resulted in not enough space on disk errors so I increased the volume on all the workers. I get errors that dont seem to helpful:

```
An error occurred while calling o109.fit.
: org.apache.spark.SparkException: Job aborted due to stage failure: Could not recover from a failed barrier ResultStage. Most recent failure reason: Stage failed because barrier task ResultTask(9, 3) finished unsuccessfully.
java.net.ConnectException: Connection refused (Connection refused)
```

Or when not using BarrierExecutionMode=True, something like:

```
An error occurred while calling o284.fit.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 19.0 failed 4 times, most recent failure: Lost task 0.3 in stage 19.0 (TID 8696) (ip-10-0-188-112.ec2.internal executor 3): java.net.ConnectException: Connection refused (Connection refused)
```

My questions:

1. Can this size of cluster support such a large dataset training with Lightgbm? My naïve thought was that it could but would be slow.
2. When dealing with large datasets, any recommendations on how to set Spark properties that may help?
3. Any suggestions on the cluster size needed to run this data?

AB#1833527