LightGBMClassifier, LightGBMRegressor hang indefinitely without error at fit()

LightGBMClassifier and LightGBMRegressor both hang indefinitely most of the time on Databricks. My 6 node cluster has Databricks runtime 5.5, Scala 2.11, Spark 2.4.3, and mmlspark_2.11:0.17.dev27. I started using the development version of mmlspark after seeing comments claiming several issues in 0.17 were fixed there, but the problem persists. Another comment suggested adding useBarrierExecutionMode=True, so I did that as well, but that also did not fix the problem.

Usually there is no error, it just hangs on fit() for hours or days until I kill the job, with nothing output to stdout or stderr. Occasionally it fails with "connection refused" error, but as stated elsewhere, this is a generic error that does not point to the root cause.

A few times the classifier successfully trained a model, which took about 1 second per iteration for the same inputs. The inconsistency, lack of error messages, and occasional success for the same inputs make this difficult to debug. I will update if I get a more informative error message.