Driver for cluster gets stuck at "TaskSchedulerImpl: Adding task set 0.0 with ..."

I have configured two clusters of six r5d.8xlarge nodes on EC2 and manually configured Spark 3.2.2 with hadoop 3.2, which I downloaded from the Apache Spark website.

On the baseline cluster, I configured vanilla Spark, and I'm able to run all TPC-DS queries. The main things I had to add to get that to work were a few jars to enable accessing S3:

```
hadoop-aws-3.3.1.jar
hadoop-common-3.3.1.jar
aws-java-sdk-bundle-1.11.901.jar
```

And I put this in the spark-defaults.conf file for every node:

```
spark.driver.memory 5g
spark.executor.memory 240g
```

The second cluster is basically the same, but I added the following jars to spark-3.2.2-bin-hadoop3.2/jars on all nodes:

```
hadoop-aws-3.3.1.jar
hadoop-common-3.3.1.jar
aws-java-sdk-bundle-1.11.901.jar
gluten-1.0.0-SNAPSHOT-jar-with-dependencies.jar
protobuf-java-3.19.4.jar
```

And I had to install some system libraries manually. This is from my script that configures all the nodes:

```
run_command_all "sudo apt install -y maven build-essential cmake libssl-dev libre2-dev libcurl4-openssl-dev clang lldb lld libz-dev git ninja-build uuid-dev"
run_command_all "sudo apt-get -y install build-essential g++ python-dev autotools-dev libicu-dev libbz2-dev libboost-all-dev"
run_command_all "sudo apt install -y libdouble-conversion-dev"
run_command_all "sudo apt install -y g++ cmake ccache ninja-build checkinstall git libssl-dev libboost-all-dev libdouble-conversion-dev libgoogle-glog-dev libbz2-dev libgflags-dev libgtest-dev libgmock-dev libevent-dev liblz4-dev libzstd-dev libre2-dev libsnappy-dev liblzo2-dev bison flex tzdata wget"
```

And finally, the spark-defaults.conf file is different:

```
spark.driver.memory 5g
spark.executor.memory 120g
spark.plugins io.glutenproject.GlutenPlugin
spark.gluten.sql.columnar.backend.lib velox
spark.memory.offHeap.enabled true
spark.memory.offHeap.size 120g
spark.gluten.sql.columnar.forceshuffledhashjoin true
spark.shuffle.manager org.apache.spark.shuffle.sort.ColumnarShuffleManager
```

After all this, I can get master and worker nodes to start and find each other. However, when I try to run a query on the cluster, the driver hangs. None of the worker or master log files indicate any problems. The driver just stops forever. Here's the tail end of what is printed by the driver:

```
22/10/25 21:36:09 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir.
22/10/25 21:36:09 INFO SharedState: Warehouse path is 'file:/home/ubuntu/TpcdsApp/spark-warehouse'.
22/10/25 21:36:11 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.31.2.206:40206) with ID 2, ResourceProfileId 0
22/10/25 21:36:11 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.31.7.85:58432) with ID 3, ResourceProfileId 0
22/10/25 21:36:11 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.31.0.50:54604) with ID 4, ResourceProfileId 0
22/10/25 21:36:11 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.31.12.41:56384) with ID 0, ResourceProfileId 0
22/10/25 21:36:11 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.31.3.35:33206) with ID 1, ResourceProfileId 0
22/10/25 21:36:11 INFO BlockManagerMasterEndpoint: Registering block manager 172.31.7.85:46691 with 183.8 GiB RAM, BlockManagerId(3, 172.31.7.85, 46691, None)
22/10/25 21:36:11 INFO BlockManagerMasterEndpoint: Registering block manager 172.31.2.206:41911 with 183.8 GiB RAM, BlockManagerId(2, 172.31.2.206, 41911, None)
22/10/25 21:36:11 INFO BlockManagerMasterEndpoint: Registering block manager 172.31.0.50:35843 with 183.8 GiB RAM, BlockManagerId(4, 172.31.0.50, 35843, None)
22/10/25 21:36:11 INFO BlockManagerMasterEndpoint: Registering block manager 172.31.12.41:46597 with 183.8 GiB RAM, BlockManagerId(0, 172.31.12.41, 46597, None)
22/10/25 21:36:11 INFO BlockManagerMasterEndpoint: Registering block manager 172.31.3.35:33087 with 183.8 GiB RAM, BlockManagerId(1, 172.31.3.35, 33087, None)
22/10/25 21:36:11 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
22/10/25 21:36:11 INFO MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
22/10/25 21:36:11 INFO MetricsSystemImpl: s3a-file-system metrics system started
22/10/25 21:36:13 INFO HadoopFSUtils: Listing leaf files and directories in parallel under 2004 paths. The first several paths are: s3a://theosib-v3-tpcds-hive/warehouse/theosib_tpcds_3000_hive/data/store_returns/sr_returned_date_sk=2450820, s3a://theosib-v3-tpcds-hive/warehouse/theosib_tpcds_3000_hive/data/store_returns/sr_returned_date_sk=2450821, s3a://theosib-v3-tpcds-hive/warehouse/theosib_tpcds_3000_hive/data/store_returns/sr_returned_date_sk=2450822, s3a://theosib-v3-tpcds-hive/warehouse/theosib_tpcds_3000_hive/data/store_returns/sr_returned_date_sk=2450823, s3a://theosib-v3-tpcds-hive/warehouse/theosib_tpcds_3000_hive/data/store_returns/sr_returned_date_sk=2450824, s3a://theosib-v3-tpcds-hive/warehouse/theosib_tpcds_3000_hive/data/store_returns/sr_returned_date_sk=2450825, s3a://theosib-v3-tpcds-hive/warehouse/theosib_tpcds_3000_hive/data/store_returns/sr_returned_date_sk=2450826, s3a://theosib-v3-tpcds-hive/warehouse/theosib_tpcds_3000_hive/data/store_returns/sr_returned_date_sk=2450827, s3a://theosib-v3-tpcds-hive/warehouse/theosib_tpcds_3000_hive/data/store_returns/sr_returned_date_sk=2450828, s3a://theosib-v3-tpcds-hive/warehouse/theosib_tpcds_3000_hive/data/store_returns/sr_returned_date_sk=2450829.
22/10/25 21:36:14 INFO SparkContext: Starting job: sql at TpcdsApp.scala:158
22/10/25 21:36:14 INFO DAGScheduler: Got job 0 (sql at TpcdsApp.scala:158) with 2004 output partitions
22/10/25 21:36:14 INFO DAGScheduler: Final stage: ResultStage 0 (sql at TpcdsApp.scala:158)
22/10/25 21:36:14 INFO DAGScheduler: Parents of final stage: List()
22/10/25 21:36:14 INFO DAGScheduler: Missing parents: List()
22/10/25 21:36:14 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[2] at sql at TpcdsApp.scala:158), which has no missing parents
22/10/25 21:36:14 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 100.7 KiB, free 122.5 GiB)
22/10/25 21:36:14 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 36.0 KiB, free 122.5 GiB)
22/10/25 21:36:14 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-31-5-233.us-east-2.compute.internal:46845 (size: 36.0 KiB, free: 122.5 GiB)
22/10/25 21:36:14 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1478
22/10/25 21:36:14 INFO DAGScheduler: Submitting 2004 missing tasks from ResultStage 0 (MapPartitionsRDD[2] at sql at TpcdsApp.scala:158) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
22/10/25 21:36:14 INFO TaskSchedulerImpl: Adding task set 0.0 with 2004 tasks resource profile 0
```