[SUPPORT] Broadcasting issue in Hudi 0.14.1

During the migration of a Spark job from Hudi 0.12.1 to Hudi 0.14.1, I faced an issue with broadcasting a small table. The total serialized output from all tasks exceeds the spark.driver.maxResultSize limit (2.0 GiB), which causes the job to abort. When I go back to the previous version the job works again.

**Environment Description**
* Hudi version : 0.14.1
* Spark version : 3.3
* Glue 4.0

**Stacktrace**

"2025-03-01 13:06:06,217 ERROR [Thread-11] datasources.FileFormatWriter (Logging.scala:logError(98)): Aborting job a32a43b3-a330-427b-8cc2-b44e197b1305.
org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 375 tasks (2.0 GiB) is bigger than spark.driver.maxResultSize (2.0 GiB)
at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2863) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2799) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2798) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) ~[scala-library-2.12.15.jar:?]
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) ~[scala-library-2.12.15.jar:?]
at scala.collection.mutable.ArrayBuffer.forea"
"ch(ArrayBuffer.scala:49) ~[scala-library-2.12.15.jar:?]
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2798) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1239) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1239) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at scala.Option.foreach(Option.scala:407) ~[scala-library-2.12.15.jar:?]
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1239) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3051) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2993) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.scheduler."
"DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2982) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.scheduler.DAGScheduler."
"runJob(DAGScheduler.scala:1009) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2229) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2250) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2269) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2294) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1021) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.rdd.RDD.withScope(RDD.scala:406) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at"
" org.apache.spark.rdd.RDD.collect(RDD.scala:1020) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.SparkPlan.executeCollectIterator(SparkPlan.scala:458) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec.org$apache$spark$sql$execution$exchange$BroadcastExchangeExec$$doComputeRelation(BroadcastExchangeExec.scala:179) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec$$anon$1.doCompute(BroadcastExchangeExec.scala:172) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec$$anon$1.doCompute(BroadcastExchangeExec.scala:168) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.AsyncDriverOperation.$anonfun$compute$1(AsyncDriverOperation.scala:73) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPla"
"nningTracker.scala:107) ~[spark-catalyst_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:216) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withExecutionId$1(SQLExecution.scala:199) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:245) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.SQLExecution$.withExecutionId(SQLExecution.scala:196) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.AsyncDriverOperation.compute(AsyncDriverOperation.scala:67) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.AsyncDriverOperation.$anonfun$computeFuture$1(AsyncDriverOperation.scala:53"
") ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$1(SQLExecution.scala:267) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_442]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_442]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_442]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_442]
at org.apache.spark.sql.execution.adaptive.AdaptiveExecutor.checkNoFailures(AdaptiveExecutor.scala:154) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.adaptive.AdaptiveExecutor.doRun(AdaptiveExecutor.scala:88) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.adaptive.AdaptiveExecutor.tryRunningAndGetFuture(AdaptiveExecutor.scala:66) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.executio"
"n.adaptive.AdaptiveExecutor.execute(AdaptiveExecutor.scala:57) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.$anonfun$getFinalPhysicalPlan$1(AdaptiveSparkPlanExec.scala:237) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.getFinalPhysicalPlan(AdaptiveSparkPlanExec.scala:236) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.withFinalPlanUpdate(AdaptiveSparkPlanExec.scala:505) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.doExecute(AdaptiveSparkPlanExec.scala:491) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:213) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3."
"3.0-amzn-1]
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:251) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:248) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:209) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:213) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:193) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at or"
"g.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:103) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107) ~[spark-catalyst_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:114) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$7(SQLExecution.scala:139) ~[spark-sql_2.12-3.3.0-a"
"mzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107) ~[spark-catalyst_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:139) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:245) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:138) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.a"
"pache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:100) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:96) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:615) ~[spark-catalyst_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:177) ~[spark-catalyst_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:615) ~[spark-catalyst_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30) ~[spark-catalyst_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql"
".catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267) ~[spark-catalyst_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263) ~[spark-catalyst_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30) ~[spark-catalyst_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30) ~[spark-catalyst_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:591) ~[spark-catalyst_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:96) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:83) ~[spark-s"
"ql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:81) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:124) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:793) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:"
"1.8.0_442]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_442]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_442]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_442]
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) ~[py4j-0.10.9.5.jar:?]
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) ~[py4j-0.10.9.5.jar:?]
at py4j.Gateway.invoke(Gateway.java:282) ~[py4j-0.10.9.5.jar:?]
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) ~[py4j-0.10.9.5.jar:?]
at py4j.commands.CallCommand.execute(CallCommand.java:79) ~[py4j-0.10.9.5.jar:?]
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) ~[py4j-0.10.9.5.jar:?]
at py4j.ClientServerConnection.run(ClientServerConnection.java:106) ~[py4j-0.10.9.5.jar:?]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_442]"
"2025-03-01 13:06:06,226 INFO [Thread-11] output.FileSystemOptimizedCommitter (FileSystemOptimizedCommitter.java:abortJob(123)): Nothing to do when job is aborted"
"2025-03-01 13:06:06,227 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 329.0 in stage 6.0 (TID 337) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,227 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 363.0 in stage 6.0 (TID 371) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,227 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 369.0 in stage 6.0 (TID 377) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,228 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 324.0 in stage 6.0 (TID 332) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,230 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 335.0 in stage 6.0 (TID 343) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,231 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 358.0 in stage 6.0 (TID 366) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,232 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 340.0 in stage 6.0 (TID 348) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,233 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 374.0 in stage 6.0 (TID 382) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,233 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 351.0 in stage 6.0 (TID 359) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,235 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 346.0 in stage 6.0 (TID 354) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,235 INFO [dispatcher-BlockManagerMaster] storage.BlockManagerInfo (Logging.scala:logInfo(61)): Added broadcast_3_piece0 in memory on 172.39.32.22:37289 (size: 39.5 KiB, free: 47.7 GiB)"
"2025-03-01 13:06:06,236 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 388.0 in stage 6.0 (TID 396) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,236 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 419.0 in stage 6.0 (TID 427) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,236 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 418.0 in stage 6.0 (TID 426) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,237 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 415.0 in stage 6.0 (TID 423) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,237 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 380.0 in stage 6.0 (TID 388) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,238 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 416.0 in stage 6.0 (TID 424) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,239 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 412.0 in stage 6.0 (TID 420) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,255 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 396.0 in stage 6.0 (TID 404) (172.39.32.22 exec"
utor 1): TaskKilled (Stage cancelled)
"2025-03-01 13:06:06,276 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 417.0 in stage 6.0 (TID 425) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,281 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 420.0 in stage 6.0 (TID 428) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,289 INFO [task-result-getter-2] scheduler.TaskSetManager (Logging.scala:logInfo(61)): Finished task 292.0 in stage 6.0 (TID 300) in 2351 ms on 172.39.32.22 (executor 1) (372/770)"
"2025-03-01 13:06:06,290 INFO [dispatcher-BlockManagerMaster] storage.BlockManagerInfo (Logging.scala:logInfo(61)): Removed taskresult_300 on 172.39.32.22:37289 in memory (size: 30.1 MiB, free: 47.8 GiB)"
"2025-03-01 13:06:06,296 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 422.0 in stage 6.0 (TID 430) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,212 ERROR [spark-listener-group-shared] glueexceptionanalysis.GlueExceptionAnalysisListener (Logging.scala:logError(9)): [Glue Exception Analysis] {""Event"":""GlueExceptionAnalysisStageFailed"",""Timestamp"":1740834366206,""Failure Reason"":""Job aborted due to stage failure: Total size of serialized results of 375 tasks (2.0 GiB) is bigger than spark.driver.maxResultSize (2.0 GiB)"",""Stack Trace"":[],""Stage ID"":6,""Stage Attempt ID"":0,""Number of Tasks"":770}"
"2025-03-01 13:06:06,303 ERROR [spark-listener-group-shared] glueexceptionanalysis.GlueExceptionAnalysisListener (Logging.scala:logError(9)): [Glue Exception Analysis] {""Event"":""GlueExceptionAnalysisJobFailed"",""Timestamp"":1740834366301,""Failure Reason"":""org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 375 tasks (2.0 GiB) is bigger than spark.driver.maxResultSize (2.0 GiB)"",""Stack Trace"":[{""Declaring Class"":""org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 375 tasks (2.0 GiB) is bigger than spark.driver.maxResultSize (2.0 GiB))"",""Method Name"":""TopLevelFailedReason"",""File Name"":""TopLevelFailedReason"",""Line Number"":-1}],""Job Id"":6,""Job Result"":""JobFailed"",""Failed Stage Id"":-1}"
"2025-03-01 13:06:06,309 WARN [task-result-getter-2] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 385.0 in stage 6.0 (TID 393) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,310 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 421.0 in stage 6.0 (TID 429) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,311 WARN [task-result-getter-2] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 424.0 in stage 6.0 (TID 432) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,349 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 426.0 in stage 6.0 (TID 434) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,351 INFO [task-result-getter-0] scheduler.TaskSetManager (Logging.scala:logInfo(61)): Finished task 303.0 in stage 6.0 (TID 311) in 2229 ms on 172.39.32.22 (executor 1) (373/770)"
"2025-03-01 13:06:06,352 INFO [dispatcher-BlockManagerMaster] storage.BlockManagerInfo (Logging.scala:logInfo(61)): Removed taskresult_311 on 172.39.32.22:37289 in memory (size: 31.4 MiB, free: 47.8 GiB)"
"2025-03-01 13:06:06,373 WARN [task-result-getter-2] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 425.0 in stage 6.0 (TID 433) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,378 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 409.0 in stage 6.0 (TID 417) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,382 WARN [task-result-getter-0] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 423.0 in stage 6.0 (TID 431) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,409 WARN [task-result-getter-2] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 406.0 in stage 6.0 (TID 414) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,439 INFO [task-result-getter-3] scheduler.TaskSetManager (Logging.scala:logInfo(61)): Finished task 313.0 in stage 6.0 (TID 321) in 2149 ms on 172.39.32.22 (executor 1) (374/770)"
"2025-03-01 13:06:06,441 INFO [dispatcher-BlockManagerMaster] storage.BlockManagerInfo (Logging.scala:logInfo(61)): Removed taskresult_321 on 172.39.32.22:37289 in memory (size: 32.0 MiB, free: 47.8 GiB)"
"2025-03-01 13:06:06,490 WARN [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 391.0 in stage 6.0 (TID 399) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,521 WARN [task-result-getter-0] scheduler.TaskSetManager (Logging.scala:logWarning(73)): Lost task 401.0 in stage 6.0 (TID 409) (172.39.32.22 executor 1): TaskKilled (Stage cancelled)"
"2025-03-01 13:06:06,521 INFO [task-result-getter-0] scheduler.TaskSchedulerImpl (Logging.scala:logInfo(61)): Removed TaskSet 6.0, whose tasks have all completed, from pool "
"2025-03-01 13:06:06,581 INFO [task-result-getter-2] scheduler.TaskSetManager (Logging.scala:logInfo(61)): Finished task 0.0 in stage 7.0 (TID 435) in 383 ms on 172.39.32.22 (executor 1) (1/1)"
"2025-03-01 13:06:06,582 INFO [task-result-getter-2] scheduler.TaskSchedulerImpl (Logging.scala:logInfo(61)): Removed TaskSet 7.0, whose tasks have all completed, from pool "
"2025-03-01 13:06:06,582 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler (Logging.scala:logInfo(61)): ResultStage 7 ($anonfun$withThreadLocalCaptured$1 at FutureTask.java:266) finished in 8.225 s"
"2025-03-01 13:06:06,582 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler (Logging.scala:logInfo(61)): Job 7 is finished. Cancelling potential speculative or zombie tasks for this job"
"2025-03-01 13:06:06,582 INFO [dag-scheduler-event-loop] scheduler.TaskSchedulerImpl (Logging.scala:logInfo(61)): Killing all running tasks in stage 7: Stage finished"
"2025-03-01 13:06:06,715 ERROR [main] glue.ProcessLauncher (Logging.scala:logError(77)): Error from Python:Traceback (most recent call last):
File ""/tmp/bdr-fees.py"", line 203, in
main()
File ""/tmp/bdr-fees.py"", line 200, in main
transform.run()
File ""/tmp/bdr-fees.py"", line 195, in run
self.export_data()
File ""/tmp/bdr-fees.py"", line 189, in export_data
bdr_fees_df.save_data_as_parquet()
File ""/tmp/localPyFiles-8fd9d98c-07f3-40b1-aca0-43c9732d6d92/DataFrameToGlueParquet.py"", line 274, in save_data_as_parquet
self.df.write.mode(self.mode).parquet(self.target_path)
File ""/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py"", line 1140, in parquet
self._jwrite.parquet(path)
File ""/opt/amazon/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py"", line 1321, in __call__
return_value = get_return_value(
File ""/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py"", line 190, in deco
return f(*a, **kw)
File ""/opt/amazon/spark/pyth"
"on/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py"", line 326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o287.parquet.
: org.apache.spark.SparkException: Job aborted.
at org.apache.spark.sql.errors.QueryExecutionErrors$.jobAbortedError(QueryExecutionErrors.scala:638)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:279)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:193)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:103)
at org.apache.s"
"park.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224)
at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:114)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$7(SQLExecution.scala:139)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:139)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:245)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:138)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
at org.apache"
".spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:100)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:96)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:615)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:177)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:615)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
"
"at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:591)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:96)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:83)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:124)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:793)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ"
"e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 375 tasks (2.0 GiB) is bigger than spark.driver.maxResultSize (2.0 GiB)
at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2863)
at "
"org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2799)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2798)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2798)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1239)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1239)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1239)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3051)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2993)"
" at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2982)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:1009)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2229)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2250)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2269)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2294)
at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1021)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:406)
at org.apache.spark.rdd.RDD.collect(RDD.scala:1020)
at org.apache.spark.sql.execution.SparkPlan.executeCollectIterator(SparkPlan.scala:458)
at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec.org$apache$spark$sql$"
"execution$exchange$BroadcastExchangeExec$$doComputeRelation(BroadcastExchangeExec.scala:179)
at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec$$anon$1.doCompute(BroadcastExchangeExec.scala:172)
at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec$$anon$1.doCompute(BroadcastExchangeExec.scala:168)
at org.apache.spark.sql.execution.AsyncDriverOperation.$anonfun$compute$1(AsyncDriverOperation.scala:73)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:216)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withExecutionId$1(SQLExecution.scala:199)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:245)
at org.apache.spark.sql.execution.SQLExecution$.withExecutionId(SQLExecution.scala:196)
at org.apache.spark.sql.execution.As"
"yncDriverOperation.compute(AsyncDriverOperation.scala:67)
at org.apache.spark.sql.execution.AsyncDriverOperation.$anonfun$computeFuture$1(AsyncDriverOperation.scala:53)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$1(SQLExecution.scala:267)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
at org.apache.spark.sql.execution.adaptive.AdaptiveExecutor.checkNoFailures(AdaptiveExecutor.scala:154)
at org.apache.spark.sql.execution.adaptive.AdaptiveExecutor.doRun(AdaptiveExecutor.scala:88)
at org.apache.spark.sql.execution.adaptive.AdaptiveExecutor.tryRunningAndGetFuture(AdaptiveExecutor.scala:66)
at org.apache.spark.sql.execution.adaptive.AdaptiveExecutor.execute(AdaptiveExecutor.scala:57)
at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.$an"
"onfun$getFinalPhysicalPlan$1(AdaptiveSparkPlanExec.scala:237)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.getFinalPhysicalPlan(AdaptiveSparkPlanExec.scala:236)
at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.withFinalPlanUpdate(AdaptiveSparkPlanExec.scala:505)
at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.doExecute(AdaptiveSparkPlanExec.scala:491)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:213)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:251)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:248)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:209)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:213)
... 48 more

Start by reproducing the migration from Hudi 0.12.1 to 0.14.1 in the stated Spark 3.3 and Glue 4.0 environment. Trace the failure through RDD.collect and BroadcastExchangeExec, then compare the two Hudi versions and verify that the broadcast no longer exceeds spark.driver.maxResultSize.

Written by the indexing model from the issue text.