Codec [lz4] is not available. Consider setting spark.io.compression.codec=snappy

When using DatasetSuiteBase in unit tests and doing operations in parallel on dataframes, getting the below error. If we remove the parallel operations on the dataframes, it works well.

[info] java.lang.IllegalArgumentException: Codec [lz4] is not available. Consider setting spark.io.compression.codec=snappy
[info] at org.apache.spark.io.CompressionCodec$.$anonfun$createCodec$2(CompressionCodec.scala:92)
[info] at scala.Option.getOrElse(Option.scala:189)
[info] at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:91)
[info] at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:78)
[info] at org.apache.spark.sql.execution.SparkPlan.decodeUnsafeRows(SparkPlan.scala:369)
[info] at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:497)
[info] at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:429)
[info] at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:48)
[info] at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3715)
[info] at org.apache.spark.sql.Dataset.$anonfun$head$1(Dataset.scala:2728)

Since DatasetSuiteBase automatically gives us a spark context we are not able to set the spark.io.compression.codec to snappy as it has to be set before the spark context is created.

Start with DatasetSuiteBase and the Spark context creation path, then reproduce the failure using parallel DataFrame operations in a unit test. Trace how the context is configured before creation and define done as parallel operations completing without the lz4 codec error.

Written by the indexing model from the issue text.