spark-ci walltime improvements

### Context
`spark-ci` dominates the overall time CI takes to run. Based on [most recent stats](https://github.com/apache/iceberg/actions/metrics/performance), it averages 1h5m while the next slowest ci job (`flink-ci`) averages around only 21m.
This means PRs must wait at least 1h for `spark-ci` to complete before merging.

A couple of things we can do to improve the overall time it takes to run `spark-ci`
1. fix the gradle cache. reduce time to download dependencies. fixed in https://github.com/apache/iceberg/pull/16356
2. parallelize execution. enabling `-DtestParallelism=auto` for all tests, following `flink-ci`. And split `spark-ci` into 2 concurrent jobs. Fixed in https://github.com/apache/iceberg/pull/16357. This reduced Spark CI walltime from ~84 min to ~58 min (−31%)

### Ask
One more potential fix is to reduce the runtime of the tests inside `spark-ci`. I've pulled data on how long each test class takes to run. See https://gist.github.com/kevinjqliu/86827bedf9d02adad36881f476484208#top-25--core-spark for more details.
Some of the tests are inherently slow, but exacerbated by parameterized matrix that dramatically multiples the number of runs.

Is it worth to dig into these tests and see if there are any optimizations?