Cleanup spark benchmarks

Currently spark benchmarks rely on an old version that has a lot of bugs. These bugs have been fixed in never versions meaning we can remove or update a lot of our scripts and make spark benchmarks much nicer to run

Start by locating the scripts and configuration used to run the Spark benchmarks, then identify which Spark version they currently rely on. Review the available newer-version fixes and determine which scripts can be removed or updated. Done means the benchmarks run cleanly with the newer Spark version and the obsolete scripts are gone.

Written by the indexing model from the issue text.