chore(spark): Update major/minor versions for 26.11.0

Inizia da boil-config.toml e spark/upload_new_spark_version.sh, quindi esamina i riferimenti indicati a getting_started, kuttl, supported-versions e demo per gli aggiornamenti di Spark, JVM, Hadoop e delle dipendenze. Crea l’immagine con il comando boil fornito ed esegui gli smoke test indicati. L’attività è completata quando gli aggiornamenti applicabili e le pull request collegate sono completi, l’immagine viene creata, i test hanno esito positivo e vengono aggiunte le note di rilascio.

Part of https://github.com/stackabletech/docker-images/issues/1641.

• Add: 4.2.0
• Remove: 3.5.8, 4.1.1

[!TIP]
Please add to the Stackable Engineering project.

Update tasks

• Update boil-config.toml to reflect the agreed upon versions in the spreadsheet (including the removal of old versions).
• Upload new version (see spark/upload_new_spark_version.sh).
• Update boil-config.toml to the latest supported version of JVM (base and devel).
• Update boil-config.toml to the latest supported version of Hadoop.
• Ensure that the transitive dependencies in boil-config.toml are compatible with the Spark version(s).
• Update other dependencies if applicable (eg: python, jmx_exporter, aws_java_sdk_bundle, etc).
• Check other operators (getting_started / kuttl / supported-versions) for usage of the versions. Add the PR(s) to the list below.
• Update the version in demos. Add the PR(s) to the list below.
• Update the version in the SparkApplication within the stackable-airflow/dags private repository.

Related Pull Requests

[!TIP]
Delete any items that do not apply so that all applicable items can be checked.
For example, if you add release notes to the documentation repository, you do not need the latter two criteria.

• Link to the docker-images PR (product update)
• Link to the operator PR (getting_started / kuttl / supported-versions)
• Link to any other operator PRs (getting_started / kuttl)
• Link to demo PR (raise against the main branch)
• Link to the Release Notes PR in the documentation repo (if not a comment below)

Acceptance

[!TIP]
This list should be completed by the assignee(s), once respective PRs have been merged. Once all items have been
checked, the issue can be moved into Development: Done.

• Can build image (either locally, or in CI)
• All kuttl tests pass (either locally, or in CI)
• Release notes added to documentation and linked as a PR above
• Release notes written in a comment below
• Applicable release-note label added to this issue

# Where x.y.z is the new version added in this PR
boil build spark-k8s=x.y.z --strip-architecture --load

kind load docker-image oci.stackable.tech/sdp/spark-k8s:x.y.z-stackable0.0.0-dev

# Change directory into the spark-k8s-operator repository and update the
# product version in tests/test-definition.yaml
./scripts/run-tests --test-suite smoke-latest # or similar

Please consider updating this template if these instructions are wrong, or
could be made clearer.