Make and keep production and test code working with "up to date" Java versions

### Describe the bug

Polaris, as a server, should be able run run recent Java versions for various reasons (GC improvements, heap usage improvements, OS integration improvements, etc etc etc).

"Client" libraries (think: Iceberg, Spark, Hadoop) however are known to cause problems with "newest" Java versions and/or recent library versions for various reasons.

Polaris relies on Hadoop and Spark for its tests and we are therefore limited to only run on Java versions that those libraries support / work with. It usually takes a lot of time until all these libraries are all compatible with "newer" Java versions - often lagging behind a few Java versions.

## Goals

1. Tests should not rely on Hadoop or Spark, at best not even Iceberg, but only on Polaris code
2. If tests (think: integration tests) really need Hadoop and/or Spark, those should run separately using a "compatible" Java version to allow Polaris to leverage modern Java versions.

## Polaris tests using Hadoop fail with Java 23

Background: The whole Java `SecurityManager` stuff was deprecated in Java 17 via [JEP 411](https://openjdk.org/jeps/411) (April 2021). Alternatives (`Subject.getSubject()` --> `Subject.current()`) have been [introduced with Java 18](https://bugs.openjdk.org/browse/JDK-8267108).

[JEP 486](https://openjdk.org/jeps/486) is about to _permanently_ disable the security manager, even the workaround to set `java.security.manager=allow` will not work beginning with Java 24 (the JVM won't start).

There's [HADOOP-19212](https://issues.apache.org/jira/browse/HADOOP-19212) (opened June 2024) with sadly rather [stale PR](https://github.com/apache/hadoop/pull/7081).

Workaround for Java 23: #723

Start with the Hadoop- and Spark-dependent tests described in the issue and review workaround #723, then compare the Java versions those dependencies support. Determine which tests can run using only Polaris code and which integration tests require a compatible Java version. Done means production and test execution support recent Java versions without the Java 23/24 SecurityManager failure.

Written by the indexing model from the issue text.