[Bug] Legacy Iceberg tables fail with Kyuubi 1.11.1 and Iceberg 1.10.0

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

After upgrading our stack from:
```text
Spark 3.5.6
Kyuubi 1.9.4
Iceberg 1.5.2
Hive Metastore 4.0.0
```
to:
```text
Spark 3.5.9
Kyuubi 1.11.1
Iceberg 1.10.0
Hive Metastore 4.0.0
```
existing Iceberg tables created with the previous stack can no longer be listed through Kyuubi. The affected tables are registered in Hive Metastore with:
```text
storage_handler = org.apache.iceberg.mr.hive.HiveIcebergStorageHandler
```
The previous Spark classpath contained:
```text
iceberg-spark-runtime-3.5_2.12-1.5.2.jar
iceberg-hive-runtime-1.5.2.jar
```
while the new one contains only:
```text
iceberg-spark-runtime-3.5_2.12-1.10.0.jar
```

## Error
A `GetTables` request fails with:
```text
java.lang.ClassNotFoundException: org.apache.iceberg.mr.hive.HiveIcebergSerDe
java.lang.ClassNotFoundException: org.apache.iceberg.mr.hive.HiveIcebergStorageHandler
```
The failure occurs in:
```text
Kyuubi
→ SparkCatalogUtils.getCatalogTablesOrViews()
→ Spark HiveClient
→ HiveExternalCatalog
→ HiveUtils.getStorageHandler()
```
Even a `CREATE TABLE ... USING iceberg` through the Kyuubi Spark SQL engine can fail before the table is created, because Spark/Kyuubi resolves the Hive catalog and calls `SparkCatalogUtils.getCatalogTablesOrViews()`, which eventually invokes `HiveClientImpl.getTablesByName()` to convert the existing Hive tables to `CatalogTable` and attempts to load the legacy `HiveIcebergStorageHandler` and `HiveIcebergSerDe` classes, which are no longer available in the Iceberg 1.10.0 runtime.

So the presence of existing Iceberg tables registered with the legacy `storage_handler` is enough to prevent normal catalog operations in the new Spark/Iceberg environment.

## Question
Is this expected with Kyuubi 1.11.1 with Iceberg 1.10.0?
It would be useful to understand whether `SparkCatalogUtils.getCatalogTablesOrViews()` is expected to resolve Hive storage handlers during `GetTables`, or whether legacy Iceberg Hive tables should be handled differently when using the Kyuubi Spark SQL engine.
As a comparison, the same setup works correctly with Iceberg 1.7.2 when both `iceberg-spark-runtime-3.5_2.12` and `iceberg-hive-runtime` 1.7.2 are present on the Spark classpath. No other components were changed.

We would prefer not to add the old `iceberg-hive-runtime` to the new Spark classpath if there is a supported metadata migration path or another recommended way to handle tables created with older Iceberg versions.

Thanks in advance.

### Affects Version(s)

1.11.1

### Kyuubi Server Log Output

```logtalk

```

### Kyuubi Engine Log Output

```logtalk
26/08/28 11:13:08 ERROR log: error in initSerDe: java.lang.ClassNotFoundException Class org.apache.iceberg.mr.hive.HiveIcebergSerDe not found
java.lang.ClassNotFoundException: Class org.apache.iceberg.mr.hive.HiveIcebergSerDe not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2592)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:447)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:440)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:281)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:263)
at org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:641)
at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:624)
at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree2$1(HiveClientImpl.scala:457)
at org.apache.spark.sql.hive.client.HiveClientImpl.org$apache$spark$sql$hive$client$HiveClientImpl$$convertHiveTableToCatalogTable(HiveClientImpl.scala:456)
at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getTablesByName$2(HiveClientImpl.scala:436)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getTablesByName$1(HiveClientImpl.scala:436)
at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:304)
at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:235)
at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:234)
at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:284)
at org.apache.spark.sql.hive.client.HiveClientImpl.getTablesByName(HiveClientImpl.scala:436)
at org.apache.spark.sql.hive.HiveExternalCatalog.getRawTablesByNames(HiveExternalCatalog.scala:124)
at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$getTablesByName$1(HiveExternalCatalog.scala:750)
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:99)
at org.apache.spark.sql.hive.HiveExternalCatalog.getTablesByName(HiveExternalCatalog.scala:750)
at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.getTablesByName(ExternalCatalogWithListener.scala:142)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getTablesByName(SessionCatalog.scala:571)
at org.apache.kyuubi.engine.spark.util.SparkCatalogUtils$.$anonfun$getCatalogTablesOrViews$2(SparkCatalogUtils.scala:192)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
at org.apache.kyuubi.engine.spark.util.SparkCatalogUtils$.getCatalogTablesOrViews(SparkCatalogUtils.scala:174)
at org.apache.kyuubi.engine.spark.operation.GetTables.runInternal(GetTables.scala:81)
at org.apache.kyuubi.operation.AbstractOperation.run(AbstractOperation.scala:197)
at org.apache.kyuubi.session.AbstractSession.runOperation(AbstractSession.scala:114)
at org.apache.kyuubi.engine.spark.session.SparkSessionImpl.runOperation(SparkSessionImpl.scala:112)
at org.apache.kyuubi.session.AbstractSession.getTables(AbstractSession.scala:175)
at org.apache.kyuubi.service.AbstractBackendService.getTables(AbstractBackendService.scala:94)
at org.apache.kyuubi.service.TFrontendService.GetTables(TFrontendService.scala:329)
at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Processor$GetTables.getResult(TCLIService.java:1770)
at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Processor$GetTables.getResult(TCLIService.java:1750)
at org.apache.kyuubi.shaded.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.kyuubi.shaded.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:35)
at org.apache.kyuubi.shaded.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

26/08/28 11:13:08 ERROR GetTables: Error operating GetTables: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.iceberg.mr.hive.HiveIcebergStorageHandler
at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:297)
at org.apache.spark.sql.hive.client.HiveClientImpl.org$apache$spark$sql$hive$client$HiveClientImpl$$convertHiveTableToCatalogTable(HiveClientImpl.scala:491)
at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getTablesByName$2(HiveClientImpl.scala:436)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getTablesByName$1(HiveClientImpl.scala:436)
at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:304)
at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:235)
at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:234)
at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:284)
at org.apache.spark.sql.hive.client.HiveClientImpl.getTablesByName(HiveClientImpl.scala:436)
at org.apache.spark.sql.hive.HiveExternalCatalog.getRawTablesByNames(HiveExternalCatalog.scala:124)
at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$getTablesByName$1(HiveExternalCatalog.scala:750)
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:99)
at org.apache.spark.sql.hive.HiveExternalCatalog.getTablesByName(HiveExternalCatalog.scala:750)
at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.getTablesByName(ExternalCatalogWithListener.scala:142)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getTablesByName(SessionCatalog.scala:571)
at org.apache.kyuubi.engine.spark.util.SparkCatalogUtils$.$anonfun$getCatalogTablesOrViews$2(SparkCatalogUtils.scala:192)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
at org.apache.kyuubi.engine.spark.util.SparkCatalogUtils$.getCatalogTablesOrViews(SparkCatalogUtils.scala:174)
at org.apache.kyuubi.engine.spark.operation.GetTables.runInternal(GetTables.scala:81)
at org.apache.kyuubi.operation.AbstractOperation.run(AbstractOperation.scala:197)
at org.apache.kyuubi.session.AbstractSession.runOperation(AbstractSession.scala:114)
at org.apache.kyuubi.engine.spark.session.SparkSessionImpl.runOperation(SparkSessionImpl.scala:112)
at org.apache.kyuubi.session.AbstractSession.getTables(AbstractSession.scala:175)
at org.apache.kyuubi.service.AbstractBackendService.getTables(AbstractBackendService.scala:94)
at org.apache.kyuubi.service.TFrontendService.GetTables(TFrontendService.scala:329)
at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Processor$GetTables.getResult(TCLIService.java:1770)
at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Processor$GetTables.getResult(TCLIService.java:1750)
at org.apache.kyuubi.shaded.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.kyuubi.shaded.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:35)
at org.apache.kyuubi.shaded.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.iceberg.mr.hive.HiveIcebergStorageHandler
at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:310)
at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:292)
... 49 more
Caused by: java.lang.ClassNotFoundException: org.apache.iceberg.mr.hive.HiveIcebergStorageHandler
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:305)
... 50 more
```

### Kyuubi Server Configurations

```yaml

```

### Kyuubi Engine Configurations

```yaml

```

### Additional context

_No response_

### Are you willing to submit PR?

- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [x] No. I cannot submit a PR at this time.

Start with SparkCatalogUtils.scala around getCatalogTablesOrViews and GetTables.scala around runInternal, then trace the reported HiveClientImpl conversion path in the stack trace. Reproduce GetTables or CREATE TABLE with a legacy Iceberg table and compare behavior with and without iceberg-hive-runtime. Done means the supported handling or migration path is established and the failing catalog operation no longer breaks, with coverage or documentation as appropriate.

Written by the indexing model from the issue text.