ClassNotFoundException: S3AFileSystem when querying federated Hive tables with s3a:// paths
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 137
Description
## Environment
- Polaris Version: 1.2.0
- Deployment method: Kubernetes (EKS)
- Client: StarRocks 3.5
- Storage: AWS S3
## Setup
Polaris is configured to federate to an existing Hive Metastore containing Iceberg tables with `s3a://` storage paths.
## Problem
When querying tables through the external catalog, metadata operations work (`SHOW DATABASES FROM polaris`), but data queries fail with `ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found`.
## Example Table Definition
The tables in the federated Hive Metastore were created with `s3a://` paths:
```
CREATE TABLE `map_attribute` (
`sc_attribute_id` bigint(20) DEFAULT NULL,
`sc_frame` varchar(1048576) DEFAULT NULL
)
PROPERTIES ("owner" = "***", "write.spark.accept-any-schema" = "true", "location" = "s3a://cake-cupcake/warehouse/blah.db/map_attribute");
```
## Stack Trace
```
2025-11-04 23:47:49,994 INFO [org.apa.pol.ser.exc.IcebergExceptionMapper] [67fe3bec-de05-486b-a6c3-9b39d6ef2c16_0000000000000000149,POLARIS] [,,,] (executor-thread-63) Handling runtimeException java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
2025-11-04 23:47:49,995 ERROR [org.apa.pol.ser.exc.IcebergExceptionMapper] [67fe3bec-de05-486b-a6c3-9b39d6ef2c16_0000000000000000149,POLARIS] [,,,] (executor-thread-63) Unhandled exception returning INTERNAL_SERVER_ERROR: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2737)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3569)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3612)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:172)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3716)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3667)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:557)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:366)
at org.apache.iceberg.hadoop.Util.getFs(Util.java:55)
at org.apache.iceberg.hadoop.HadoopInputFile.fromLocation(HadoopInputFile.java:56)
at org.apache.iceberg.hadoop.HadoopFileIO.newInputFile(HadoopFileIO.java:87)
at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:294)
at org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$0(BaseMetastoreTableOperations.java:180)
at org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$1(BaseMetastoreTableOperations.java:199)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:199)
at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:176)
at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:171)
at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:131)
at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:88)
at org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:71)
at org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:49)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogHandler.loadTable(IcebergCatalogHandler.java:767)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.lambda$loadTable$8(IcebergCatalogAdapter.java:434)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.withCatalog(IcebergCatalogAdapter.java:201)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.loadTable(IcebergCatalogAdapter.java:429)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_Subclass.loadTable$$superforward(Unknown Source)
at org.apache.polaris.service.catalog.iceberg.IcebergRestCatalogEventServiceDelegator_Gj_WCptqTcdHu-fbZfgVkAwPXCI_Delegate_Subclass.loadTable(Unknown Source)
at org.apache.polaris.service.catalog.iceberg.IcebergRestCatalogEventServiceDelegator.loadTable(IcebergRestCatalogEventServiceDelegator.java:268)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_Subclass.loadTable(Unknown Source)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_ClientProxy.loadTable(Unknown Source)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi.loadTable(IcebergRestCatalogApi.java:565)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.loadTable$$superforward(Unknown Source)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass$$function$$13.apply(Unknown Source)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$8(FaultToleranceInterceptor.java:364)
at io.smallrye.faulttolerance.core.Future.from(Future.java:85)
at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$9(FaultToleranceInterceptor.java:364)
at io.smallrye.faulttolerance.core.FaultToleranceContext.call(FaultToleranceContext.java:20)
at io.smallrye.faulttolerance.core.Invocation.apply(Invocation.java:29)
at io.smallrye.faulttolerance.core.metrics.MetricsCollector.apply(MetricsCollector.java:98)
at io.smallrye.faulttolerance.FaultToleranceInterceptor.syncFlow(FaultToleranceInterceptor.java:367)
at io.smallrye.faulttolerance.FaultToleranceInterceptor.intercept(FaultToleranceInterceptor.java:205)
at io.smallrye.faulttolerance.FaultToleranceInterceptor_Bean.intercept(Unknown Source)
at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor.timedMethod(MicrometerTimedInterceptor.java:79)
at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor_Bean.intercept(Unknown Source)
at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
at io.quarkus.security.runtime.interceptor.SecurityHandler.handle(SecurityHandler.java:27)
at io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor.intercept(RolesAllowedInterceptor.java:29)
at io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor_Bean.intercept(Unknown Source)
at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
at io.quarkus.resteasy.reactive.server.runtime.StandardSecurityCheckInterceptor.intercept(StandardSecurityCheckInterceptor.java:44)
at io.quarkus.resteasy.reactive.server.runtime.StandardSecurityCheckInterceptor_RolesAllowedInterceptor_Bean.intercept(Unknown Source)
at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.loadTable(Unknown Source)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi$quarkusrestinvoker$loadTable_56b49d5a1874f2749e5229d18bb00aeb8fe7fdc8.invoke(Unknown Source)
at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:183)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:645)
at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2641)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2735)
... 80 more
```
## Question
Is this expected behavior? Should S3-related dependencies be bundled with Polaris for federated catalog support, or is there additional configuration needed to enable S3A filesystem support?
### To Reproduce
1. Create external catalog in StarRocks that federates to Hive:
```
CREATE EXTERNAL CATALOG `polaris`
PROPERTIES ("iceberg.catalog.scope" = "PRINCIPAL_ROLE:ALL",
"iceberg.catalog.credential" = "****",
"iceberg.catalog.uri" = "https://****/api/catalog",
"iceberg.catalog.warehouse" = "****",
"iceberg.catalog.security" = "oauth2",
"type" = "iceberg",
"iceberg.catalog.type" = "rest"
);
```
2. Verify that metadata queries work: `SHOW DATABASES FROM polaris;`
3. Attempt to query a table: `SELECT * FROM polaris.database.table;`
Contributor guide
Research direction
Start with the IcebergCatalogHandler.loadTable path named in the stack trace and inspect the runtime dependencies and configuration used for Hadoop S3A access. Reproduce the federated Hive query with the supplied StarRocks steps, then determine whether the missing S3A class requires bundled dependencies or additional configuration. Done means the query works or the expected setup limitation is clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100