Standalone install not working with S3
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Standalone install not working with AWS S3 tables
# Install Java JDK 1.8
# Reference JAVA_HOME
# Install Maven 3.8.6
# Install Spark 3.2.2
{code:java}
wget https://dlcdn.apache.org/spark/spark-3.2.2/spark-3.2.2-bin-hadoop3.2.tgz
{code}
# Reference SPARK_HOME
# Download Hudi Sources 0.12.1
# Build Hudi
# Download Client jars for S3 and reference CLIENT_JAR
/!\ Note that documentation needs to precise where '/lib/spark-3.2.0-bin-hadoop3.2/jars/...' should apply
# Launch Hudi CLI
{code:java}
hudi->connect --path s3://stef-raw-euwest1-435665164035-prt/dms/infoquai/hudi_iq_log_quai_mor/
16388 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient [] - Loading HoodieTableMetaClient from s3://stef-raw-euwest1-435665164035-prt/dms/infoquai/hudi_iq_log_quai_mor/
16961 [main] WARN org.apache.hadoop.util.NativeCodeLoader [] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Failed to get instance of org.apache.hadoop.fs.FileSystem
Details of the error have been omitted. You can use the stacktrace command to print the full stacktrace.
hudi->stacktrace
org.apache.hudi.exception.HoodieIOException: Failed to get instance of org.apache.hadoop.fs.FileSystem
at org.apache.hudi.common.fs.FSUtils.getFs(FSUtils.java:112)
at org.apache.hudi.common.table.HoodieTableMetaClient.getFs(HoodieTableMetaClient.java:295)
at org.apache.hudi.common.table.HoodieTableMetaClient.(HoodieTableMetaClient.java:127)
at org.apache.hudi.common.table.HoodieTableMetaClient.newMetaClient(HoodieTableMetaClient.java:642)
at org.apache.hudi.common.table.HoodieTableMetaClient.access$000(HoodieTableMetaClient.java:80)
at org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:711)
at org.apache.hudi.cli.HoodieCLI.refreshTableMetadata(HoodieCLI.java:89)
at org.apache.hudi.cli.HoodieCLI.connectTo(HoodieCLI.java:95)
at org.apache.hudi.cli.commands.TableCommand.connect(TableCommand.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.shell.command.invocation.InvocableShellMethod.doInvoke(InvocableShellMethod.java:306)
at org.springframework.shell.command.invocation.InvocableShellMethod.invoke(InvocableShellMethod.java:232)
at org.springframework.shell.command.CommandExecution$DefaultCommandExecution.evaluate(CommandExecution.java:158)
at org.springframework.shell.Shell.evaluate(Shell.java:208)
at org.springframework.shell.Shell.run(Shell.java:140)
at org.springframework.shell.jline.InteractiveShellRunner.run(InteractiveShellRunner.java:73)
at org.springframework.shell.DefaultShellApplicationRunner.run(DefaultShellApplicationRunner.java:65)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:762)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:752)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295)
at org.apache.hudi.cli.Main.main(Main.java:34)
Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3225)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3245)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:121)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3296)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3264)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:475)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
at org.apache.hudi.common.fs.FSUtils.getFs(FSUtils.java:110)
{code}
See Hudi CLI launch output [^hudi-cli_trace.txt]
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5156
- Type: Bug
- Attachment(s):
- 03/Nov/22 18:24;benoit.colas;hudi-cli_trace.txt;https://issues.apache.org/jira/secure/attachment/13051761/hudi-cli_trace.txt
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the standalone Hudi CLI setup with the listed Java, Maven, Spark, and client JAR steps, then inspect the FSUtils.java and HoodieCLI stack-trace entry points. Confirm how the S3 filesystem dependency is loaded and clarify the documented Spark JAR path. Done means the CLI can connect to the supplied s3:// path without the UnsupportedFileSystemException and the installation instructions are precise.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, hadoop, java, spark
- Domain
- cli, cloud, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100