apache / apache/hudi

Standalone install not working with S3

Open
#15,541 0 comments 0 reactions 0 assignees View on GitHub
component:cli from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Standalone install not working with AWS S3 tables
# Install Java JDK 1.8
# Reference JAVA_HOME
# Install Maven 3.8.6
# Install Spark 3.2.2
{code:java}
wget https://dlcdn.apache.org/spark/spark-3.2.2/spark-3.2.2-bin-hadoop3.2.tgz
{code}
# Reference SPARK_HOME
# Download Hudi Sources 0.12.1
# Build Hudi
# Download Client jars for S3 and reference CLIENT_JAR
/!\ Note that documentation needs to precise where '/lib/spark-3.2.0-bin-hadoop3.2/jars/...' should apply
# Launch Hudi CLI

{code:java}
hudi->connect --path s3://stef-raw-euwest1-435665164035-prt/dms/infoquai/hudi_iq_log_quai_mor/
16388 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient [] - Loading HoodieTableMetaClient from s3://stef-raw-euwest1-435665164035-prt/dms/infoquai/hudi_iq_log_quai_mor/
16961 [main] WARN org.apache.hadoop.util.NativeCodeLoader [] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Failed to get instance of org.apache.hadoop.fs.FileSystem
Details of the error have been omitted. You can use the stacktrace command to print the full stacktrace.
hudi->stacktrace
org.apache.hudi.exception.HoodieIOException: Failed to get instance of org.apache.hadoop.fs.FileSystem
at org.apache.hudi.common.fs.FSUtils.getFs(FSUtils.java:112)
at org.apache.hudi.common.table.HoodieTableMetaClient.getFs(HoodieTableMetaClient.java:295)
at org.apache.hudi.common.table.HoodieTableMetaClient.(HoodieTableMetaClient.java:127)
at org.apache.hudi.common.table.HoodieTableMetaClient.newMetaClient(HoodieTableMetaClient.java:642)
at org.apache.hudi.common.table.HoodieTableMetaClient.access$000(HoodieTableMetaClient.java:80)
at org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:711)
at org.apache.hudi.cli.HoodieCLI.refreshTableMetadata(HoodieCLI.java:89)
at org.apache.hudi.cli.HoodieCLI.connectTo(HoodieCLI.java:95)
at org.apache.hudi.cli.commands.TableCommand.connect(TableCommand.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.shell.command.invocation.InvocableShellMethod.doInvoke(InvocableShellMethod.java:306)
at org.springframework.shell.command.invocation.InvocableShellMethod.invoke(InvocableShellMethod.java:232)
at org.springframework.shell.command.CommandExecution$DefaultCommandExecution.evaluate(CommandExecution.java:158)
at org.springframework.shell.Shell.evaluate(Shell.java:208)
at org.springframework.shell.Shell.run(Shell.java:140)
at org.springframework.shell.jline.InteractiveShellRunner.run(InteractiveShellRunner.java:73)
at org.springframework.shell.DefaultShellApplicationRunner.run(DefaultShellApplicationRunner.java:65)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:762)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:752)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295)
at org.apache.hudi.cli.Main.main(Main.java:34)
Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3225)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3245)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:121)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3296)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3264)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:475)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
at org.apache.hudi.common.fs.FSUtils.getFs(FSUtils.java:110)
{code}

See Hudi CLI launch output [^hudi-cli_trace.txt]

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5156
- Type: Bug
- Attachment(s):
- 03/Nov/22 18:24;benoit.colas;hudi-cli_trace.txt;https://issues.apache.org/jira/secure/attachment/13051761/hudi-cli_trace.txt

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the standalone Hudi CLI setup with the listed Java, Maven, Spark, and client JAR steps, then inspect the FSUtils.java and HoodieCLI stack-trace entry points. Confirm how the S3 filesystem dependency is loaded and clarify the documented Spark JAR path. Done means the CLI can connect to the supplied s3:// path without the UnsupportedFileSystemException and the installation instructions are precise.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, hadoop, java, spark
Domain
cli, cloud, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.