apache / apache/hudi

[BUG]hudi cli command with Wrong FS error

Open
#9,903 4 comments 0 reactions 0 assignees View on GitHub
component:cli priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

hudi cli command with Wrong FS

**To Reproduce**

Steps to reproduce the behavior:

1. compaction validate --instant 20231023061611278

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version :0.14

* Spark version :3.1.2

* Hive version :3.1.3

* Hadoop version :3.2.2

* Storage (HDFS/S3/GCS..) :s3

* Running on Docker? (yes/no) :
no

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```
INFO [Thread-11] - 23/10/23 08:08:12 INFO ShutdownHookManager: Deleting directory /tmp/spark-0ef5650a-608b-4ebf-896b-28338acbd031
Wrong FS: s3:/tmp/147797f8-317c-496f-8987-0a68a8593c5a.ser, expected: s3://big-data-us
Details of the error have been omitted. You can use the stacktrace command to print the full stacktrace.
hudi:dwd_device_dealer_relation_rt->
hudi:dwd_device_dealer_relation_rt->stacktrace
java.lang.IllegalArgumentException: Wrong FS: s3:/tmp/147797f8-317c-496f-8987-0a68a8593c5a.ser, expected: s3://tuya-big-data-us
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:737)
at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:597)
at com.amazon.ws.emr.hadoop.fs.staging.DefaultStagingMechanism.isStagingDirectoryPath(DefaultStagingMechanism.java:49)
at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.getFileStatus(S3NativeFileSystem.java:493)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1690)
at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.exists(EmrFileSystem.java:436)
at org.apache.hudi.common.fs.HoodieWrapperFileSystem.exists(HoodieWrapperFileSystem.java:566)
at org.apache.hudi.cli.commands.CompactionCommand.validateCompaction(CompactionCommand.java:508)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.shell.command.invocation.InvocableShellMethod.doInvoke(InvocableShellMethod.java:306)
at org.springframework.shell.command.invocation.InvocableShellMethod.invoke(InvocableShellMethod.java:232)
at org.springframework.shell.command.CommandExecution$DefaultCommandExecution.evaluate(CommandExecution.java:158)
at org.springframework.shell.Shell.evaluate(Shell.java:208)
at org.springframework.shell.Shell.run(Shell.java:140)
at org.springframework.shell.jline.InteractiveShellRunner.run(InteractiveShellRunner.java:73)
at org.springframework.shell.DefaultShellApplicationRunner.run(DefaultShellApplicationRunner.java:65)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:762)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:752)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295)
at org.apache.hudi.cli.Main.main(Main.java:34).```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in org.apache.hudi.cli.commands.CompactionCommand at validateCompaction, identified at line 508 in the stack trace, and reproduce `compaction validate --instant 20231023061611278` in the listed Hudi, Spark, Hive, Hadoop, and S3 environment. Inspect the full stack trace and the Wrong FS path handling; done means the command no longer fails with this error for the reported environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, spark
Domain
cli, cloud, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.