[BUG]hudi cli command with Wrong FS error
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
hudi cli command with Wrong FS
**To Reproduce**
Steps to reproduce the behavior:
1. compaction validate --instant 20231023061611278
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :0.14
* Spark version :3.1.2
* Hive version :3.1.3
* Hadoop version :3.2.2
* Storage (HDFS/S3/GCS..) :s3
* Running on Docker? (yes/no) :
no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```
INFO [Thread-11] - 23/10/23 08:08:12 INFO ShutdownHookManager: Deleting directory /tmp/spark-0ef5650a-608b-4ebf-896b-28338acbd031
Wrong FS: s3:/tmp/147797f8-317c-496f-8987-0a68a8593c5a.ser, expected: s3://big-data-us
Details of the error have been omitted. You can use the stacktrace command to print the full stacktrace.
hudi:dwd_device_dealer_relation_rt->
hudi:dwd_device_dealer_relation_rt->stacktrace
java.lang.IllegalArgumentException: Wrong FS: s3:/tmp/147797f8-317c-496f-8987-0a68a8593c5a.ser, expected: s3://tuya-big-data-us
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:737)
at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:597)
at com.amazon.ws.emr.hadoop.fs.staging.DefaultStagingMechanism.isStagingDirectoryPath(DefaultStagingMechanism.java:49)
at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.getFileStatus(S3NativeFileSystem.java:493)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1690)
at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.exists(EmrFileSystem.java:436)
at org.apache.hudi.common.fs.HoodieWrapperFileSystem.exists(HoodieWrapperFileSystem.java:566)
at org.apache.hudi.cli.commands.CompactionCommand.validateCompaction(CompactionCommand.java:508)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.shell.command.invocation.InvocableShellMethod.doInvoke(InvocableShellMethod.java:306)
at org.springframework.shell.command.invocation.InvocableShellMethod.invoke(InvocableShellMethod.java:232)
at org.springframework.shell.command.CommandExecution$DefaultCommandExecution.evaluate(CommandExecution.java:158)
at org.springframework.shell.Shell.evaluate(Shell.java:208)
at org.springframework.shell.Shell.run(Shell.java:140)
at org.springframework.shell.jline.InteractiveShellRunner.run(InteractiveShellRunner.java:73)
at org.springframework.shell.DefaultShellApplicationRunner.run(DefaultShellApplicationRunner.java:65)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:762)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:752)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295)
at org.apache.hudi.cli.Main.main(Main.java:34).```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in org.apache.hudi.cli.commands.CompactionCommand at validateCompaction, identified at line 508 in the stack trace, and reproduce `compaction validate --instant 20231023061611278` in the listed Hudi, Spark, Hive, Hadoop, and S3 environment. Inspect the full stack trace and the Wrong FS path handling; done means the command no longer fails with this error for the reported environment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, spark
- Domain
- cli, cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100