[SUPPORT]Unable to connect to an s3 hudi table
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
I am unable to connect to an s3 hudi table using the hudi client. I keep getting an error:
Loading HoodieTableMetaClient from s3://test-datalake/datasets//test_table
Failed to get instance of org.apache.hadoop.fs.FileSystem
Details of the error have been omitted. You can use the stacktrace command to print the full stacktrace.
I can't see any guidance on how to set up hudi to connect to s3 hudi tables locally and i am completely stuck with this.
I have tried building locally with:
> # Checkout code and build
> git clone https://github.com/apache/hudi.git && cd hudi
> mvn clean package -DskipTests -Dspark3.2 -Dscala-2.12
I then navigate to the hudi client directory and can start up the client and try to connect to my s3 dataset and get this error. I have also tried to build hudi using the docker quickstart guide and exec to the container:
docker exec -it adhoc-2 /bin/bash
navigate to the hudi client folder:
cd /var/hoodie/ws/hudi-cli
I then try and connect there and i still get the same issue.
**Expected behavior**
I expect to connect to the s3 table and view commits so that i can clean/repair failed commits to the hudie table.
**Environment Description**
Mac
* Hudi version :
0.13.0
* Spark version :
3.3.2
* Hive version :
* Hadoop version :
3.3.4
* Storage (HDFS/S3/GCS..) :
s3
* Running on Docker? (yes/no) :
yes and also tried without
**Additional context**
I have set my hadoop files as follows:
core-site.xml file looks like
```
fs.defaultFS
hdfs://localhost:9000
```
hdfs-site.xml file looks like:
```
dfs.replication
3
dfs.namenode.name.dir
/hadoop/namenode/
dfs.datanode.data.dir
/hadoop/namenode
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the failure in the hudi client using the Docker quickstart and print the full stacktrace as suggested. Inspect the supplied core-site.xml and hdfs-site.xml alongside the Hudi 0.13.0, Spark 3.3.2, and Hadoop 3.3.4 environment; done means connecting to the S3 table and viewing its commits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, hadoop, java, spark
- Domain
- cloud, data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100