apache / apache/hudi

[SUPPORT]Unable to connect to an s3 hudi table

Open
#8,144 4 comments 0 reactions 0 assignees View on GitHub
area:aws status:triaged
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I am unable to connect to an s3 hudi table using the hudi client. I keep getting an error:

Loading HoodieTableMetaClient from s3://test-datalake/datasets//test_table
Failed to get instance of org.apache.hadoop.fs.FileSystem
Details of the error have been omitted. You can use the stacktrace command to print the full stacktrace.

I can't see any guidance on how to set up hudi to connect to s3 hudi tables locally and i am completely stuck with this.

I have tried building locally with:

> # Checkout code and build
> git clone https://github.com/apache/hudi.git && cd hudi
> mvn clean package -DskipTests -Dspark3.2 -Dscala-2.12

I then navigate to the hudi client directory and can start up the client and try to connect to my s3 dataset and get this error. I have also tried to build hudi using the docker quickstart guide and exec to the container:

docker exec -it adhoc-2 /bin/bash

navigate to the hudi client folder:

cd /var/hoodie/ws/hudi-cli

I then try and connect there and i still get the same issue.

**Expected behavior**

I expect to connect to the s3 table and view commits so that i can clean/repair failed commits to the hudie table.
**Environment Description**
Mac

* Hudi version :
0.13.0

* Spark version :
3.3.2

* Hive version :

* Hadoop version :
3.3.4
* Storage (HDFS/S3/GCS..) :
s3
* Running on Docker? (yes/no) :
yes and also tried without

**Additional context**
I have set my hadoop files as follows:
core-site.xml file looks like
```


fs.defaultFS
hdfs://localhost:9000

```
hdfs-site.xml file looks like:

```

dfs.replication
3

dfs.namenode.name.dir
/hadoop/namenode/

dfs.datanode.data.dir
/hadoop/namenode

```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the failure in the hudi client using the Docker quickstart and print the full stacktrace as suggested. Inspect the supplied core-site.xml and hdfs-site.xml alongside the Hudi 0.13.0, Spark 3.3.2, and Hadoop 3.3.4 environment; done means connecting to the S3 table and viewing its commits.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, hadoop, java, spark
Domain
cloud, data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.