apache / apache/gravitino-playground

Error when run gravitino-fileset-example.ipynb

Open
#76 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
83
Forks
55
Avg merge
8h 39m
Merged PRs (30d)
2

Description

Start the playground docker images, and then open Jupyter, run the gravitino-fileset-example.ipynb.

In the second step, I got this error:

```
from hdfs import InsecureClient

# Create a HDFS connector client
hdfs_client = InsecureClient('http://hive:50070', user='root')

# List HDFS file and directories
print(hdfs_client.list('/user/datastrato'))

HdfsError Traceback (most recent call last)
Cell In[2], line 7
4 hdfs_client = InsecureClient('http://hive:50070/', user='root')
6 # List HDFS file and directories
----> 7 print(hdfs_client.list('/user/datastrato'))
9 # hdfs_client.delete("/user/datastrato")

File /opt/conda/lib/python3.11/site-packages/hdfs/client.py:1118, in Client.list(self, hdfs_path, status)
1116 _logger.info('Listing %r.', hdfs_path)
1117 hdfs_path = self.resolve(hdfs_path)
-> 1118 statuses = self._list_status(hdfs_path).json()['FileStatuses']['FileStatus']
1119 if len(statuses) == 1 and (
1120 not statuses[0]['pathSuffix'] or self.status(hdfs_path)['type'] == 'FILE'
1121 # HttpFS behaves incorrectly here, we sometimes need an extra call to
1122 # make sure we always identify if we are dealing with a file.
1123 ):
1124 raise HdfsError('%r is not a directory.', hdfs_path)

File /opt/conda/lib/python3.11/site-packages/hdfs/client.py:118, in _Request.to_method..api_handler(client, hdfs_path, data, strict, **params)
116 if err.exception not in ('RetriableException', 'StandbyException'):
117 if strict:
--> 118 raise err
119 return res
121 attempted_hosts.add(host)

HdfsError: File /user/datastrato does not exist.
```

Then I look into the HDFS, there is no "/user/datastrato" folder; Under "/user" there is only "/user/hive"

`root@3c98b46a73b9:/# hadoop fs -ls /user
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
24/09/19 06:50:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x - root hdfs 0 2024-09-19 06:32 /user/hive
`

Contributor guide

No contributing guide indexed for this repository

Research direction

Open gravitino-fileset-example.ipynb and run the playground Docker images and notebook through the second step. Inspect how the HDFS client uses /user/datastrato and compare it with the available /user/hive path; done means the example can list the intended HDFS location without the reported HdfsError.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, jupyter-notebook, python
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.