alibaba / alibaba/GraphScope

Support reading HDFS files with Kerberos authentication

Open
#4,654 2 comments 0 reactions 0 assignees View on GitHub
stale
Dominant language
C++
Stars
3.6k
Forks
468
Avg merge
29m
Merged PRs (30d)
1

Description

Currently, GraphScope supports reading graph data from local files, object storage, and non-secure HDFS. However, many production Hadoop clusters require Kerberos authentication for security compliance.

When attempting to load graph data from a Kerberized HDFS (e.g., `hdfs://namenode:8020/path/to/data`), there is no built-in mechanism to provide Kerberos credentials (keytab, principal, krb5.conf), making it impossible to access such data sources directly.

I would like GraphScope to support reading graph data from HDFS clusters with Kerberos authentication enabled. The expected solution could include:

1. **Configuration parameters** in `graphscope.session()` to specify Kerberos settings, e.g.:
```python
sess = graphscope.session(
hdfs_kerberos_enabled=True,
hdfs_kerberos_principal="user/host@REALM",
hdfs_kerberos_keytab="/path/to/keytab",
hdfs_krb5_conf="/etc/krb5.conf" # optional
)

2. Automatic authentication before accessing HDFS files, ensuring all Engine pods can authenticate to the NameNode and DataNodes.

3. Support for both HDFS input (loading graph data) and HDFS output (storing results).

Contributor guide

Open the contributing guide

Research direction

Start by tracing how graphscope.session() configures existing non-secure HDFS input and output, then inspect how Engine pods receive storage settings. Define how keytab, principal, and optional krb5.conf are supplied and authenticated across all pods; done means Kerberized HDFS reads and writes work alongside existing local, object-storage, and non-secure HDFS access.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop
Domain
data-engineering, distributed-systems, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.