Support reading HDFS files with Kerberos authentication
- Dominant language
- C++
- Stars
- 3.6k
- Forks
- 468
- Avg merge
- 29m
- Merged PRs (30d)
- 1
Description
Currently, GraphScope supports reading graph data from local files, object storage, and non-secure HDFS. However, many production Hadoop clusters require Kerberos authentication for security compliance.
When attempting to load graph data from a Kerberized HDFS (e.g., `hdfs://namenode:8020/path/to/data`), there is no built-in mechanism to provide Kerberos credentials (keytab, principal, krb5.conf), making it impossible to access such data sources directly.
I would like GraphScope to support reading graph data from HDFS clusters with Kerberos authentication enabled. The expected solution could include:
1. **Configuration parameters** in `graphscope.session()` to specify Kerberos settings, e.g.:
```python
sess = graphscope.session(
hdfs_kerberos_enabled=True,
hdfs_kerberos_principal="user/host@REALM",
hdfs_kerberos_keytab="/path/to/keytab",
hdfs_krb5_conf="/etc/krb5.conf" # optional
)
2. Automatic authentication before accessing HDFS files, ensuring all Engine pods can authenticate to the NameNode and DataNodes.
3. Support for both HDFS input (loading graph data) and HDFS output (storing results).
Contributor guide
Research direction
Start by tracing how graphscope.session() configures existing non-secure HDFS input and output, then inspect how Engine pods receive storage settings. Define how keytab, principal, and optional krb5.conf are supplied and authenticated across all pods; done means Kerberized HDFS reads and writes work alongside existing local, object-storage, and non-secure HDFS access.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop
- Domain
- data-engineering, distributed-systems, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100