blaze / blaze/odo

Support for kerberos HDFS clients

Open
#396 0 comments 0 reactions 0 assignees View on GitHub
HDFS
Dominant language
Python
Stars
1k
Forks
131
PR merge metrics
No merged PRs in 30d

Description

The cluster I have access to uses kerberos authentication exclusively, and so I've needed a kerberos client in order to use odo's HDFS functionality.

The [hdfs](https://pypi.python.org/pypi/hdfs/) package has both kerberos and avro extensions.

Something like

``` python
import hdfs
from hdfs.ext.kerberos import KerberosClient

class _KHDFS(object):
""" Parent class for data on Hadoop File System

Examples
--------

>>> KHDFS(CSV)('/path/to/file.csv', host='54.91.255.255',
... port=14000, user='hdfs') # doctest: +SKIP

Alternatively use resource strings

>>> resource('khdfs://hdfs@54.91.255.255:/path/to/file.csv') # doctest: +SKIP
"""
def __init__(self, *args, **kwargs):
hdfs = kwargs.get('hdfs', None)
host = kwargs.get('host', None)
port = kwargs.get('port', None)
if not hdfs and (host and port):
url = "http://{host}:{port}/".format(host=host,port=port)
root = "/"
hdfs = KerberosClient(url, root=root)

if not hdfs:
raise ValueError("No WebHDFS url found:.\n"
"Must supply keywords\n"
" host=, port=")
self.hdfs = hdfs
kwargs.update(hdfs=hdfs)

self.subtype.__init__(self, *args, **kwargs)
```

I've got a functional version of this in a branch, but I'm doing some hacky things that I wouldn't put in a PR.
1. `KerberosClient`'s API is materially different than the `PyWebHdfsClient` object in odo, so I ended up writing parallel implementations using `KerberosClient` of most of the methods in `odo/backends/hdfs.py`
2. In order to allow the two implementations to coexist, I used a different resource pattern `khdfs://user@ip:port:/path/to/file.csv`. This is not nice at all, and it would be better to be able to configure which implementation is used rather than expect the user to change resource URLs.

I'm wondering if this is valuable to anyone else, and if anyone has recommendations about how to solve these two issues.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.