Some exceptions with solutions when I using odo
- Dominant language
- Python
- Stars
- 1k
- Forks
- 131
- PR merge metrics
- No merged PRs in 30d
Description
I found `odo` this week, and it awesome.
Below is some exceptions when I using types of backends(`odo version: 0.5.1, Hadoop: 2.7.3`), I try to modify the source local, finally it works, below is some notes:
**Q1**. when using `ssh`/`hdfs`: IOError: [Errno 2] No such file or directory: 'hdfs://xxx.xxx.xxx.xxx:/tmp/x.csv'
recommend install:
```
pip install paramiko
pip install pywebhdfs
```
**Q2**. after Q1 `hdfs` : IOError: [Errno 2] No such file or directory: 'hdfs://xxx.xxx.xxx.xxx:/tmp/x.csv'
modify `odo/backends/hdfs.py +31`
```
with ignoring(ImportError):
from pywebhdfs.webhdfs import PyWebHdfsClient
from pywebhdfs.errors import FileNotFound
```
to:
```
from pywebhdfs.webhdfs import PyWebHdfsClient
from pywebhdfs.errors import FileNotFound
```
**Q3**. `S3`: HostRequiredError: BotoClientError: When using SigV4, you must specify a 'host' parameter.
referer: http://docs.aws.amazon.com/zh_cn/general/latest/gr/rande.html#s3_region
modify: `odo/backends/aws.py +51`, we using `us-west-1`, add host in line 51:
```
return boto.connect_s3(aws_access_key_id, aws_secret_access_key,
host='s3-us-west-1.amazonaws.com',
anon=anon)
```
> maybe add `region` or `host` parameters when call `resource`.
**Q4**: `HDFS` backend: KeyError: 'name' --> name is first column name in csv
because hdfs pass `dshape` param when call `source.hdfs.read_file`, so, modify `odo/backends/hdfs.py +434`, remove `dshape` from `kwargs` when call `source.hdfs.read_file`.
more information:
```
# uri_without_host = 'http://{host}:50070/webhdfs/v1/tmp/y.csv?op=OPEN&user=hdfs&host=127.0.0.1&port=50070&dshape=var * {name: string, age: int64}&user.name=hdfs'
uri = uri_without_host.format(host=host) # uri_without_host include dshape which has `{}`, so it will raise exception using format
```
**Q5**: `odo` not support collections of files in `S3`/`HDFS`.
It clear in the documents http://odo.pydata.org/en/latest/uri.html , but other tools(such as spark or hive) always create lots of csv file in hdfs or s3 path, so we need it.
----
I did not modify as a patch, because I am not sure whether it is correct. Please give me some suggestions if you met the same questions.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.