dask / dask/dask

Unable to load ORC table using `read_orc`

Open
#7,850 6 comments 0 reactions 0 assignees View on GitHub
dataframe io needs attention
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

**What happened**:

I am trying to use `read_orc` to read an ORC table located in a Kerberised HDFS. I can use `read_parquet` but `read_orc` fails with the following attribute error:

```py
/projects/gds/chavesrl/condapv/envs/N2D/lib/python3.7/site-packages/dask/dataframe/io/orc.py in read_orc(path, columns, storage_options)
74 raise ValueError("Incompatible schemas while parsing ORC files")
75 nstripes_per_file.append(o.nstripes)
---> 76 schema = _get_pyarrow_dtypes(schema, categories=None)
77 if columns is not None:
78 ex = set(columns) - set(schema)

/projects/gds/chavesrl/condapv/envs/N2D/lib/python3.7/site-packages/dask/dataframe/io/utils.py in _get_pyarrow_dtypes(schema, categories)
9
10 # Check for pandas metadata
---> 11 has_pandas_metadata = schema.metadata is not None and b"pandas" in schema.metadata
12 if has_pandas_metadata:
13 pandas_metadata = json.loads(schema.metadata[b"pandas"].decode("utf8"))

AttributeError: 'NoneType' object has no attribute 'metadata'
```

after running the following code:
```py
from dask.dataframe import read_orc
read_orc(info['location']+'/*',
storage_options = {
'user':'{username}',
'kerb_ticket' : '/tmp/krb5cc_132855'
})
```
where `info['location']` is of the likes of: `hdfs://cluster/db/schema/table_name`

**What you expected to happen**:

To have the table be loaded as a dask dataframe in the same way parquet tables are read succesfully

**Anything else we need to know?**:

**Environment**:

- Dask version: '2021.04.0'
- Python version: 3.7.10
- Operating System: Linux RedHat
- Install method (conda, pip, source): conda

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.