NVIDIA / NVIDIA/cudf

[BUG] read_avro method fails to handle avro.schema.UnionSchema

Open
#6,319 1 comment 0 reactions 0 assignees View on GitHub
bug cuIO libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**

`read_avro` method fails to read avro file with columns that have multiple possible types.

- When `type` of a column is an array, it does not work. Example: schema ( [sample.avsc.txt](https://github.com/rapidsai/cudf/files/5279434/sample.avsc.txt) ), corresponding avro file ( [peters.avro.txt](https://github.com/rapidsai/cudf/files/5279433/peters.avro.txt) )
```
{
"name": "feature9996",
"type": [
"null",
"boolean"
]
},
```
- The following works though:
```
{
"name": "feature9996",
"type": "boolean"
},
```
Example schema: [samplew.avsc.txt](https://github.com/rapidsai/cudf/files/5279435/samplew.avsc.txt); Corresponding Avro file: [petersw.avro.txt](https://github.com/rapidsai/cudf/files/5279436/petersw.avro.txt)

**Steps/Code to reproduce bug**

```
import cudf
a = cudf.io.read_avro('peters.avro')
```

results in:
```
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
in
1 import cudf
----> 2 a = cudf.io.read_avro('peters.avro')

/opt/conda/envs/rapids/lib/python3.7/site-packages/cudf/io/avro.py in read_avro(filepath_or_buffer, engine, columns, skip_rows, num_rows, **kwargs)
26 return DataFrame._from_table(
27 libcudf.avro.read_avro(
---> 28 filepath_or_buffer, columns, skip_rows, num_rows
29 )
30 )

cudf/_lib/avro.pyx in cudf._lib.avro.read_avro()

cudf/_lib/avro.pyx in cudf._lib.avro.read_avro()

RuntimeError: cuDF failure at: /opt/conda/envs/rapids/conda-bld/libcudf_1598487768118/work/cpp/src/io/avro/reader_impl.cu:80: Cannot parse metadata
```

**Expected behavior**

`read_avro` should be able to read the avro file. You can use the following code to validate the avro file:

```
from avro.datafile import DataFileReader, DataFileWriter
from avro.io import DatumReader, DatumWriter

reader = DataFileReader(open("peters.avro", "rb"), DatumReader())
for user in reader:
print(user)
reader.close()
```

`pip install avro-python3` may be needed to execute the above piece of code.

**Environment overview (please complete the following information)**
- Environment location: Docker
- Method of cuDF install: Docker
- If method of install is [Docker], provide `docker pull` & `docker run` commands used
```
docker pull nvcr.io/nvidia/rapidsai/rapidsai:0.15-cuda11.0-base-ubuntu18.04
docker run --gpus all --rm -it -p 7777:8888 -p 7676:8787 -p 7675:8786 nvcr.io/nvidia/rapidsai/rapidsai:0.15-cuda11.0-base-ubuntu18.04
```

**Environment details**
Please run and paste the output of the `cudf/print_env.sh` script here, to gather any other relevant environment details

[env.log](https://github.com/rapidsai/cudf/files/5279465/env.log)

**Additional context**
N/A

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.