[BUG] read_avro method fails to handle avro.schema.MapSchema
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
`read_avro` method fails to read avro file with columns that has Map data type.
- When `type` of a column is an array, it does not work. Example: schema ( [sample.avsc.txt](https://github.com/rapidsai/cudf/files/5279666/sample.avsc.txt) ), corresponding avro file ( [peters.avro.txt](https://github.com/rapidsai/cudf/files/5279668/peters.avro.txt) )
```
{
"name": "feature22183",
"type":
{
"type": "map",
"values": "double"
}
},
```
- The following works though:
```
{
"name": "feature9996",
"type": "boolean"
},
```
Example schema: [samplew.avsc.txt](https://github.com/rapidsai/cudf/files/5279435/samplew.avsc.txt); Corresponding Avro file: [petersw.avro.txt](https://github.com/rapidsai/cudf/files/5279436/petersw.avro.txt)
**Steps/Code to reproduce bug**
```
import cudf
a = cudf.io.read_avro('peters.avro')
```
results in:
```
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
in
1 import cudf
----> 2 a = cudf.io.read_avro('peters.avro')
/opt/conda/envs/rapids/lib/python3.7/site-packages/cudf/io/avro.py in read_avro(filepath_or_buffer, engine, columns, skip_rows, num_rows, **kwargs)
26 return DataFrame._from_table(
27 libcudf.avro.read_avro(
---> 28 filepath_or_buffer, columns, skip_rows, num_rows
29 )
30 )
cudf/_lib/avro.pyx in cudf._lib.avro.read_avro()
cudf/_lib/avro.pyx in cudf._lib.avro.read_avro()
RuntimeError: cuDF failure at: /opt/conda/envs/rapids/conda-bld/libcudf_1598487768118/work/cpp/src/io/avro/reader_impl.cu:80: Cannot parse metadata
```
**Expected behavior**
`read_avro` should be able to read the avro file. You can use the following code to validate the avro file:
```
from avro.datafile import DataFileReader, DataFileWriter
from avro.io import DatumReader, DatumWriter
reader = DataFileReader(open("peters.avro", "rb"), DatumReader())
for user in reader:
print(user)
reader.close()
```
`pip install avro-python3` may be needed to execute the above piece of code.
output:
```
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature22183': {'key1': 2.5, 'key2': 2.5, 'key3': 2.5, 'key4': 2.5, 'key5': 2.5, 'key6': 2.5}, 'timestamp': 2}
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature22183': {'key1': 2.5, 'key2': 2.5, 'key3': 2.5, 'key4': 2.5, 'key5': 2.5, 'key6': 2.5}, 'timestamp': 2}
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature22183': {'key1': 2.5, 'key2': 2.5, 'key3': 2.5, 'key4': 2.5, 'key5': 2.5, 'key6': 2.5}, 'timestamp': 2}
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature22183': {'key1': 2.5, 'key2': 2.5, 'key3': 2.5, 'key4': 2.5, 'key5': 2.5, 'key6': 2.5}, 'timestamp': 2}
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature22183': {'key1': 2.5, 'key2': 2.5, 'key3': 2.5, 'key4': 2.5, 'key5': 2.5, 'key6': 2.5}, 'timestamp': 2}
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature22183': {'key1': 2.5, 'key2': 2.5, 'key3': 2.5, 'key4': 2.5, 'key5': 2.5, 'key6': 2.5}, 'timestamp': 2}
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature22183': {'key1': 2.5, 'key2': 2.5, 'key3': 2.5, 'key4': 2.5, 'key5': 2.5, 'key6': 2.5}, 'timestamp': 2}
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature22183': {'key1': 2.5, 'key2': 2.5, 'key3': 2.5, 'key4': 2.5, 'key5': 2.5, 'key6': 2.5}, 'timestamp': 2}
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature22183': {'key1': 2.5, 'key2': 2.5, 'key3': 2.5, 'key4': 2.5, 'key5': 2.5, 'key6': 2.5}, 'timestamp': 2}
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature22183': {'key1': 2.5, 'key2': 2.5, 'key3': 2.5, 'key4': 2.5, 'key5': 2.5, 'key6': 2.5}, 'timestamp': 2}
```
**Environment overview (please complete the following information)**
- Environment location: Docker
- Method of cuDF install: Docker
- If method of install is [Docker], provide `docker pull` & `docker run` commands used
```
docker pull nvcr.io/nvidia/rapidsai/rapidsai:0.15-cuda11.0-base-ubuntu18.04
docker run --gpus all --rm -it -p 7777:8888 -p 7676:8787 -p 7675:8786 nvcr.io/nvidia/rapidsai/rapidsai:0.15-cuda11.0-base-ubuntu18.04
```
**Environment details**
Please run and paste the output of the `cudf/print_env.sh` script here, to gather any other relevant environment details
[env.log](https://github.com/rapidsai/cudf/files/5279465/env.log)
**Additional context**
N/A
Contributor guide
Assessment
This issue has not been assessed yet.