NVIDIA / NVIDIA/cudf

[BUG] read_avro method incorrectly reads avro.schema.ArraySchema

Open
#6,322 2 comments 0 reactions 0 assignees View on GitHub
bug cuIO libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**

`read_avro` method incorrectly reads avro file with columns that have array data type.

- When `type` of a column is an array, it does not read correctly. Example: schema ( [sample.avsc.txt](https://github.com/rapidsai/cudf/files/5279551/sample.avsc.txt) ), corresponding avro file (
[peters.avro.txt](https://github.com/rapidsai/cudf/files/5279548/peters.avro.txt) )
```
{
"name": "feature81078",
"type":
{
"type": "array",
"items": "string"
}
},
```
- The following works though:
```
{
"name": "feature9996",
"type": "boolean"
},
```
Example schema: [samplew.avsc.txt](https://github.com/rapidsai/cudf/files/5279435/samplew.avsc.txt); Corresponding Avro file: [petersw.avro.txt](https://github.com/rapidsai/cudf/files/5279436/petersw.avro.txt)

**Steps/Code to reproduce bug**

```
import cudf
a = cudf.io.read_avro('peters.avro')
```

results in:
```
username tweet feature81079 timestamp
0 Sample Sample True 0
1 False 0
2 False 0
3 False 0
4 False 0
5 False 0
6 False 0
7 False 0
8 False 0
9 False 0
```

**Expected behavior**

`read_avro` should be able to read the avro file. You can use the following code to validate the avro file:

```
from avro.datafile import DataFileReader, DataFileWriter
from avro.io import DatumReader, DatumWriter

reader = DataFileReader(open("peters.avro", "rb"), DatumReader())
for user in reader:
print(user)
reader.close()
```

`pip install avro-python3` may be needed to execute the above piece of code.

output:
```
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature81078': ['Sample', 'Sample', 'Sample', 'Sample', 'Sample'], 'timestamp': 2}
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature81078': ['Sample', 'Sample', 'Sample', 'Sample', 'Sample'], 'timestamp': 2}
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature81078': ['Sample', 'Sample', 'Sample', 'Sample', 'Sample'], 'timestamp': 2}
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature81078': ['Sample', 'Sample', 'Sample', 'Sample', 'Sample'], 'timestamp': 2}
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature81078': ['Sample', 'Sample', 'Sample', 'Sample', 'Sample'], 'timestamp': 2}
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature81078': ['Sample', 'Sample', 'Sample', 'Sample', 'Sample'], 'timestamp': 2}
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature81078': ['Sample', 'Sample', 'Sample', 'Sample', 'Sample'], 'timestamp': 2}
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature81078': ['Sample', 'Sample', 'Sample', 'Sample', 'Sample'], 'timestamp': 2}
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature81078': ['Sample', 'Sample', 'Sample', 'Sample', 'Sample'], 'timestamp': 2}
{'username': 'Sample', 'tweet': 'Sample', 'feature81079': True, 'feature81078': ['Sample', 'Sample', 'Sample', 'Sample', 'Sample'], 'timestamp': 2}
```

**Environment overview (please complete the following information)**
- Environment location: Docker
- Method of cuDF install: Docker
- If method of install is [Docker], provide `docker pull` & `docker run` commands used
```
docker pull nvcr.io/nvidia/rapidsai/rapidsai:0.15-cuda11.0-base-ubuntu18.04
docker run --gpus all --rm -it -p 7777:8888 -p 7676:8787 -p 7675:8786 nvcr.io/nvidia/rapidsai/rapidsai:0.15-cuda11.0-base-ubuntu18.04
```

**Environment details**
Please run and paste the output of the `cudf/print_env.sh` script here, to gather any other relevant environment details

[env.log](https://github.com/rapidsai/cudf/files/5279465/env.log)

**Additional context**
N/A

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.