parquet-cli reports nested columns as null
- Lenguaje dominante
- Java
- Estrellas
- 3.1k
- Forks
- 1.6k
- Merge medio
- 3 d 12 h
- PR fusionados (30 d)
- 33
Descripción
### Describe the bug, including details regarding any error messages, version, and platform.
Using Parquet CLI 1.15.0 via Mac Homebrew, I noticed some surprising behaviour with the `parquet-cli` and nested columns.
`parquet schema catalog.parquet` returns a schema showing the nested types (I've trimmed the field list slightly):
```avro
{
"type" : "record",
"name" : "schema",
"fields" : [ {
"name" : "item_id",
"type" : "string"
}, {
"name" : "title",
"type" : [ "null", "string" ],
"default" : null
}, {
"name" : "language",
"type" : [ "null", "string" ],
"default" : null
}, {
"name" : "subjects",
"type" : [ "null", {
"type" : "array",
"items" : {
"type" : "record",
"name" : "list",
"fields" : [ {
"name" : "element",
"type" : "string"
} ]
}
} ],
"default" : null
}, {
"name" : "authors",
"type" : [ "null", {
"type" : "array",
"items" : {
"type" : "record",
"name" : "list",
"namespace" : "list2",
"fields" : [ {
"name" : "element",
"type" : "string"
} ]
}
} ],
"default" : null
} ]
}
```
`parquet dictionary -c subjects.list.element catalog.parquet` will return the expected values for those fields as well:
```
Row group 0 dictionary for "subjects.list.element":
0: "Bestsellers"
1: "Biography"
2: "Fantasy Fiction"
3: "Music Theory"
4: "Disability"
5: "Family"
6: "Young Adult"
```
However, when using `cat` or `head` to display the file contents those fields are displayed as null:
```
{"bmc_id": "id1", "title": null, "language": "en", "subjects": null, "authors": null}
{"bmc_id": "id2", "title": null, "language": "en", "subjects": null, "authors": null,}
{"bmc_id": "id3", "title": null, "language": "en", "subjects": null, "authors": null}
```
Other tools like PyArrow or Pandas do display those values as arrays. I created this as a bug because it _looks_ like it's working and if those fields are nullable, there's no way to tell whether the null value is correct.
### Component(s)
_No response_
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Línea de trabajo
Reproduzca el problema con Parquet CLI 1.15.0 usando `parquet schema`, `parquet dictionary -c subjects.list.element` y `cat` o `head` sobre `catalog.parquet`. Compare los valores de la columna anidada informados por el comando de diccionario con los registros mostrados e inspeccione la ruta de parquet-cli que gestiona esos comandos. La tarea estará terminada cuando `cat` y `head` muestren arrays anidados en lugar de mostrarlos incorrectamente como null.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- java
- Área
- cli, data
- Tipo de issue
- Error
- Dificultad
- 3/5
- Tiempo estimado
- 1-2 días
- Estado de actividad
- Estancado
- Claridad
- Necesita aclaración
- Aptitud para principiantes
- 35/100