apache / apache/arrow

[C++][Parquet] Add support for writing DictionaryArrays with nested value types to Parquet

Open
#39,432 2 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Parquet Status: stale-warning Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Writing DictionaryArrays where the value type is nested is not currently implemented, see:

https://github.com/apache/arrow/blob/75c6b642b5ff1ed171bc1d1a758a70098539c48e/cpp/src/parquet/arrow/path_internal.cc#L747-L749

The following code exercises the above code path, where a DictionaryArray is created over a ListArray:

```python
import pyarrow as pa
import pyarrow.parquet as pq

arr = pa.DictionaryArray.from_arrays(
...: pa.array([0, 1, 2]),
...: pa.array([["a"], ["b"], ["c"]])
...: )
tbl = pa.Table.from_arrays([arr], ["x"])
pq.write_table(tbl, "nested.pq")
# => ArrowNotImplementedError: Writing DictionaryArray with nested dictionary type not yet supported
```

I didn't see an existing issue for this but please comment below if one does.

### Component(s)

C++, Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.