apache / apache/arrow

[R] write_parquet() a large data.table with index will make the parquet file unreadable

Open
#40,050 6 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: R Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Saving a data.frame with a big attribute (like an index commonly used in the `data.table` package) will make the parquet file unreadable and produce this error :

```
Error in `open_dataset()`:
! IOError: Error creating dataset. Could not read schema from 'path/example.parquet'.
Is this a 'parquet' file?:
Could not open Parquet input source 'path/example.parquet':
Couldn't deserialize thrift: TProtocolException: Exceeded size limit
```

Bug as understood by the [stackoverflow issue](https://stackoverflow.com/questions/77982801/error-package-arrowr-read-parquet-open-dataset-couldnt-deserialize-thrift-t) : The normal efficiency of binary-data-storage in parquet files is not afforded to R attributes, so a big attribute (like `data.table` indexes) would break the format.

**This bug has important reliability implications for the parquet format**

### Repex :
```r
library(arrow)
library(data.table)
# Seed
set.seed(1L)
# Big enough data.table
dt = data.table(x = sample(1e5L, 1e7L, TRUE), y = runif(100L))
# Save in parquet format
write_parquet(dt, "example_ok.parquet")
# Readable
dt_ok <- open_dataset("example_ok.parquet")
# Simple filter
dt[x == 989L]
# Save in parquet format
write_parquet(dt, "example_error.parquet")
# Error
dt_error <- open_dataset("example_error.parquet")
```

### Component(s)

Parquet, R

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.