apache / apache/arrow

[R] Parquet files of indexed data.table objects error in read_parquet

Open
#43,742 4 comments 1 reaction 0 assignees View on GitHub
Component: Parquet Component: R Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

When a data.table object is indexed (e.g., with `setindex()`), this can cause problems in reading the parquet back in (`Error: IOError: Couldn't deserialize thrift: TProtocolException: Exceeded size limit`) and it explodes the size of the file (e.g., from 400MB to 2GB with the only change being the index). See reprex below:

```r
library(data.table)
library(arrow)

dt<-data.table(x=c(1:1e8), y = round(runif(n=1:1e8, min=1, max=5)))

#Looking at rows where y == 3
dt[y == 3,]

#Creating a new variable, which is done uniformly across all rows (suggesting the previous row index isn't applicable?)
dt[, z := 1]

#Save the dt
write_parquet(dt, "example.parquet")
gc()

#Cannot open the dt
dt_open<-read_parquet("example.parquet")

#Removing indexing that was created when looking at the y==3 subset before saving allows
#the file to be opened after re-saving.
setindex(dt, NULL)

write_parquet(dt, "example2.parquet")
dt_open<-read_parquet("example2.parquet")
```

### Component(s)

Parquet, R

Contributor guide

Open the contributing guide

Research direction

Start by running the provided R reprex with data.table and arrow, comparing parquet output before and after creating an index. Trace the indexed-object write and read path; done means an indexed data.table round-trips through read_parquet without the deserialization error and does not cause the reported file-size inflation.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.