[R] Parquet files of indexed data.table objects error in read_parquet
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
When a data.table object is indexed (e.g., with `setindex()`), this can cause problems in reading the parquet back in (`Error: IOError: Couldn't deserialize thrift: TProtocolException: Exceeded size limit`) and it explodes the size of the file (e.g., from 400MB to 2GB with the only change being the index). See reprex below:
```r
library(data.table)
library(arrow)
dt<-data.table(x=c(1:1e8), y = round(runif(n=1:1e8, min=1, max=5)))
#Looking at rows where y == 3
dt[y == 3,]
#Creating a new variable, which is done uniformly across all rows (suggesting the previous row index isn't applicable?)
dt[, z := 1]
#Save the dt
write_parquet(dt, "example.parquet")
gc()
#Cannot open the dt
dt_open<-read_parquet("example.parquet")
#Removing indexing that was created when looking at the y==3 subset before saving allows
#the file to be opened after re-saving.
setindex(dt, NULL)
write_parquet(dt, "example2.parquet")
dt_open<-read_parquet("example2.parquet")
```
### Component(s)
Parquet, R
Contributor guide
Research direction
Start by running the provided R reprex with data.table and arrow, comparing parquet output before and after creating an index. Trace the indexed-object write and read path; done means an indexed data.table round-trips through read_parquet without the deserialization error and does not cause the reported file-size inflation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100