apache / apache/arrow

[R] How to read (out of memory) a large matrix with an arrow package R

Open
#37,103 1 comment 0 reactions 0 assignees View on GitHub
Component: R Type: usage
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Hi!
imagine that I have a huge matrix (small example)
```
dat <- matrix(1:100,ncol = 10)
colnames(dat) <- paste0("X",1:ncol(dat))
dat
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10
[1,] 1 11 21 31 41 51 61 71 81 91
[2,] 2 12 22 32 42 52 62 72 82 92
[3,] 3 13 23 33 43 53 63 73 83 93
[4,] 4 14 24 34 44 54 64 74 84 94
[5,] 5 15 25 35 45 55 65 75 85 95
[6,] 6 16 26 36 46 56 66 76 86 96
[7,] 7 17 27 37 47 57 67 77 87 97
```
this matrix is divided into `columns` and written in several files
```
path <- paste0(getwd(),"/example/")
dir.create(path = path)
colum_id <- matrix(1:10,ncol = 2,byrow = T)
for(i in 1:nrow(colum_id)) write.csv(dat[,colum_id[i,]], file = paste0(path,i,".csv"), row.names = F)
```
files in a folder
```
dir(path)
[1] "1.csv" "2.csv" "3.csv" "4.csv" "5.csv"
```
..
```
paths <- paste0(path,dir(path))
lapply(paths , \(Path) head(read.csv(Path),2) )

[[1]]
X1 X2
1 1 11
2 2 12

[[2]]
X3 X4
1 21 31
2 22 32

[[3]]
X5 X6
1 41 51
2 42 52

[[4]]
X7 X8
1 61 71
2 62 72

[[5]]
X9 X10
1 81 91
2 82 92
```
The question is how can I read this matrix (out of memory) in the arrow so that I can access to any element of the full matrix.
All the examples that I saw with the use of the arrow were with reading files separated by lines but not by columns.

### Component(s)

R

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.