[R] How to read (out of memory) a large matrix with an arrow package R
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Hi!
imagine that I have a huge matrix (small example)
```
dat <- matrix(1:100,ncol = 10)
colnames(dat) <- paste0("X",1:ncol(dat))
dat
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10
[1,] 1 11 21 31 41 51 61 71 81 91
[2,] 2 12 22 32 42 52 62 72 82 92
[3,] 3 13 23 33 43 53 63 73 83 93
[4,] 4 14 24 34 44 54 64 74 84 94
[5,] 5 15 25 35 45 55 65 75 85 95
[6,] 6 16 26 36 46 56 66 76 86 96
[7,] 7 17 27 37 47 57 67 77 87 97
```
this matrix is divided into `columns` and written in several files
```
path <- paste0(getwd(),"/example/")
dir.create(path = path)
colum_id <- matrix(1:10,ncol = 2,byrow = T)
for(i in 1:nrow(colum_id)) write.csv(dat[,colum_id[i,]], file = paste0(path,i,".csv"), row.names = F)
```
files in a folder
```
dir(path)
[1] "1.csv" "2.csv" "3.csv" "4.csv" "5.csv"
```
..
```
paths <- paste0(path,dir(path))
lapply(paths , \(Path) head(read.csv(Path),2) )
[[1]]
X1 X2
1 1 11
2 2 12
[[2]]
X3 X4
1 21 31
2 22 32
[[3]]
X5 X6
1 41 51
2 42 52
[[4]]
X7 X8
1 61 71
2 62 72
[[5]]
X9 X10
1 81 91
2 82 92
```
The question is how can I read this matrix (out of memory) in the arrow so that I can access to any element of the full matrix.
All the examples that I saw with the use of the arrow were with reading files separated by lines but not by columns.
### Component(s)
R
Contributor guide
Assessment
This issue has not been assessed yet.