fwrite: avoid converting matrix input to data.table
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
Research direction
Start in fwriteR.c at the loop that populates args.columns[] and trace how those pointers are passed to fwrite.c. Verify how matrix storage and its dimension attribute can be handled without conversion to data.table. Done means matrix input avoids the conversion while preserving existing behavior, with tests covering factor and character matrices.
Written by the indexing model from the issue text.
Description
PR #3125 implemented #2613 to enable to fwrite to accept matrix. This was merged in v1.12.0. However it converts matrix to data.table which can be costly.
The convert can be avoided as follows.
fwriteR.c contains this :
for (int j=0; j<args.ncol; j++) {
SEXP column = VECTOR_ELT(DFcoerced, j);
if (args.nrow != length(column))
error("Column %d's length (%d) is not the same as column 1's length (%d)", j+1, length(column), args.nrow);
int32_t wf = whichWriter(column);
if (wf<0) {
error("Column %d's type is '%s' - not yet implemented in fwrite.", j+1, type2char(TYPEOF(column)));
}
args.columns[j] = (wf==WF_CategString ? column : (void *)DATAPTR(column));
args.whichFun[j] = (uint8_t)wf;
if (TYPEOF(column)==VECSXP && firstListColumn==0) firstListColumn = j+1;
}
This loop populates the pointers in arg.columns[] which are then passed to fwrite.c. That DATAPTR returns a pointer to where the data (e.g. int *, double *) for the R vector starts. In the case of matrix, this args.columns[] just needs to be populated with offsets into matrix vector. In R a matrix is a single very long vector just with dimension attribute attached. A matrix in R is columnar, just like a data.table, so there doesn't need to be a transpose. It should be fairly simple to do and work well.
Careful to add tests for a factor matrix as well as a character matrix.
I described how to do it in case @fparages or @MichaelChirico wanted to give it a go.
- Dominant language
- R
- Stars
- 3.9k
- Forks
- 1.1k
- Avg merge
- 14h 4m
- Merged PRs (30d)
- 4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Rdatatable/data.table
-
as.data.table() recurses without end on a survival::Surv object (or any data.frame carrying one) Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Rdatatable/data.table#7887 ·
-
consistency tests
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#7853 · 3 comments ·
-
internals
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#6938 · 1 comment ·
-
encoding fread
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#5179 · 8 comments ·
-
documentation programming
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#3199 · 3 comments ·
All issues in Rdatatable/data.table
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
r-lib/pkgdepends#485 · 3 comments ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
beginners blocker
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enviPathR OpenBuild Error Build OK Build Warning policies-accepted pre-review precheck-passed
Difficulty 1/5 Under an hour Newbie friendliness 84/100
Bioconductor/BiocContributions#207 · 6 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
datacarpentry/semester-biology#1255 ·