C++ Implementation of data.table::fread
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
Research direction
Start by reviewing dt_stdio.h, fread.c, fread.h, and myomp.h, then trace how fread is exposed through the R package. Define whether a supported C++ interface is feasible and what standalone integration would need to preserve; done should be a concrete interface proposal with validation against existing fread behavior.
Written by the indexing model from the issue text.
Description
How would I go about using data.table directly in C++?
I am huge fan of data.table and use it all the time in R. Honestly, it's the main reason why I continue to use R over Python.
I've been developing some programs in C++ lately and would love to be able to quickly read in large csv files. I have tried the following two solutions:
https://github.com/p-ranav/csv2
https://github.com/ben-strasser/fast-cpp-csv-parser
But I cannot seem to get anything to read in data nearly as fast as data.table can.
From the following post it seems that data.table memory maps the csv files then uses C to quickly parse the information:
https://stackoverflow.com/questions/24424361/reason-behind-speed-of-fread-in-data-table-package-in-r
as mentioned verbose=TRUE helps understand whats going on a little bit more
> fread("test.csv",verbose=TRUE)
Input contains no \n. Taking this to be a filename to open
File opened, filesize is 0.486 GB
File is opened and mapped ok
Detected eol as \n only (no \r afterwards), the UNIX and Mac standard.
Using line 30 to detect sep (the last non blank line in the first 'autostart') ... sep=','
Found 6 columns
First row with 6 fields occurs on line 1 (either column names or first row of data)
All the fields on line 1 are character fields. Treating as the column names.
Count of eol after first data row: 10000001
Subtracted 1 for last eol and any trailing empty lines, leaving 10000000 data rows
Type codes ( first 5 rows): 113431
Type codes (+ middle 5 rows): 113431
Type codes (+ last 5 rows): 113431
Type codes: 113431 (after applying colClasses and integer64)
Type codes: 113431 (after applying drop or select (if supplied)
Allocating 6 column slots (6 - 0 dropped)
Read 10000000 rows and 6 (of 6) columns from 0.486 GB file in 00:00:44
13.420s ( 31%) Memory map (rerun may be quicker)
0.000s ( 0%) sep and header detection
3.210s ( 7%) Count rows (wc -l)
0.000s ( 0%) Column type detection (first, middle and last 5 rows)
1.310s ( 3%) Allocation of 10000000x6 result (xMB) in RAM
25.580s ( 59%) Reading data
0.000s ( 0%) Allocation for type bumps (if any), including gc time if triggered
0.000s ( 0%) Coercing data already read in type bumps (if any)
0.040s ( 0%) Changing na.strings to NA
43.560s Total
is it possible to compile the relevant C files into my C++ project? files such as:
dt_stdio.h, fread.c, fread.h, myomp.h
is there some easier interface outside of R directly to access fread function?
will data.table be available in other languages outside of R? I have seen Python have a package based off this datatable
- Dominant language
- R
- Stars
- 3.9k
- Forks
- 1.1k
- Avg merge
- 14h 4m
- Merged PRs (30d)
- 4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Rdatatable/data.table
-
as.data.table() recurses without end on a survival::Surv object (or any data.frame carrying one) Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Rdatatable/data.table#7887 ·
-
consistency tests
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#7853 · 3 comments ·
-
internals
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#6938 · 1 comment ·
-
encoding fread
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#5179 · 8 comments ·
-
documentation programming
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#3199 · 3 comments ·
All issues in Rdatatable/data.table
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
r-lib/pkgdepends#485 · 3 comments ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
beginners blocker
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enviPathR OpenBuild Error Build OK Build Warning policies-accepted pre-review precheck-passed
Difficulty 1/5 Under an hour Newbie friendliness 84/100
Bioconductor/BiocContributions#207 · 6 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
datacarpentry/semester-biology#1255 ·