C++ Implementation of data.table::fread

Open
#5,343 10 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
c, cpp, r
Domain
data

Research direction

Start by reviewing dt_stdio.h, fread.c, fread.h, and myomp.h, then trace how fread is exposed through the R package. Define whether a supported C++ interface is feasible and what standalone integration would need to preserve; done should be a concrete interface proposal with validation against existing fread behavior.

Written by the indexing model from the issue text.

Description

fread question

How would I go about using data.table directly in C++?

I am huge fan of data.table and use it all the time in R. Honestly, it's the main reason why I continue to use R over Python.

I've been developing some programs in C++ lately and would love to be able to quickly read in large csv files. I have tried the following two solutions:
https://github.com/p-ranav/csv2
https://github.com/ben-strasser/fast-cpp-csv-parser

But I cannot seem to get anything to read in data nearly as fast as data.table can.

From the following post it seems that data.table memory maps the csv files then uses C to quickly parse the information:
https://stackoverflow.com/questions/24424361/reason-behind-speed-of-fread-in-data-table-package-in-r

as mentioned verbose=TRUE helps understand whats going on a little bit more

> fread("test.csv",verbose=TRUE)
Input contains no \n. Taking this to be a filename to open
File opened, filesize is 0.486 GB
File is opened and mapped ok
Detected eol as \n only (no \r afterwards), the UNIX and Mac standard.
Using line 30 to detect sep (the last non blank line in the first 'autostart') ... sep=','
Found 6 columns
First row with 6 fields occurs on line 1 (either column names or first row of data)
All the fields on line 1 are character fields. Treating as the column names.
Count of eol after first data row: 10000001
Subtracted 1 for last eol and any trailing empty lines, leaving 10000000 data rows
Type codes (   first 5 rows): 113431
Type codes (+ middle 5 rows): 113431
Type codes (+   last 5 rows): 113431
Type codes: 113431 (after applying colClasses and integer64)
Type codes: 113431 (after applying drop or select (if supplied)
Allocating 6 column slots (6 - 0 dropped)
Read 10000000 rows and 6 (of 6) columns from 0.486 GB file in 00:00:44
  13.420s ( 31%) Memory map (rerun may be quicker)
   0.000s (  0%) sep and header detection
   3.210s (  7%) Count rows (wc -l)
   0.000s (  0%) Column type detection (first, middle and last 5 rows)
   1.310s (  3%) Allocation of 10000000x6 result (xMB) in RAM
  25.580s ( 59%) Reading data
   0.000s (  0%) Allocation for type bumps (if any), including gc time if triggered
   0.000s (  0%) Coercing data already read in type bumps (if any)
   0.040s (  0%) Changing na.strings to NA
  43.560s        Total

is it possible to compile the relevant C files into my C++ project? files such as:
dt_stdio.h, fread.c, fread.h, myomp.h
is there some easier interface outside of R directly to access fread function?

will data.table be available in other languages outside of R? I have seen Python have a package based off this datatable

Dominant language
R
Stars
3.9k
Forks
1.1k
Avg merge
14h 4m
Merged PRs (30d)
4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Rdatatable/data.table

All issues in Rdatatable/data.table

Similar issues

More R issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.