Comparison with fast C++ csv parser
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Domain
- data-engineering, performance
Research direction
Review the linked readr proof-of-concept commit, the readr:::read_trip_fare and data.table::fread benchmark commands, and the fast C++ CSV parser it references. Compare the stated timings and parser assumptions first; the issue does not define a concrete implementation or completion criterion.
Written by the indexing model from the issue text.
Description
In https://github.com/jimhester/readr/commit/33b793621c33b915e896fb3778c5a47152ccd73d @jimhester implements a proof-of-concept of https://github.com/ben-strasser/fast-cpp-csv-parser with impressive timings on a static 1.56 GB file (especially on the 'hot' second timing).
> system.time(y <- readr:::read_trip_fare(normalizePath("trip_fare_1.csv")))
user system elapsed
19.97 1.01 20.15
> system.time(y <- data.table::fread(normalizePath("trip_fare_1.csv")))
|--------------------------------------------------|
|==================================================|
user system elapsed
23.88 0.75 18.24
> system.time(y <- readr:::read_trip_fare(normalizePath("trip_fare_1.csv")))
user system elapsed
12.81 1.08 12.91
> system.time(y <- data.table::fread(normalizePath("trip_fare_1.csv")))
|--------------------------------------------------|
|==================================================|
user system elapsed
24.36 0.66 17.92
> dim(y)
[1] 14776615 11
The purpose of this issue is recognize its performance and broadcast awareness of this parser, to anticipate comparisons with fread and to gather what, if anything, can be learnt from this implementation. From what I understand, the function requires a lot of knowledge of the csv's structure well in advance of it being read. (In data.table parlance, perhaps, a 'fast and unfriendly file finagler'.) Nonetheless I believe there is a use-case for such a function: a kind-of plain-text cached version could be very valuable if fast.
- Dominant language
- R
- Stars
- 3.9k
- Forks
- 1.1k
- Avg merge
- 14h 4m
- Merged PRs (30d)
- 4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Rdatatable/data.table
-
as.data.table() recurses without end on a survival::Surv object (or any data.frame carrying one) Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Rdatatable/data.table#7887 ·
-
consistency tests
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#7853 · 3 comments ·
-
internals
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#6938 · 1 comment ·
-
encoding fread
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#5179 · 8 comments ·
-
documentation programming
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#3199 · 3 comments ·
All issues in Rdatatable/data.table
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
r-lib/pkgdepends#485 · 3 comments ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
beginners blocker
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enviPathR OpenBuild Error Build OK Build Warning policies-accepted pre-review precheck-passed
Difficulty 1/5 Under an hour Newbie friendliness 84/100
Bioconductor/BiocContributions#207 · 6 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
datacarpentry/semester-biology#1255 ·