optional arg that returns a list of parse symbols fread() used to intuit raw file

Open
#2,437 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
r
Domain
data

Research direction

Start by reviewing fread() and the parser decisions described in this issue, then compare the proposed schema workflow with the prototype linked in datzen's scheme.md. Clarify which inferred delimiters and other parsing settings must be exposed, how the optional result should be represented, and what tests would demonstrate that the returned schema matches fread()'s behavior.

Written by the indexing model from the issue text.

Description

feature request fread

I think it would be a terrific feature addition if some kind of argument like fread(return_parse=TRUE) was implemented.

I know the developers have put in a lot of work to 'intuit / guess ' these type of delimiter characters in the fread() body, such as

https://github.com/Rdatatable/data.table/pull/2436
https://github.com/Rdatatable/data.table/issues/2248
https://github.com/Rdatatable/data.table/issues/2431

All of these internal parsing decisions would be a benefit if it was optionally returned to the end user outside of fread()

This request is motivated by the rant blogged about here
http://www.johnmyleswhite.com/notebook/2016/09/23/no-juice-for-you-csv-format-it-just-makes-you-more-awful/

Basically, i think a 'csv schema' would be useful for the type of workflow

raw csv -> fread -> user transforms data -> schema -> write external transformed data

I mocked up a protottype of a function that writes this basic schema here

https://github.com/mikejacktzen/datzen/blob/master/man_md/scheme.md

and realized that fread() internally tries to detect these types of raw parser characters

edit 10/24 for more use case context

I see this being useful if no pre-existing schema / preamble / yaml for csvy exists.
So the user wishes to create one painlessly using fread()

I do not think many people exhaustively lists the parser spec first.
If, like me, people ham handedly read the raw file into R with fread, we rely on fread to guess the spec (the hard work).
Once the fread generated spec is there, the user can then write out the scheme to whatever format they want, json, yaml, the preamble for the csvy.

Dominant language
R
Stars
3.9k
Forks
1.1k
Avg merge
14h 4m
Merged PRs (30d)
4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Rdatatable/data.table

All issues in Rdatatable/data.table

Similar issues

More R issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.