fread for directories

Open
#2,582 12 comments 1 reaction 1 assignee View on GitHub

@MichaelChirico is already working on this.

Since Feb 2, 2018.

Assessment

This issue has not been assessed yet.

Description

fread

Some file I/O APIs I've worked with have a simple idiom for reading full directories:

path/to/
  file1.csv
  file2.csv
  file3.csv

would be read as, e.g. in spark,

spark.read.option("header", "true").csv("/path/to/*.csv")

A basic idiom has developed for fread to do this by adding bells and whistles to the following:

rbindlist(lapply(list.files('path/to', full.names = TRUE), fread))

It would be simple enough to wedge directory reading into the fread API by changing:

if (isTRUE(file.info(input)$isdir)) {
  stop("'input' can not be a directory name, but must be a single character string containing a file name, a command, full path to a file, a URL starting 'http[s]://', 'ftp[s]://' or 'file://', or the input data itself.")
}

to (pseudocode around the match.call() part)

if (isTRUE(file.info(input)$isdir)) {
  return(rbindlist(lapply(list.files(input, pattern = '\\.csv', full.names = TRUE), fread, match.call())))
}

However, it might be nice to build in some flexibility to this, e.g. allowing the list.files to optionally be recursive, implementing some API for automatic source naming (if there are subdirectories, and the names of the subdirectories contain information, the manual version of this allows a bit more flexibility), specifying idcol or fill, etc.

So there's two questions here

  1. Is this worth implementing?
  2. What's the desired API for doing so? New named arguments to fread? Just add ... and post-process if the is.dir branch is reached? Separate function call altogether?
Dominant language
R
Stars
3.9k
Forks
1.1k
Avg merge
14h 4m
Merged PRs (30d)
4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Rdatatable/data.table

All issues in Rdatatable/data.table

Similar issues

More R issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.