fpaste: fwrite output as a character vector

Open
#4,572 8 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
r
Domain
data

Research direction

Start by tracing the existing fwrite entry point and how fread handles line-oriented output. Determine how fwrite could return a character vector while preserving the requested separator and omitting column names, then verify that the result matches the fpaste examples and benchmark behavior described in the issue.

Written by the indexing model from the issue text.

Description

fwrite top request

Given the speed of fwrite, it can be used in conjunction with fread as an alternative to do.call(paste, ...) to flatten multiple columns into a character vector. It would be nice to be able to capture the output of fwrite directly as a character vector.

It is much faster than some of the other idiomatic approaches that are often considered.

Here's the behavior I'm hoping to be able to replicate:

fpaste <- function(dt, sep = ",") {
  x <- tempfile()
  fwrite(dt, file = x, sep = sep, col.names = FALSE)
  fread(x, sep = "\n", header = FALSE)
}

d <- data.frame(a = 1:3, b = c('a','b','c'), c = c('d','e','f'), d = c('g','h','i')) 
cols = c("b", "c", "d")

fpaste(d[cols], "-")
#       V1
# 1: a-d-g
# 2: b-e-h
# 3: c-f-i

Here's a comparison with a straightforward paste in a data.table:

set.seed(1) 
d2 <- d[sample(1:3,1e6,TRUE),]
d3 <- as.data.table(d2)

bench::mark(fpaste(d2[cols], "-")$V1, d3[, paste(b, c, d, sep = "-")])
## # A tibble: 2 x 13
##   expression                          min  median `itr/sec` mem_alloc `gc/sec` n_itr  n_gc total_time
##   <bch:expr>                      <bch:t> <bch:t>     <dbl> <bch:byt>    <dbl> <int> <dbl>   <bch:tm>
## 1 fpaste(d2[cols], "-")$V1         90.2ms  93.8ms     10.8     8.41MB     3.60     3     1      278ms
## 2 d3[, paste(b, c, d, sep = "-")] 220.9ms 223.2ms      4.43   30.55MB     0        3     0      678ms
## # … with 4 more variables: result <list>, memory <list>, time <list>, gc <list>
Dominant language
R
Stars
3.9k
Forks
1.1k
Avg merge
14h 4m
Merged PRs (30d)
4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Rdatatable/data.table

All issues in Rdatatable/data.table

Similar issues

More R issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.