Numbers become char in fread/fwrite roundtrip

Open
#4,802 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
r
Domain
data

Research direction

Start by reproducing the issue's tmp.csv example with fread(), numeric coercion, fwrite(), and a second fread(). Read the fread/fwrite type-detection and roundtrip behavior involved; completion should establish and test the intended handling of large numeric values, including the colClasses override behavior.

Written by the indexing model from the issue text.

Description

fread

Consider the following tmp.csv:

writeLines("q01
-0.207623873704532
-1.4852856113776
-2432100838991580000
-2.33192914235436
-44.8581139833089", tmp <- tempfile())

There's a big number in it. We'll get a character column from this csv:

library(data.table)

dt <- fread(tmp)
dt

                    q01
                 <char>
1:   -0.207623873704532
2:     -1.4852856113776
3: -2432100838991580000
4:    -2.33192914235436
5:    -44.8581139833089

Coerce the column into numeric:

dt[, q01 := as.numeric(q01)]
dt

             q01
           <num>
1: -2.076239e-01
2: -1.485286e+00
3: -2.432101e+18
4: -2.331929e+00
5: -4.485811e+01

Then fwrite it to another csv and re-fread it:

fwrite(dt, tmp2 <- tempfile())

dt2 <- fread(tmp2)
dt2

We still get character column.

                    q01
                 <char>
1:   -0.207623873704532
2:     -1.4852856113776
3: -2432100838991580000
4:    -2.33192914235436
5:    -44.8581139833089

I'm not sure if it is intended behavior. It makes sense if we want to preserve all information but such behavior could be surprising if user assumes that numbers written by fwrite() could be read into numbers by fread(). In this example, it is not the case. Even if we specify colClasses = "numeric", it won't work either:

> dt2 <- fread(tmp2, colClasses = "numeric")
Warning message:
In fread(tmp2, colClasses = "numeric") :
  Attempt to override column 1 <<q01>> of inherent type 'string' down to 'float64' ignored. Only overrides to a higher type are currently supported. If this was intended, please coerce to the lower type afterwards.

> dt2
                    q01
                 <char>
1:   -0.207623873704532
2:     -1.4852856113776
3: -2432100838991580000
4:    -2.33192914235436
5:    -44.8581139833089
Dominant language
R
Stars
3.9k
Forks
1.1k
Avg merge
14h 4m
Merged PRs (30d)
4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Rdatatable/data.table

All issues in Rdatatable/data.table

Similar issues

More R issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.