Error in reading large file in R

Open
#5,311 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
r
Domain
data

Research direction

Start with the fread("phyloseq.txt", ...) call and the supplied sessionInfo: R 3.6.1, data.table 1.12.8, and CentOS 7. Reproduce the segfault with a comparable 20GB tab-separated file, then inspect existing data.table fread issues and tests. Done means the failure is explained and reproducible, or a verified fix or documented limitation is identified.

Written by the indexing model from the issue text.

Description

fread

Hi,

I am trying to read the large file (20GB) in R on the HPC server but getting an error, please suggest how I can fix it.

Error;

> phyloseq <- fread("phyloseq.txt",header=T,sep='\t',check.names=F,fill=TRUE)
 *** caught segfault ***
address 0x7f9d8fac2d44, cause 'memory not mapped'

Traceback:
 1: fread("phyloseq.txt", header = T, sep = "\t", check.names = F,     fill = TRUE)

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace




> sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS:   /opt/gridware/depots/54e7fb3c/el7/pkg/apps/R/3.6.1/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/lib/libRblas.so
LAPACK: /opt/gridware/depots/54e7fb3c/el7/pkg/apps/R/3.6.1/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_GB.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_GB.UTF-8        LC_COLLATE=en_GB.UTF-8
 [5] LC_MONETARY=en_GB.UTF-8    LC_MESSAGES=en_GB.UTF-8
 [7] LC_PAPER=en_GB.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] data.table_1.12.8

loaded via a namespace (and not attached):
[1] compiler_3.6.1

Many thanks,

Dominant language
R
Stars
3.9k
Forks
1.1k
Avg merge
14h 4m
Merged PRs (30d)
4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Rdatatable/data.table

All issues in Rdatatable/data.table

Similar issues

More R issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.