Calling fread multiple times on different files with default number of threads causes higher chance of segfault

Start with the reproducible loop calling fread on chr1.csv through chr22.csv after setDTthreads(8), then compare behavior with setDTthreads(1) and 2. Trace the parallel read path used by fread at the reported “Read the data” step and identify why repeated reads can segfault. Done means the reproduction no longer crashes with the default thread count and a regression test covers repeated reads of different files.

Dear data.table developers and users,

Thank you for building this package. It has tremendously helped me in my work. Recently I have to process many csv files one by one in a loop. I am using fread function to quickly load the files and do some other data.table operation. After loading many different files, I encountered segfault (invalid permissions) at step 11 of fread operation:

....
[10] Allocate memory for the datatable
  Allocating 2 column slots (2 - 0 dropped) with 4626114 rows
[11] Read the data
  jumps=[0..48), chunk_size=978545, total_size=46970185

 *** caught segfault ***
address 0x7f806860d38b, cause 'invalid permissions'

Traceback:
 1: fread(paste0("coordinate/chr", i, ".csv"), verbose = T)
An irrecoverable exception occurred. R is aborting now ...
Segmentation fault (core dumped)

As a short-term solution, I use base::read.csv and the segfault goes away.

After going through reading some of other issues here, I found out some different issues dealing with different number of CPU threads. So, I try setDTthreads to 2 (default at 8) just to see if it helps. Yes it did helps and I have a very low chance of getting segfault. I stress-tested with while loop, I still get segfault for n = 2. Then, I setDTthreads to 1, and I never see the segfault so far.

Minimal reproducible example

untar('example.tgz')

library(data.table)

n = 8
setDTthreads(n)

for (i in 1:22) fread(paste0("chr", i, ".csv"), verbose = T)

Attached is the minimal set of csv files that I use. I have tried fread a same single csv file in a loop with the default half number of threads with no issue.

Session Info

> sessionInfo()

R version 4.1.1 (2021-08-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Pop!_OS 21.04

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.1.1

$ uname -r
5.13.0-7614-generic

I am running on Ryzen 5700G CPU.