fread memory maps entire file even when select= is used to restrict to specific columns

Start with the fread implementation and the behavior discussed in issue #4329; run the supplied R reproduction using select= on a very wide file. Done means the selected six columns can be read without attempting to memory-map the entire input or requiring its full virtual-memory footprint.

I am attempting to use fread with the select= option to read the first 6 columns from a file that has 408485 columns and and 125616 rows (including the header).

It fails with the following message:

Error in data.table::fread(infile, header = T, select = cnames[1:6],  : 
  Opened 191.2GB (205254407682 bytes) file ok but could not memory map it. This is a 64bit process. There is probably not enough contiguous virtual memory available.

I see that there has been a previous fixed issue where fread used to do the same thing when nrows= was specified, and that issue has been fixed. But I do not see any previous bug reports for this happening when select= is used to select columns.

(The file is extremely large because this is genetic data. The first six columns are information about what genetic variants are included in each file, the remaining columns that I don't want to read are genotypes for people in the sample. For now I am using command-line tools to extract the first 6 columns from each file so they can be read into R, but it would be nice to be able to do this all in a single step in R.)

Output of sessionInfo():

R version 4.4.2 (2024-10-31)
Platform: x86_64-pc-linux-gnu
Running under: Rocky Linux 8.10 (Green Obsidian)

Matrix products: default
BLAS/LAPACK: [path redacted for github post]/libopenblasp-r0.3.28.so;  LAPACK version 3.12.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: [redacted for github post]
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.16.4

loaded via a namespace (and not attached):
 [1] compiler_4.4.2      bigparallelr_0.3.2  flock_0.7          
 [4] parallel_4.4.2      Rcpp_1.0.13-1       codetools_0.2-20   
 [7] doParallel_1.0.17   iterators_1.0.14    foreach_1.5.2      
[10] bigassertr_0.1.6    RhpcBLASctl_0.23-42

Reproducible example:

# How much fake data to make
num_cols <- 408485
num_datarows <- 125615
num_samples <- num_cols - 6

# Make fake SNPs and fake sample IDs
set.seed(1)
snpmeta <- data.frame(CHR=rep(2,num_datarows),SNP=paste0('rs',1:num_datarows),`(C)M`=0, POS=1:num_datarows,
  COUNTED=sample(c('A','C'),num_datarows,replace=T), ALT=sample(c('T','G'),num_datarows,replace=T))
samp_ids <- sample(100000:999999,num_samples,replace=F)
samp_ids <- paste0(samp_ids,'_',samp_ids)

# Write the fake file row-by-row so we don't fill memory with fake genotypes
outfile <- file(paste0('fakedat.traw'),"wt")
writeLines(paste(c('CHR','SNP','(C)M','POS','COUNTED','ALT',samp_ids),collapse='\t'),con=outfile) # Header row
for(i in 1:num_datarows) {
  genos <- sample(c(0,1,2,NA),num_samples,replace=T)
  writeLines(paste(c(unlist(snpmeta[i,]),genos),collapse='\t'),con=outfile) # Data for the ith row
}
close(outfile)

readmeta <- fread('fakedat.traw',select=c('CHR','SNP','(C)M','POS','COUNTED','ALT'),header=T,stringsAsFactors=F,data.table=F)

I have tested this with smaller files to make sure it's not a problem with how I specify columns to select=. With smaller 17-40 Gb files the fread command completes successfully. It still uses large amounts of memory while reading the file, then when it finishes the memory usage abruptly drops and the result is the requested 6 columns of data.