Sometimes `pushBuffer` calls the R allocator from the wrong thread

Start by running the provided R and shell reproduction with OMP_THREAD_LIMIT=2. Inspect src/fread.c around line 2392, src/freadR.c around line 612, and the postprocessBuffer call at line 2602 to trace the OpenMP thread coordination. Done means the reproduction no longer reaches R allocator calls from the wrong thread or crashes with the reported fatal errors.

# Minimal reproducible example

This one is weird. Might depend on timing or address space layout, I'm not sure.

export OMP_THREAD_LIMIT=2 # 4-thread processor with default limits => 2 threads
while R -q -s -e '
for (i in 100:99) {
 writeLines(c("a", rep("", i), strrep("a", 10000000), rep("", 50)), "foo.txt")
 data.table::fread("foo.txt") -> foo
}'; do :; done

Replace the shell while loop with an R repeat loop, or remove the R for loop, and it refuses to reproduce. After running for a few tens of seconds, the above command gives me a recursive crash:

Fatal error: Wrong thread calling 'RunFinalizers'
Fatal error: Wrong thread calling 'RunFinalizers'
Fatal error: Wrong thread calling 'RunFinalizers'
Segmentation fault

Trying the same thing with R -d gdb and pressing r a lot, I get:

#24889 0x00005555557aac5a in R_Suicide (s=s@entry=0x7ffff0a06d00 "Wrong thread calling 'RunFinalizers'") at system.c:70
#24890 0x00005555556c44bc in R_check_thread (s=0x5555557d10a8 <__func__.38> "RunFinalizers") at memory.c:3180
#24891 R_check_thread (s=0x5555557d10a8 <__func__.38> "RunFinalizers") at memory.c:3167
#24892 RunFinalizers () at memory.c:1536
#24893 0x00005555556c478a in R_RunExitFinalizers () at memory.c:1619
#24894 0x00005555557aa28e in Rstd_CleanUp (saveact=SA_SUICIDE, status=2, runLast=0) at sys-std.c:1289
#24895 0x00005555557aad19 in R_CleanUp (saveact=saveact@entry=SA_SUICIDE, status=status@entry=2, runLast=runLast@entry=0) at system.c:87
#24896 0x00005555557a9136 in Rstd_Suicide (s=<optimized out>) at sys-std.c:74
#24897 0x00005555557aac5a in R_Suicide (s=s@entry=0x7ffff0a07570 "Wrong thread calling 'Rf_protect'") at system.c:70
#24898 0x00005555556b5fda in R_check_thread (s=s@entry=0x5555557d1088 <__func__.30> "Rf_protect") at memory.c:3180
#24899 0x00005555556b7608 in R_check_thread (s=0x5555557d1088 <__func__.30> "Rf_protect") at memory.c:3175
#24900 Rf_protect (s=s@entry=0x7fffec865340) at memory.c:3428
#24901 0x000055555565b908 in Rf_mkCharLenCE (name=0x7ffff3400066 'a' <repeats 200 times>..., len=10000000, enc=<optimized out>) at envir.c:4365
#24902 0x000055555565be47 in Rf_mkCharLenCE (name=<optimized out>, len=<optimized out>, enc=<optimized out>) at envir.c:4467
#24903 0x00007ffff41b2065 in pushBuffer (ctx=ctx@entry=0x7ffff0a07c40) at freadR.c:612
#24904 0x00007ffff41a85f6 in freadMain._omp_fn.0 () at fread.c:2392

(gdb) frame 24904
#24904 0x00007ffff41a85f6 in freadMain._omp_fn.0 () at fread.c:2392
2392              pushBuffer(&ctx);

https://github.com/Rdatatable/data.table/blob/39e2875a17fdf1ef6fe05cf79f48ffd6613808de/src/fread.c#L2392

(gdb) frame 24903
#24903 0x00007ffff41b2065 in pushBuffer (ctx=ctx@entry=0x7ffff0a07c40) at freadR.c:612
612                   SET_STRING_ELT(dest, ctx->DTi + i, mkCharLenCE(str, strLen, ienc));

https://github.com/Rdatatable/data.table/blob/39e2875a17fdf1ef6fe05cf79f48ffd6613808de/src/freadR.c#L612

Meanwhile, thread 1 is in a different part of fread:

#0  0x00007ffff7a1b449 in ?? () from /usr/lib/x86_64-linux-gnu/libgomp.so.1
#1  0x00007ffff7a0f087 in GOMP_ordered_start () from /usr/lib/x86_64-linux-gnu/libgomp.so.1
#2  0x00007ffff41a76b3 in freadMain._omp_fn.0 () at fread.c:2602
#3  0x00007ffff7a100b6 in GOMP_parallel () from /usr/lib/x86_64-linux-gnu/libgomp.so.1
#4  0x00007ffff41ab538 in freadMain (_args=...) at fread.c:2331
#5  0x00007ffff41b035d in freadR (inputArg=<optimized out>, isFileNameArg=<optimized out>, sepArg=<optimized out>, decArg=<optimized out>,
    quoteArg=<optimized out>, headerArg=0x5555559a1040, nrowLimitArg=0x555556584848, skipArg=0x555557b999a0, NAstringsArg=0x5555565847a0,
    stripWhiteArg=0x555556584570, skipEmptyLinesArg=0x555556584500, fillArg=0x555558d4f838, showProgressArg=0x5555559a1078,
    nThreadArg=0x555558d4f528, verboseArg=0x555558d4fe58, warnings2errorsArg=0x5555559a1078, logical01Arg=0x5555565843b0,
    logicalYNArg=0x555556584340, selectArg=0x55555599f190, dropArg=0x55555599f190, colClassesArg=0x55555599f190, integer64Arg=0x555556584618,
    encodingArg=0x5555565845a8, keepLeadingZerosArgs=0x5555565842d0, noTZasUTC=0x555557b99930) at freadR.c:226
(gdb) frame 2
#2  0x00007ffff41a76b3 in freadMain._omp_fn.0 () at fread.c:2602
2602            postprocessBuffer(&ctx);

https://github.com/Rdatatable/data.table/blob/39e2875a17fdf1ef6fe05cf79f48ffd6613808de/src/fread.c#L2602

Somehow, the critical section in pushBuffer() is almost, but not quite always enough to run mkCharLenCE() on the main thread. And yet sometimes it runs on the non-main thread and blows up the process. This needs more investigation.

# Output of sessionInfo()

R Under development (unstable) (2025-07-26 r88457)
Platform: x86_64-pc-linux-gnu
Running under: Debian GNU/Linux 12 (bookworm)

Matrix products: default
BLAS:   REDACTED/R-devel/lib/libRblas.so
LAPACK: REDACTED/R-devel/lib/libRlapack.so;  LAPACK version 3.12.1

locale:
 [1] LC_CTYPE=ru_RU.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=ru_RU.UTF-8        LC_COLLATE=ru_RU.UTF-8
 [5] LC_MONETARY=ru_RU.UTF-8    LC_MESSAGES=ru_RU.UTF-8
 [7] LC_PAPER=ru_RU.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=ru_RU.UTF-8 LC_IDENTIFICATION=C

time zone: Europe/Moscow
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] data.table_1.17.99

loaded via a namespace (and not attached):
[1] compiler_4.6.0