[C++] Add option to disable jemalloc background thread on Linux
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
We noticed that after we upgraded pyarrow to version 0.17.1 from 0.9.0 we now get two unwanted side effects just by linking in arrow libs into our c++ build. We link arrow libs into one of our low level core libraries so now all of our applications are affected by these side effects:
1) a "jemalloc_bg_thd" thread is spawned on dlinit before we ever hit main
2) all our apps are now hitting valgrind leak warnings due to a (potential) leak in jemalloc code:
==33515== 656 bytes in 1 blocks are possibly lost in loss record 1 of 1
==33515== at 0x402E9EA: calloc (vg_replace_malloc.c:752)
==33515== by 0x4011F44: _dl_allocate_tls (in /usr/lib64/ld-2.17.so)
==33515== by 0x5DFF9C0: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.17.so)
==33515== by 0x589186B: je_arrow_private_je_pthread_create_wrapper (background_thread.c:48)
==33515== by 0x589186B: background_thread_create_signals_masked (background_thread.c:365)
==33515== by 0x589186B: background_thread_create_locked (background_thread.c:573)
==33515== by 0x5891A47: je_arrow_private_je_background_thread_create (background_thread.c:598)
==33515== by 0x400F502: _dl_init (in /usr/lib64/ld-2.17.so)
==33515== by 0x40011A9: ??? (in /usr/lib64/ld-2.17.so)
**Reporter**: [Rob Ambalu](https://issues.apache.org/jira/browse/ARROW-9530) / @robambalu
#### PRs and other links:
- [GitHub Pull Request #10240](https://github.com/apache/arrow/pull/10240)
**Note**: *This issue was originally created as [ARROW-9530](https://issues.apache.org/jira/browse/ARROW-9530). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
The report points to jemalloc's background_thread.c and startup in _dl_init; first review PR #10240 and trace those reported entry points. Done means a Linux-specific option can disable the background thread before applications reach main and the reported Valgrind warning is addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, linux
- Domain
- operating-systems, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100