memory leak on linux

Hi,

I am preprocessing a large dataset: "HuggingFaceTB/smollm-corpus", specifically "fineweb-edu-dedup" using the simple `preprocess.py` script from nanotron (`nanotron/tools/preprocess_data.py`) on my linux machine.

I encountered a significant memory leak issue where RAM usage would grow continuously with each batch, eventually crashing the process. The issue appears to be related to the interaction between HuggingFace's caching/multi-threading optimizations and Linux's default memory allocator (ptmalloc2). I resolved it by switching to jemalloc (https://github.com/jemalloc/jemalloc).

Environment details:
- tokenizers: 0.21.0
- datasets: 3.1.0
- python: 3.11.6
- OS: Linux

While this might be more of a system-level issue than a HuggingFace-specific one, I thought it would be valuable to document this for other users who might encounter similar memory problems, especially when processing large datasets.

Best,
Klara