huggingface / huggingface/tokenizers
Risk of global variable memory leaks when calling train_from_iterator
- Dominant language
- Rust
- Stars
- 11k
- Forks
- 1.2k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 26
Description
Recently we found that global variables of classes bound by nanobind would leak if we use the `train_from_iterator` method in `tokenizers` with a class based iterator (yield works fine). You could find the detailed information and the [minimal reproduction code](https://github.com/wjakob/nanobind/issues/644#issue-2420893804) in the following issues.
* https://github.com/Natooz/MidiTok/issues/190
* https://github.com/Yikai-Liao/symusic/issues/49
* https://github.com/wjakob/nanobind/issues/644
The maintainer of nanobind said that nanobind just detects such leak, and it is not a bug of nanobind.
And here is what he said in `nanobind`'s document:
> **Interactions with other tools that leak references**. Python extension libraries—especially huge ones with C library components like PyTorch, Tensorflow, etc., have been observed to leak references to nanobind objects.
> Some of these frameworks cache JIT-compiled functions based on the arguments with which they were called, and such caching schemes could leak references to nanobind types if they aren’t cleaned up by the responsible extensions (this is a hypothesis). In this case, the leak would be benign—even so, it should be fixed in the responsible framework so that leak warnings aren’t cluttered with flukes and can be more broadly useful.
Well, I don't find jit in tokenizers. There could be other reasons, or maybe its a bug of cpython. But I hope this better not be a python's bug, otherwise this bug may only be removed in a higher version of python (3.14, or later).
Contributor guide
Assessment
This issue has not been assessed yet.