Memory leak of MorphAnalysis object.
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
I have encountered a crucial bug, which makes running a continuous tokenization using Japanese tokenizer close to impossible. It's all due so memory leak of MorphAnalysis
## How to reproduce the behaviour
```
import spacy
import tracemalloc
tracemalloc.start()
tokenizer = spacy.blank("ja")
tokenizer.add_pipe("sentencizer")
for _ in range(1000):
text = " ".join(["a"] * 1000)
snapshot = tracemalloc.take_snapshot()
with tokenizer.memory_zone():
doc = tokenizer(text)
tokenizer.max_length = len(text) + 10
import gc
gc.collect()
snapshot2 = tracemalloc.take_snapshot()
# Compare the two snapshots
p_stats = snapshot2.compare_to(snapshot, "lineno")
# Pretty print the top 10 differences
print("[ Top 10 ]")
# Stop here with pdb
for stat in p_stats[:10]:
if stat.size_diff > 0:
print(stat)
```
Run this script and observe how memory keeps growing:

It all happens due to the this line:
`token.morph = MorphAnalysis(self.vocab, morph)`. I have checked the implementation itself and there is neither code for dealocation implemented, nor it supports the memory_zone.
Contributor guide
Assessment
This issue has not been assessed yet.