explosion / explosion/spaCy

Memory leak of MorphAnalysis object.

Open
#13,684 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

I have encountered a crucial bug, which makes running a continuous tokenization using Japanese tokenizer close to impossible. It's all due so memory leak of MorphAnalysis
## How to reproduce the behaviour
```
import spacy
import tracemalloc

tracemalloc.start()
tokenizer = spacy.blank("ja")
tokenizer.add_pipe("sentencizer")

for _ in range(1000):
text = " ".join(["a"] * 1000)
snapshot = tracemalloc.take_snapshot()
with tokenizer.memory_zone():
doc = tokenizer(text)
tokenizer.max_length = len(text) + 10
import gc
gc.collect()
snapshot2 = tracemalloc.take_snapshot()
# Compare the two snapshots
p_stats = snapshot2.compare_to(snapshot, "lineno")
# Pretty print the top 10 differences
print("[ Top 10 ]")
# Stop here with pdb
for stat in p_stats[:10]:
if stat.size_diff > 0:

print(stat)
```

Run this script and observe how memory keeps growing:
![image](https://github.com/user-attachments/assets/9c6ef7b2-c243-43bd-af28-71da92dcbe21)
It all happens due to the this line:
`token.morph = MorphAnalysis(self.vocab, morph)`. I have checked the implementation itself and there is neither code for dealocation implemented, nor it supports the memory_zone.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.