RAM OOM

### Bug
RAM starts building up when processing some pdf files, and it keeps building up eating up all memory (128 GB). It happens for specific pdf file, which contain some hyperlinks connecting diagrams spanning over several pages. Apologies, I cannot share sample file to test.
...

### Steps to reproduce
I have tried disabling/enabling table extraction, cell matching, OCR, different VLM models. All in vain. Things got better once I replaced the backend with `PyPdfiumDocumentBackend`.

Settings snippet:
```
accelerator_set = AcceleratorOptions(num_threads=NUM_CORES//2, device=device, cuda_use_flash_attention2=flash_att)

from docling.backend.docling_parse_v2_backend import DoclingParseV2DocumentBackend
from docling.backend.pypdfium2_backend import PyPdfiumDocumentBackend
from docling.datamodel.pipeline_options import (
VlmPipelineOptions,
)
from docling.pipeline.vlm_pipeline import VlmPipeline

### Creating individual type pipelines
#* PDF Pipeline
# pdf_pipe_options = PdfPipelineOptions() # Define pipeline
pdf_pipe_options = VlmPipelineOptions() # Define pipeline
pdf_pipe_options.accelerator_options = accelerator_set # Setting accelator options if available

from docling.datamodel import vlm_model_specs

from docling.datamodel.pipeline_options_vlm_model import (
InferenceFramework,
InlineVlmOptions,
ResponseFormat,
TransformersModelType,
TransformersPromptStyle,
)

llava_qwen = InlineVlmOptions(
repo_id="llava-hf/llava-interleave-qwen-0.5b-hf",
# prompt="Read text in the image.",
prompt="Convert this page to markdown. Do not miss any text and only output the bare markdown!",
# prompt="Parse the reading order of this document.",
response_format=ResponseFormat.MARKDOWN,
inference_framework=InferenceFramework.TRANSFORMERS,
transformers_model_type=TransformersModelType.AUTOMODEL_IMAGETEXTTOTEXT,
supported_devices=[AcceleratorDevice.CUDA, AcceleratorDevice.CPU],
scale=2.0,
temperature=0.0,
)

pdf_pipe_options.vlm_options = llava_qwen

# Converter
converter = DocumentConverter(
allowed_formats=[
InputFormat.PDF,
InputFormat.DOCX,
InputFormat.PPTX,
InputFormat.XLSX,
InputFormat.CSV,
],
format_options={
InputFormat.PDF: PdfFormatOption(
pipeline_cls=VlmPipeline,
pipeline_options=pdf_pipe_options, backend=PyPdfiumDocumentBackend), # Using default backend
# InputFormat.PDF: PdfFormatOption(pipeline_options=pdf_pipe_options, backend=PyPdfiumDocumentBackend),
}
```
...

### Docling version
docling: 2.48.0
docling-core: 2.45.0

...

### Python version
Python 3.12.11
...