docling-project / docling-project/docling

Enable automatic AVX-512 optimizations for Docling models on Intel/AMD CPUs

Open
#2,332 2 comments 0 reactions 0 assignees View on GitHub
question triage/close-stale
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Question

I want to run Docling on CPU-only VMs (to cut costs) since we don’t need OCR. Specifically, I’d like to optimize Docling’s default models (e.g., the layout_model) for Intel CPUs with AVX-512 support, and possibly new AMD CPUs that also support AVX-512.

The idea is to apply something like:

```
import intel_extension_for_pytorch as ipex
model = model.to("cpu")
model = ipex.optimize(model)
```

However, I couldn’t find any working examples. Both Grok and ChatGPT only returned outdated code, as shown below.

My questions are:

- Is this optimization approach achievable for Docling models?
- If yes, how should it be done correctly?
- Ideally, Docling’s internal code would detect AVX-512 support and enable these optimizations automatically, since Grok claims this could yield a 1.5–2× speedup.

```
class CustomPdfPipeline(StandardPdfPipeline):
def __init__(self, pipeline_options: PdfPipelineOptions):
super().__init__(pipeline_options)
# Optimize the layout model's PyTorch backbone for Intel CPU
# Assuming recent releases use 'layout_model' which contains 'layout_predictor'
if hasattr(self, 'layout_model') and hasattr(self.layout_model, 'layout_predictor'):
model_to_optimize = self.layout_model.layout_predictor.model
else:
# Fallback if structure differs; log or raise if needed
raise AttributeError("Layout predictor not found under expected attributes. Check Docling source for updates.")

if torch.cpu.is_bf16_supported():
model_to_optimize = ipex.optimize(model_to_optimize, dtype=torch.bfloat16)
else:
model_to_optimize = ipex.optimize(model_to_optimize, dtype=torch.float32)
# For inference, ensure eval mode (already set, but confirm)
model_to_optimize.eval()

# Set up the converter with the custom pipeline
pipeline_options = PdfPipelineOptions() # Add OCR or other options as before
converter = DocumentConverter(
format_options={
InputFormat.PDF: PdfFormatOption(
pipeline_cls=CustomPdfPipeline,
pipeline_options=pipeline_options,
),
}
)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.