docling-project / docling-project/docling
Enable automatic AVX-512 optimizations for Docling models on Intel/AMD CPUs
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
### Question
I want to run Docling on CPU-only VMs (to cut costs) since we don’t need OCR. Specifically, I’d like to optimize Docling’s default models (e.g., the layout_model) for Intel CPUs with AVX-512 support, and possibly new AMD CPUs that also support AVX-512.
The idea is to apply something like:
```
import intel_extension_for_pytorch as ipex
model = model.to("cpu")
model = ipex.optimize(model)
```
However, I couldn’t find any working examples. Both Grok and ChatGPT only returned outdated code, as shown below.
My questions are:
- Is this optimization approach achievable for Docling models?
- If yes, how should it be done correctly?
- Ideally, Docling’s internal code would detect AVX-512 support and enable these optimizations automatically, since Grok claims this could yield a 1.5–2× speedup.
```
class CustomPdfPipeline(StandardPdfPipeline):
def __init__(self, pipeline_options: PdfPipelineOptions):
super().__init__(pipeline_options)
# Optimize the layout model's PyTorch backbone for Intel CPU
# Assuming recent releases use 'layout_model' which contains 'layout_predictor'
if hasattr(self, 'layout_model') and hasattr(self.layout_model, 'layout_predictor'):
model_to_optimize = self.layout_model.layout_predictor.model
else:
# Fallback if structure differs; log or raise if needed
raise AttributeError("Layout predictor not found under expected attributes. Check Docling source for updates.")
if torch.cpu.is_bf16_supported():
model_to_optimize = ipex.optimize(model_to_optimize, dtype=torch.bfloat16)
else:
model_to_optimize = ipex.optimize(model_to_optimize, dtype=torch.float32)
# For inference, ensure eval mode (already set, but confirm)
model_to_optimize.eval()
# Set up the converter with the custom pipeline
pipeline_options = PdfPipelineOptions() # Add OCR or other options as before
converter = DocumentConverter(
format_options={
InputFormat.PDF: PdfFormatOption(
pipeline_cls=CustomPdfPipeline,
pipeline_options=pipeline_options,
),
}
)
```
Contributor guide
Assessment
This issue has not been assessed yet.