docling-project / docling-project/docling-eval

Performance optimisations for the computation of the evaluation metrics and the data loading

Open
#192 1 comment 0 reactions 1 assignee Claimed by @nikos-livathinos View on GitHub
Dominant language
Python
Stars
77
Forks
14
PR merge metrics
No merged PRs in 30d

Description

We want to speedup the evaluation runtime:

1. Parallelize the computation of evaluation metrics. This can be easily done for those metrics where each sample can be computed independently from each other.
2. Redesign the data loading to pre-fetch and cache the ground truth and prediction documents from the disk. The loaded `DoclingDocument` objects can be then shared across all evaluators, instead of having each evaluator reloading the data. This is particularly useful when we run on slow storage, where I/O time dominates.
3. Refactor the API/CLI to allow the computation of multiple modalities in one go. This is essential to take advantage of the point 2 as it maximizes the usage of the pre-loaded data.

For the computation of the evaluation metrics, we want:

- [x] Parallelize `TableEvaluator` to compute TEDS scores of each table independently.
- [x] Parallelize `PixelLayoutEvaluator` to compute confusion matrices and metrics for each page independently.
- [x] Parallelize `MarkDownTextEvaluator` to compute text metrics for each page independently.
...

For the optimizations in the data loading, we want:

- [ ] Load and cache ground truth `DoclingDocument` objects from a `parquet` dataset.
- [ ] Load and cache prediction `DoclingDocument` objects from a `parquet` dataset.
- [x] Load and cache prediction `DoclingDocument` objects from externally provided files (`dt`, `json`, `yaml`, etc.)
- [x] Refactor the `evaluate()` method and the CLI to receive multiple modalities.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.