docling-project / docling-project/docling-eval

Introduce a num_of_sample/evaluated_samples parameter to the evaluate function in the docling-eval module

Open
#141 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
77
Forks
14
PR merge metrics
No merged PRs in 30d

Description

I noticed that the **DatasetEvaluation** class includes a variable **evaluated_samples**

```python
class DatasetEvaluation(BaseModel):
evaluated_samples: int = -1
rejected_samples: Dict[EvaluationRejectionType, int] = {}
```

However, it seems the current evaluator classes only use this parameter to process the entire dataset (test split) in the benchmark. I’m wondering if we could allow an arbitrary value to be passed during the evaluation dataset construction phase. This could help speed up the evaluation process for benchmark like OmniDocBench, which currently takes about an hour to complete on my machine.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.