bench(extract-worker): perform load testing
@ClemDoum is already working on this.
Since Jun 12, 2026.
Assessment
This issue has not been assessed yet.
Description
In order to avoid discovering our system at large scale the first time we'll need it, perform load testing at moderately large scale to assess:
- end-to-end processing time (with temporal overhead, machine provisioning and so on): "how many hours does it take to extract markdown content from P pages"
- does it scale properly
- can we find a fast/lightweight/cheap config vs. precise/full VLM config
- how much does it cost ?
- is worker startup fast enough for small volumes of documents ?
- cache model download on workers: https://docling-project.github.io/docling/usage/advanced_options/#model-prefetching-and-offline-usage
- Dominant language
- Python
- Stars
- 7
- Forks
- 0
- Avg merge
- 2h 5m
- Merged PRs (30d)
- 18
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ICIJ/datashare-python
-
ICIJ/datashare-python#89 · 1 comment · 1 assignee ·
-
ICIJ/datashare-python#88 · 1 comment · 1 assignee ·
-
datashare-python: configure worker timeouts (heartbeat + activity timeout) on the worker config side Open
ICIJ/datashare-python#87 · 1 comment · 2 assignees ·
-
ICIJ/datashare-python#61 · 1 assignee ·
-
ICIJ/datashare-python#53 · 1 comment · 2 assignees ·
All issues in ICIJ/datashare-python
Similar issues
-
link-check link-check:sphinx-theme
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
qgis/QGIS-Documentation#11275 ·
-
bug priority:normal ready-for-dev
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
OpenHands/extensions#626 · 1 comment ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
CSCfi/sd-search-api#39 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100