epam / epam/Indigo

Add pagination to bingo-elastic [Java]

Open
#1,826 3 comments 1 reaction 1 assignee Claimed by @khyurri View on GitHub
User`s Feedback
Dominant language
C++
Stars
406
Forks
134
Avg merge
2d 11h
Merged PRs (30d)
24

Description

Update 13 Aug 2025:
Pagination is already implemented for Python using elasticsearch scroll feature. Please see https://github.com/epam/Indigo/pull/3089
Currently this functionality should be added to Java as well.

-----
Original issue text for reference
vvv

**Background**
Currently, users can only access a limited count of compounds (up to 10,000 by default) from ElasticSearch. We need to incorporate sorting and pagination functionality into the bingo-elastic driver. This will allow users to receive more than 10,000 compounds from the index.

**Solution**
ElasticSearch offers a pagination feature (PIT). Visit https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html to learn more.
We need to incorporate both the Java and Python functionality to enable transparent use of PIT, thus eliminating the need for user-level management. Therefore, if a user sets a sorting parameter, the bingo-elastic driver will automatically continue fetching documents until the limit is reached.
Importantly, after implementing this change, the limit option should be optional. This will allow users to download the entire index if they choose to do so.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.