Materials-Consortia / Materials-Consortia/optimade-python-tools
Support for deep pagination of Elasticsearch queries
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 91
- Forks
- 54
- PR merge metrics
- No merged PRs in 30d
Description
Empirically, large queries to Elasticsearch fail at the point of pagination after 10000 entries (presumably this is configurable), see e.g., querying for a middle page of NOMAD:
https://nomad-lab.eu/prod/rae/optimade/v1/structures?page_offset=10000, which returns
RequestError: RequestError(400,
'search_phase_execution_exception', 'Result window is too large, from + size must be less than or equal to: [10000] but was [10020]. See the scroll api for a more efficient way to
request large data sets. This limit can be set by changing the index level setting.')
This might be desired behavior for elasticsearch-based OPTIMADE APIs (pending @markus1978 response), but perhaps we could use the suggested scroll API within OPT to abstract this away for these large queries.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failure with the NOMAD structures endpoint at page_offset=10000 and review how Elasticsearch pagination is currently handled. Investigate whether the Elasticsearch scroll API can provide the requested deep pages without exposing backend details through the OPTIMADE API. Done means large-offset queries work consistently, or the supported behavior and limits are documented and tested.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, python
- Domain
- api, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100