Speed up Search API (e.g. for search results map)
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.1k
- Forks
- 564
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 29
Description
Overview of the Suggestion
Can the trade-off between Search API speed and metadata richness be improved?
Background: Accessing the search API causes a DB lookup step which fetches the structured metadata to generate the JSON response. query_entities=false can be used to skip this step and accelerate the response (up to a factor of 10), at the cost of getting a response with only minimal metadata.
I've recently run into this when working on this search results map PR. Geolocations must be in the search response, but the speed-up from query_entities=false would be important as well.
Now, the question is whether the search API can return more rich metadata without having to do the DB look-up. Three options came to mind:
- During indexing, all the structured metadata gets loaded once, and saved into a single Solr field (
indexed="false"&stored="true") as JSON. With costs to storage and indexing effort, this could allow skipping the DB lookup and getting a rich response. - Across all currently indexed fields, Solr already contains much more metadata than
query_entities=falsequeries return. A repeatable Search API parameter (e.g.solr_fetch=fieldName) could append contents of the specified Solr field to the response. This could probably serve many cases in whichquery_entities=truewould otherwise be needed. - The Solr fields needed in this case (
geolocation/boundingBox) could simply be added to the fixed set of Solr fields that are added to the search API response by default.
What kind of user is the suggestion intended for?
Depends on how this is addressed: Option 1 should benefit anyone using the search page / Option 2 would benefit developers implementing features that need to search for datasets and receive specific fields / Option 3 would only benefit those developers who require geolocation/boundingBox.
What inspired this idea?
Comment by @qqmyers mentioning the potential for speed-up: https://github.com/IQSS/dataverse-frontend/pull/1027#issuecomment-5192448650
What existing behavior do you want changed?
The metadata returned by the Search API with query_entities=false.
Any open or closed issues related to this suggestion?
Not aware of any.
Are you thinking about creating a pull request for this issue?
If one of these options, or another method, is deemed realistic and welcome, I can contribute to the implementation.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Search API behavior for query_entities=false and the geolocation/boundingBox fields described in the issue. Compare the three proposed approaches, including their indexing, storage, and response implications; done means agreeing on a realistic approach and defining the resulting metadata and speed behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend-api-design, databases, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100