exogen / exogen/graphbrainz

Batch lookup queries using the search endpoint with ID fields

Open
#78 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
JavaScript
Stars
166
Forks
29
PR merge metrics
No merged PRs in 30d

Description

I've been running a large amount of complex queries, and often it's the case that deeply nested results lead to lots of individual lookup queries for entities.

I hacked up a proof-of-concept that uses [dataloader](https://github.com/facebook/dataloader)'s batching feature to recognize such lookups and make them using the `search` endpoint.

e.g. `/ws/2/recording?query=rid:"{MBID}" OR rid:"{MBID}" OR ...`

Then it will automatically map the results to the original, individual lookup requests, so they don't even know a different endpoint was used.

It works surprisingly well. The search endpoint also tends to return more data for each entity by default. This can speed up queries by 20–50x and avoid waiting for rate limits to clear up.

I'm planning on a rewrite of the whole GraphBrainz "query engine" to support this.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.