Refactor/optimize the `/compareSummary` API
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.1k
- Forks
- 564
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 29
Description
Opening this per #12071.
Repeating/rephrasing from https://github.com/IQSS/dataverse/issues/12071#issuecomment-3745372326:
The performance of /api/datasets/.../versions/compareSummary seems unreasonable/unacceptable on datasets with many files.
Testing on qa w/ one of our control datasets, w/ ~1.5 K files, /api/datasets/:persistentId/versions/compareSummary?persistentId=doi:10.7910/DVN/29236 takes ~17 sec. as of 6.9. The actual differences between versions (3 total) are minimal (~1Kb of json output). Most of this time must be from instantiating, then crawling through the lists of filemetadatas/datafiles for each version.
My educated guess is that it should be easy to replace this with native queries and achieve the same result much faster.
We already have a named native query in FileMetadata.java for finding files changed between versions that @qqmyers created for the purposes of optimizing file indexing. It can be used as a model/to build on top of.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the /api/datasets/.../versions/compareSummary endpoint and inspect FileMetadata.java, especially the existing named native query for finding files changed between versions. Compare the current behavior and output with the query-based approach using a dataset with many files. Done means preserving the compareSummary result while substantially reducing its response time.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend, databases, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100