kit-data-manager / kit-data-manager/base-repo

weak search performance under certain conditions

Open
#22 1 comment 0 reactions 0 assignees View on GitHub
wontfix
Dominant language
Java
Stars
1
Forks
3
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
Under certain conditions, search performance using the '/api/v1/dataresources/search' endpoint is unexpectedly slow. Mainly by including the ResourceType attribute, queries may cost a factor of 10. Due to relatively fast queries this effect gets relevant for very huge repositories (spotted at 84k DataResources)

**To Reproduce**
Steps to reproduce the behavior:
1. Use search endpoint, e.g., http://{{HOSTNAME}}:{{PORT}}/api/v1/dataresources/search
2. Provide a (valid) example DataResource including resourceType attribute in the request body, e.g.
```
{
"resourceType":{
"typeGeneral":"TEXT"
},
"publicationYear":"2019"
}
```
3. Measure the response time, e.g., 2 seconds
4. Change the example DataResource to include a full resourceType, e.g.

```
{
"resourceType":{
"typeGeneral":"TEXT",
"value":"manuscriptMetadata"
},
"publicationYear":"2019"
}
```
5. Measure the response time, e.g., 20 seconds
6. Change the example DataResource again to include another top-level, primitive attribute, e.g., publisher
7. Measure the response time, e.g., 2 seconds

**Expected behavior**
The expected behaviour is, that the response time does not change within an entire order of magnitude. Including resourceType will require a more complex query than querying only for, e.g., publisher and publicationYear. However, the impact should be much less and it should not depend on the number of primitive attributes within the query (see difference in measurement 5 and 7).

**Screenshots**
none

**Desktop (please complete the following information):**
- OS: MacOS, Windows, Unix
- REST Tool: Postman
- Version 7.36.0

**Additional context**
none

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the /api/v1/dataresources/search endpoint with the three request bodies described in the issue and compare response times. Trace the search implementation from that endpoint to identify why a full resourceType causes the slowdown. Done means resourceType queries no longer produce an order-of-magnitude regression on large repositories.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.