kit-data-manager / kit-data-manager/base-repo
weak search performance under certain conditions
- Dominant language
- Java
- Stars
- 1
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
Under certain conditions, search performance using the '/api/v1/dataresources/search' endpoint is unexpectedly slow. Mainly by including the ResourceType attribute, queries may cost a factor of 10. Due to relatively fast queries this effect gets relevant for very huge repositories (spotted at 84k DataResources)
**To Reproduce**
Steps to reproduce the behavior:
1. Use search endpoint, e.g., http://{{HOSTNAME}}:{{PORT}}/api/v1/dataresources/search
2. Provide a (valid) example DataResource including resourceType attribute in the request body, e.g.
```
{
"resourceType":{
"typeGeneral":"TEXT"
},
"publicationYear":"2019"
}
```
3. Measure the response time, e.g., 2 seconds
4. Change the example DataResource to include a full resourceType, e.g.
```
{
"resourceType":{
"typeGeneral":"TEXT",
"value":"manuscriptMetadata"
},
"publicationYear":"2019"
}
```
5. Measure the response time, e.g., 20 seconds
6. Change the example DataResource again to include another top-level, primitive attribute, e.g., publisher
7. Measure the response time, e.g., 2 seconds
**Expected behavior**
The expected behaviour is, that the response time does not change within an entire order of magnitude. Including resourceType will require a more complex query than querying only for, e.g., publisher and publicationYear. However, the impact should be much less and it should not depend on the number of primitive attributes within the query (see difference in measurement 5 and 7).
**Screenshots**
none
**Desktop (please complete the following information):**
- OS: MacOS, Windows, Unix
- REST Tool: Postman
- Version 7.36.0
**Additional context**
none
Contributor guide
Research direction
Start by reproducing the /api/v1/dataresources/search endpoint with the three request bodies described in the issue and compare response times. Trace the search implementation from that endpoint to identify why a full resourceType causes the slowdown. Done means resourceType queries no longer produce an order-of-magnitude regression on large repositories.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100