geonetwork / geonetwork/core-geonetwork
OGC CSW 2.0.2 Harvesting / Performance / Configuration of getRecords-Value
- Dominant language
- Java
- Stars
- 521
- Forks
- 514
- Avg merge
- 6d 13h
- Merged PRs (30d)
- 19
Description
**Is your feature request related to a problem? Please describe.**
This is a suggestion for improving harvesting performance by configuring the `maxRecords` value for the `getRecords` request per harvester.
An impact of the `getRecord` value on performance was noticed by the following observation in a harvester.
> [!WARNING]
By reducing the response of a CSW harvester to 10 data records (instead of 20), the harvesting time has increased enormously from 13 hours to 26 hours.
We used the profiling tool VisualVM to analyze which methods require the most time during the harvesting process:
The analysis showed that the `align` method with the initialization of the `UUIDMapper` class was called twice as often.
Therefore, for every 10 data records (instead of 20), a DB query on the metadata table with filtering of the data for the harvester is executed. With 259,188 metadata records, this corresponds to 25,918 DB queries which is evident from the number of geonetwork warnings
> Declared number of returned records (10) does not match requested record count (20)
in the harvester log file.
Before the switch to 10 records, only 12,959 DB queries would have been necessary.
Additionally, a matching of the local metadata with the remote metadata is performed for every 10 data records. Therefore, 10 metadata records of the CSW response are compared to all 259,188 metadata records of the harvester stored in the DB. This matching process is repeated 25,918 times (instead of 12,959 times with 20 metadata records within the CSW response). In total about 3.3 billion metadata records were compared during one harvesting process of 259,188 metadata records.
The database queries and matching represent a bottleneck due to partially time-consuming methods (`setDateAndTime`).
In addition, more `getRecords` queries against the CSW interface are necessary to retrieve all data.
**Describe the solution you'd like**
CSW interfaces might support a higher response value than 20 for `maxRecords`.
For each response to the `getRecords` query, the `align` method is called, which creates a new instance of the `UUIDMapper`. When the `UUIDMapper` is instantiated, the `findAllSimple` method is called, which determines all metadata records already available in the GN for the given harvester with a DB query.
With fewer `getRecords` queries due to a higher `maxRecords` value, the `align` method is called less often and therefore fewer DB queries are required.
An additional setting in the harvester settings to set this value per harvester might significantly improve harvesting performance.
Default value: 20
**Additional context**
Result of Visual VM analysis:

Contributor guide
Research direction
Start by tracing the CSW harvester's getRecords request into align, UUIDMapper, and findAllSimple; the issue names no files, so locate these entry points first. Check how maxRecords is currently set to 20 and how per-harvester settings are loaded. Done means the value is configurable per harvester, defaults to 20, and harvesting uses the configured value.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend, databases, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100