Implement an incremental harvest-time process for opensearch sync
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 200
- Avg merge
- 4h 56m
- Merged PRs (30d)
- 5
Description
## Feature/what we're after
Clearing opensearch and then re-filling isn't a good long term solution, so we need a way to sync harvester changes to opensearch at the time of harvest.
## Anticipated/hypothesized benefits
- opensearch has most recent harvested data (no daily delay)
## Measurements/metrics
- When a harvest is completed, then the changes (insertions, updates, deletions) are synced to open search and is able to be confirmed via search or API.
## References/background
- Related to recent needs to re-sync and having the counts go to 0 (and search being limited)
## Sketch
This could be done on a dataset individual level or in a batch at the end of the harvest job, whichever is easier. Batch would probably be faster, but individual may be easier to code/implement with the different cases of addition, update, and delete.
Contributor guide
Assessment
This issue has not been assessed yet.