GSA / GSA/data.gov

Implement an incremental harvest-time process for opensearch sync

Open
#5,607 9 comments 0 reactions 1 assignee Claimed by @FuhuXia View on GitHub
Dominant language
Python
Stars
1.1k
Forks
200
Avg merge
4h 56m
Merged PRs (30d)
5

Description

## Feature/what we're after

Clearing opensearch and then re-filling isn't a good long term solution, so we need a way to sync harvester changes to opensearch at the time of harvest.

## Anticipated/hypothesized benefits

- opensearch has most recent harvested data (no daily delay)

## Measurements/metrics

- When a harvest is completed, then the changes (insertions, updates, deletions) are synced to open search and is able to be confirmed via search or API.

## References/background

- Related to recent needs to re-sync and having the counts go to 0 (and search being limited)

## Sketch

This could be done on a dataset individual level or in a batch at the end of the harvest job, whichever is easier. Batch would probably be faster, but individual may be easier to code/implement with the different cases of addition, update, and delete.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.