GSA / GSA/data.gov

exclude invalid and unmodified records from harvesting

Open
#5,299 4 comments 0 reactions 1 assignee Claimed by @palakGSA View on GitHub
Harvest-Runner
Dominant language
Python
Stars
1.1k
Forks
200
Avg merge
4h 56m
Merged PRs (30d)
5

Description

## User Story

In order to improve harvesting by minimizing work, datagov wants to exclude previously invalid records which are unmodified from harvesting.

## Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using [BDD](https://en.wikipedia.org/wiki/Behavior-driven_development#Behavioral_specifications).]

- [ ] GIVEN an invalid record stored in the db \
WHEN harvesting happens and the record is unmodified since the last time \
THEN it will be skipped \

## Background
- we want to minimize the work the harvest runner needs to do.
- there's no point in processing an invalid record which hasn't changed

## Security Considerations ([required](https://nvd.nist.gov/800-53/Rev4/control/CM-4))

[comment]: # "Our SSP says 'The Data.gov team ensures security implications are considered as part of the agile requirements refinement process by including a section in the issue template used as a basis for new work.' so please don't remove this section without care."
[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

## Sketch

[Notes or a checklist reflecting our understanding of the selected approach]

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.