Add process to check for major drops in data between updates

Terms

• I have searched open and closed feature requests
• I agree to follow Scribe-Server's Code of Conduct

Description

Based on https://github.com/scribe-org/Scribe-Data/issues/68, we need to keep in mind that there will be cases that a property on Wikidata will change such that there will be a large drop in data. In the referenced issue, Portuguese verbs are using a non-standard past perfect PID that could be combined with the more widely used one at some point.

This issue would look into ways of diffing the current data coverage against the new data coming in, which could be as simple as total keys and total non-null values of keys of sub-objects. We could then discuss a viable cutoff, and trigger some kind of warning or a Scribe-Data issue if it's too low 😊

Contribution

Would be happy to discuss! Could also help implement, but might be better if others get to this eventually as I'm a long way off on Go :)

Start by reviewing the referenced Scribe-Data issue #68 and how Scribe-Server currently receives or processes updated data. Define what coverage metrics should be compared, agree on a cutoff for a major drop, and document what warning or Scribe-Data issue should be triggered when it is crossed.