deepchecks / deepchecks/deepchecks
[DEE-13] [FEAT] Control charts for temporal datasets
- Dominant language
- Python
- Stars
- 4.1k
- Forks
- 303
- PR merge metrics
- No merged PRs in 30d
Description
## Is your feature request related to a problem? Please describe.
I'd like to be able to validate future data based on what old data looked like, and flag outlying values. The goal is to catch errors and process failures, not monitor slow drift.
**Examples:**
- Suppose I have a list of historical performance of a model. I'd like to check if this year's model performance is in the same range as before or if something has changed. I don't care much about gradual changes to the performance. Rather, I want to detect sudden changes that might indicate a bug or major failure.
- Suppose I have a large dataset that is updated in batches (of size 1+) over time. At each new batch, I'd like to validate that the data falls within the range expected from past data. Again, the goal is to catch bugs, errors, and invalid data, not to detect slow drift over time.
## Describe the solution you'd like
I'd like to use [**control charts**](https://en.wikipedia.org/wiki/Control_chart) to do this kind of monitoring. Learn a model from past data, get what you think new data should look like, and then compare the new data to the expectation.
The implementation could look like this:
```python
check = ControlChart("performance_score", timedate="timedate_column", check_range="latest")
check.add_condition_statistical_control(confidence=0.99) # Only flag data falling outside of the 99% confidence interval.
check.add_condition_limits(range=(0.8, 0.9)) # Check that latest performance is between 0.8 and 0.9.
check.run(dataset) # Check that the latest performance score is as expected.
```
The name "control chart" could be replaced by "process monitoring" or something more explicit.
## Describe alternatives you've considered
Some of this is already done in train/test validation checks and there's already an outlier detection check. However, the goal is to detect drift rather than to check for errors given incremental changes.
I think that some form of control charts or incremental outlier detection is necessary for Deepchecks to be used as part of a monitoring pipeline.
[DEE-13](https://linear.app/deepchecks/issue/DEE-13/[feat]-control-charts-for-temporal-datasets)
Contributor guide
Assessment
This issue has not been assessed yet.