Enabling new parameters in DataDriftDetector.update()

Dear colleagues,

I have the following scenarion:

- On monthly basis we are reloading the raw data that we used to train some AML models.
- Our training pipeline consist of a preprocessing step, and hyperdrive step and a model register one (where we pickup the best model by hyperdrive)
- I have created an AML monitoring pipeline were I am creating programatically a datadrift object and setting the baseline dataset = target dataset if it is the first run. Otherwise the plan is to run a backfill job between the dataset attached to the models when doing best_run.register_model(...., datasets=[('training_data': my_training_data)]. If there is drift then an event is sent to event hub and an ADF pipeline triggers a retraining process.

However I have found something very problematic with this approach. When looking to the method DataDriftDetector.update(). There is no possibility to update the target and the baseline datasets. I can only update the treshold or the features I am measuring the drift for.

Basically the code looks like this:

```python
logging.info("Getting workspace configuration")

run = Run.get_context()
ws = Workspace.from_config() if not hasattr(
run, 'experiment') else run.experiment.workspace

old_model_monitor_object = get_current_drift_detector(
ws=ws, monitoring_name=args.drift_detector_name) #if it doesnt find it it set the variable to none
monitoring_features = get_features_to_monitor(ws=ws, arguments=args)
latest_training_data = get_latest_dataset(ws, arguments=args)
baseline_training_data = get_reference_dataset(
monitor_object=old_model_monitor_object, ml_data=latest_training_data) #if there is no monitor object baseline=latest
monitoring_cluster = ws.compute_targets['preprocessing-cluster-cpu'] if not hasattr(
run, 'experiment') else ws.compute_targets[run.get_details()['target']]
monitoring_alert_list = subscription_email_list()
last_time = pd.to_datetime(latest_training_data.to_pandas_dataframe(
)[args.param_time_index_column], format='%Y-%m-%d').max().to_pydatetime()
baseline_time = pd.to_datetime(baseline_training_data.to_pandas_dataframe(
)[args.param_time_index_column], format='%Y-%m-%d').max().to_pydatetime()

execute_monitoring_logic(monitor_object=old_model_monitor_object,
monitoring_name=args.drift_detector_name,
baseline_df=baseline_training_data,
target_df=latest_training_data,
compute_target=monitoring_cluster,
features_to_monitor=monitoring_features,
treshold=args.detection_treshold,
monitoring_alert=monitoring_alert_list,
last_timestamp=last_time,
current_time=baseline_time,
current_run=run)
```

The execute monitoring function is as follows:

```python
def execute_monitoring_logic(monitor_object, monitoring_name, baseline_df, target_df,
compute_target, features_to_monitor, treshold, monitoring_alert,
last_timestamp, current_time, current_run):
try:
if monitor_object:
# we update the monitoring object with the last target_df which
# corresponds to the last version of the dataset
monitor_object_new = DataDriftDetector.create_from_datasets(workspace=ws,
name=monitoring_name,
baseline_dataset=baseline_df,
target_dataset=target_df,
compute_target=compute_target,
feature_list=features_to_monitor,
drift_threshold=treshold,
alert_config=monitoring_alert)
# we backfill the metrics from the last timestamp of the previous baseline
# to the last timestamp of the current dataset
logging.info("Calculating backfill metrics")
backfill_job = monitor_object_new.backfill(
start_time=last_timestamp, end_date=current_time, compute_target=compute_target)
backfill_job.wait_for_completion()
if monitor_object_new.get_output(run_id=backfill_job.id)[0][0]['result'][0]['has_drift']:
# if drift is detected we need to recreate the datadrift detector
# to assign as new baseline the last dataset (current target)
logging.info("Drift detected!")
monitor_object_new = DataDriftDetector.create_from_datasets(workspace=ws,
name=monitoring_name,
baseline_dataset=target_df,
target_dataset=target_df,
compute_target=compute_target,
feature_list=features_to_monitor,
drift_threshold=treshold,
alert_config=monitoring_alert)
logging.info(
"Updating monitor object, replacing baseline dataset with current target")
else:
logging.info("Creating monitor object for the first time")
monitor_object = DataDriftDetector.create_from_datasets(workspace=ws,
name=monitoring_name,
baseline_dataset=baseline_df,
target_dataset=target_df,
compute_target=compute_target,
feature_list=features_to_monitor,
drift_threshold=treshold,
alert_config=monitoring_alert)
except Exception as e:
logging.error(e)
logging.error("Failed to create or update model monitoring object")
cancel_pipeline_run(current_run)
```

Because I can not run:

`DataDriftDetector.update(new_baseline='target_df', new_target='target_df')`

When drift is detected, I need to recreate the model monitoring object all the time (which is not showed in the code above), is there any chance that this can be implemented?.

At the end the logic that Im trying to follow if this one:

- Everytime we reload data in ADF we trigger a new AML pipeline (the monitoring one)
- The AML pipeline runs the preprocess step to create additional features and register dataset as new training data.
- Then the AML pipelines runs the monitoring step.
- The monitoring step checks if there is a azure datadrift object of a given name (which is a pipeline parameter for the AML monitoring pipeline)
- If not, it creates a datadrift object and it sets as baseline dataset the one that we have just registered. Also it setups the target dataset as the baseline (as there is nothing to compare against)
- If the datadrift object already exists (because it was created before) we get the latest timestamp from the baseline dataset linked to the datadrift object:
- Then we get the latest timestamp from the latest version of the training data.
- We run a backfill job (between these two timestamps) and we compute if there is drift.
- If there is no drift we update the datadrift object and we register the latest version of the training data as the new target dataset.
- If there is drift then we update the datadrift object and we register the latest version of the training data as new baseline and target
- If there is drift the datadrift object will send an event to event grid which will trigger the retraining pipeline from ADF (we have not configured this yet in ADF).

Is there an easy way of doing this? At the end for me it seems natural that we are updating the baselines when we retrain a model but maybe I am in the wrong direction.

If I am on the right track is there any possibility to include this in the SDK?

BR
E

まず Azure ML SDK の DataDriftDetector.update() と create_from_datasets() の実装を見つけ、要求されている baseline データセットと target データセットの変更を、ここで示されている再作成による回避策と比較します。完了の条件は、detector が再作成なしで両方のデータセットを更新でき、その動作が関連するテストでカバーされていることです。issue にはファイルパスもテストパスも記載されていません。

索引モデルが issue の本文から書いたものです。