Custom data preprocessor component for Monitoring - Azure ML Python SDK v2 and Examples

Hello guys, I am working on creating a Monitoring using Azure ML mainly for checking the Data Quality signal (Quality of the baseline and production data) and Data Drift signal (Change in distribution of the production data with respect to the baseline data).

I have followed the official Azure ML Python SDK v2 documentation: Please Refer the Section - **Set up model monitoring of production data** in [https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2&tabs=python](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2&tabs=python).

My data is in `.csv` files rather than `.jsonl` files as mentioned in the example. I have modified the custom preprocessor code given in AML examples referred from the documentation: [https://github.com/Azure/azureml-examples/blob/main/cli/monitoring/components/custom_preprocessing/src/run.py](https://github.com/Azure/azureml-examples/blob/main/cli/monitoring/components/custom_preprocessing/src/run.py)

Some issues arise during the creation of pipeline for monitoring,
1. There is some issue in the example code: custom preprocessor component while creating a MLTable in the example provided and this is resolved.
2. The runtime version provided in the documentation is `3.3`. Due to the below exception, I have changed the spark compute runtime to `3.4`
```
MlException:

1) At least one required parameter is missing

Details:

(x) Compute runtime version must be 3.4

```

This is resolved and the below is the one used for monitoring job creation,
```
from azure.ai.ml.entities import ServerlessSparkCompute

spark_compute = ServerlessSparkCompute(
instance_type="standard_e4s_v3",
runtime_version="3.4"
)
```
Finally, I am able to run the entire monitoring pipeline i.e., select the data by `pattern`, create the `MLTable`, convert into `parquet` files and upload to the `input_preprocessed_data` location which in turn used by the Monitoring pipeline job.

The monitor job upon triggered is getting failed,
This is the error I am getting in the AML Console by checking out **Output + Logs** of `custom data preprocessor` component of the monitoring pipeline job:
```
The version of cryptography does not match the loaded shared object. This can happen if you have multiple copies of cryptography installed in your Python path. Please try creating a new virtual environment to resolve this issue. Loaded python version: 40.0.2, shared object version: b'42.0.5'
```

**Additional Notes:**
- I have added the cryptography 42.0.5 in the component.yaml under conda requirements and facing issues that - several packages are broken in the Azure ML console error logs.
- Apparently, some azure-ml packages require cryptography < 41 but the spark runtime environment created by the monitoring pipeline requires 42.0.5. I have replicated the same issue in the local system by using Python 3.8 environment and installing packages required for the custom preprocessor component.